CentOS 7 安装Kubernetes记录(带ca认证)

一、环境准备

最好为服务器配置静态IP,不然关机后很多地方的ip需要修改。参考文章

1. 所有节点配置host，使各个Node间可以通过主机名称进行解析。

主机名可以参考上面的文章链接进行修改

$ vi /etc/hosts
#加入如下片段(ip地址和servername替换成自己的)
172.16.136.201 server01
172.16.136.202 server02
172.16.136.203 server03

2.所有节点关闭防火墙

systemctl disable firewalld
systemctl stop firewalld

3.所有节点禁用SELinux

setenforce 0

4.所有节点关闭NetworkManager(虚拟机环境)

systemctl stop NetworkManager
systemctl disable NetworkManager

在虚拟机环境有时会出现Failed to start LSB: Bring up/down networking.错误，network无法启动。

5.关闭swap

为了性能考虑

swapoff -a

二、安装Docker (所有节点)

1.安装脚本

wget -qO- https://get.docker.com/ | sh

2.修改docker的网络转发规则

vi /lib/systemd/system/docker.service
#找到ExecStart=xxx，在这行上面加入一行，内容如下：(k8s的网络需要)
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT

3.(可选)给docker配置加速

可以到DaoCloud上注册账号免费申请,在页面底部的加速器选项。

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://0ed63f2c.m.daocloud.io

4.使配置生效并启动docker

systemctl daemon-reload
systemctl start docker
systemctl enable docker

5.验证是否安装完成

$ docker version

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:09:15 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:13:03 2018
  OS/Arch:      linux/amd64
  Experimental: false

三、Kubernetes安装

1. 设置系统参数 (所有节点)

• 允许路由转发，不对bridge的数据进行处理

#写入配置文件
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

#生效配置文件
sysctl -p /etc/sysctl.d/k8s.conf

2. 下载kubernetes二进制文件 (所有节点)

下载地址

下载完成后上传到服务器用户目录并解压，并将其目录加入到PATH环境变量中。

tar -zxf kubernetes-bins.tar.gz;rm -rf kubernetes-bins.tar.gz;mv kubernetes-bins bin

#添加环境变量
vim /etc/profile
export PATH=$PATH:/root/bin
source /etc/profile

3. 下载配置文件并生成配置文件 (所有节点)

使用yum install git安装Git

git clone https://github.com/KingBoyWorld/kubernetes-starter

3.1 生成带有权限认证的配置文件 (所有节点)

cd ~/kubernetes-starter && vi config.properties && ./gen-config.sh with-ca

需要注意的是其中主节点使用https， 如下示例配置

#kubernetes二进制文件目录,eg: /home/michael/bin
BIN_PATH=/root/bin

#当前节点ip,eg: 192.168.1.102
NODE_IP=172.16.136.201

#etcd服务集群列表,eg: http://192.168.1.102:2379
#如果已有etcd集群可以填写现有的。没有的话填写：http://${MASTER_IP}:2379 （MASTER_IP自行替换成自己的主节点ip）
ETCD_ENDPOINTS=https://172.16.136.202:2379

#kubernetes主节点ip地址,eg: 192.168.1.102
MASTER_IP=172.16.136.202

4. 安装cfssl并生成根证书 (所有节点)

4.1 安装cfssl(所有节点)

用来生成证书

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 && wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 && chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 && mv cfssl_linux-amd64 /usr/local/bin/cfssl && mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

cfssl version

4.2 生成根证书 (主节点)

mkdir -p /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-config.json /etc/kubernetes/ca && cp ~/kubernetes-starter/target/ca/ca-csr.json /etc/kubernetes/ca && cd /etc/kubernetes/ca && cfssl gencert -initca ca-csr.json | cfssljson -bare ca && ls && cd ~/kubernetes-starter

5. 安装ETCD服务 (主节点)

5.1 生成ETCD证书 (主节点)

mkdir -p /etc/kubernetes/ca/etcd && cp ~/kubernetes-starter/target/ca/etcd/etcd-csr.json /etc/kubernetes/ca/etcd/ && cd /etc/kubernetes/ca/etcd/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd && ls && cd ~/kubernetes-starter

5.2 安装并启动ETCD (主节点)

mkdir -p /var/lib/etcd && cp ~/kubernetes-starter/target/master-node/etcd.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable etcd && systemctl start etcd

验证是否安装完成

$ ETCDCTL_API=3 etcdctl \
  --endpoints=https://172.16.136.202:2379  \
  --cacert=/etc/kubernetes/ca/ca.pem \
  --cert=/etc/kubernetes/ca/etcd/etcd.pem \
  --key=/etc/kubernetes/ca/etcd/etcd-key.pem \
  endpoint health

#如下输出说明成功
https://172.16.136.202:2379 is healthy: successfully committed proposal: took = 1.04668ms

6. 安装APIServer (主节点)

6.1 生成证书 (主节点)

mkdir -p /etc/kubernetes/ca/kubernetes && cp ~/kubernetes-starter/target/ca/kubernetes/kubernetes-csr.json /etc/kubernetes/ca/kubernetes/ && cd /etc/kubernetes/ca/kubernetes/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes && ls && cd ~/kubernetes-starter

6.2 生成token认证文件 (主节点)

生成随机token

$ head -c 16 /dev/urandom | od -An -t x | tr -d ' '
8afdf3c4eb7c74018452423c29433609

按照固定格式写入token.csv，注意替换token内容

echo "8afdf3c4eb7c74018452423c29433609,kubelet-bootstrap,10001,\"system:kubelet-bootstrap\"" > /etc/kubernetes/ca/kubernetes/token.csv

6.3 启动APIServer (主节点)

cp ~/kubernetes-starter/target/master-node/kube-apiserver.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-apiserver.service && systemctl start kube-apiserver && cd ~/kubernetes-starter

验证

journalctl -f -u kube-apiserver

7. 安装ControllerManager (主节点)

安装

cp ~/kubernetes-starter/target/master-node/kube-controller-manager.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager

验证

journalctl -f -u kube-controller-manager

8. 安装Scheduler (主节点)

安装

cp ~/kubernetes-starter/target/master-node/kube-scheduler.service /lib/systemd/system/ && systemctl enable kube-scheduler.service && systemctl start kube-scheduler

验证

journalctl -f -u kube-scheduler

9. 配置kubectl (主节点)

9.1 生成证书 (主节点)

mkdir -p /etc/kubernetes/ca/admin && cp ~/kubernetes-starter/target/ca/admin/admin-csr.json /etc/kubernetes/ca/admin/ && cd /etc/kubernetes/ca/admin/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin && ls && cd ~/kubernetes-starter

9.2 配置kubectl (主节点)

注意修改其中的IP地址

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 && kubectl config set-credentials admin --client-certificate=/etc/kubernetes/ca/admin/admin.pem --embed-certs=true --client-key=/etc/kubernetes/ca/admin/admin-key.pem && kubectl config set-context kubernetes --cluster=kubernetes --user=admin && kubectl config use-context kubernetes

查看生成的文件内容

cat ~/.kube/config

验证

$ kubectl get componentstatus
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}
scheduler            Healthy   ok

10 配置calico-node (所有节点)

10.1 生成证书 (主节点)

mkdir -p /etc/kubernetes/ca/calico && cp ~/kubernetes-starter/target/ca/calico/calico-csr.json /etc/kubernetes/ca/calico/ && cd /etc/kubernetes/ca/calico/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes calico-csr.json | cfssljson -bare calico && ls && cd ~/kubernetes-starter

10.2 拷贝证书到其它机器上 (主节点)

可能需要在两台机器上建立/etc/kubernetes目录

scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/
scp -r /etc/kubernetes/ca/ root@172.16.136.201:/etc/kubernetes/ca/

10.2 启动calio-node (所有节点)

需要下载docker镜像，会慢一些

cp ~/kubernetes-starter/target/all-node/kube-calico.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kube-calico && systemctl start kube-calico

查看日志

journalctl -f -u kube-calico

验证

calicoctl node status

11 安装kubelet (工作节点)

11.1 配置查看 (主节点)

#可以通过下面命令查询clusterrole列表
$ kubectl -n kube-system get clusterrole
#可以回顾一下token文件的内容
$ cat /etc/kubernetes/ca/kubernetes/token.csv
8afdf3c4eb7c74018452423c29433609,10001,"system:kubelet-bootstrap"

11.2 环境准备

(工作节点)

mkdir -p /var/lib/kubelet && mkdir -p /etc/kubernetes && mkdir -p /etc/cni/net.d

(主节点)

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

11.3 生成bootstrap.kubeconfig配置文件 (工作节点)

注意替换IP地址 和 token

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=bootstrap.kubeconfig && kubectl config set-credentials kubelet-bootstrap --token=8afdf3c4eb7c74018452423c29433609 --kubeconfig=bootstrap.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig && kubectl config use-context default --kubeconfig=bootstrap.kubeconfig && mv bootstrap.kubeconfig /etc/kubernetes/

11.4 cni配置 (工作节点)

cp ~/kubernetes-starter/target/worker-node/10-calico.conf /etc/cni/net.d/

11.5 启动kubelet服务 (工作节点)

cp ~/kubernetes-starter/target/worker-node/kubelet.service /lib/systemd/system/ && systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet

#启动kubelet之后到master节点允许worker加入(批准worker的tls证书请求)
#--------*在主节点执行*---------
$ kubectl get csr|grep 'Pending' | awk '{print $1}'| xargs kubectl certificate approve
#-----------------------------

检查日志

journalctl -f -u kubelet

12 安装kube-proxy (工作节点)

创建工作目录 (工作节点)

mkdir -p /var/lib/kube-proxy

12.1 生成证书 (工作节点)

mkdir -p /etc/kubernetes/ca/kube-proxy && cp ~/kubernetes-starter/target/ca/kube-proxy/kube-proxy-csr.json /etc/kubernetes/ca/kube-proxy/ && cd /etc/kubernetes/ca/kube-proxy/ && cfssl gencert -ca=/etc/kubernetes/ca/ca.pem -ca-key=/etc/kubernetes/ca/ca-key.pem -config=/etc/kubernetes/ca/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy && ls && cd ~/kubernetes-starter

12.2 生成kube-proxy.kubeconfig配置 (工作节点)

注意替换IP地址

kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ca/ca.pem --embed-certs=true --server=https://172.16.136.202:6443 --kubeconfig=kube-proxy.kubeconfig && kubectl config set-credentials kube-proxy --client-certificate=/etc/kubernetes/ca/kube-proxy/kube-proxy.pem --client-key=/etc/kubernetes/ca/kube-proxy/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig && kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig && kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig && mv kube-proxy.kubeconfig /etc/kubernetes/kube-proxy.kubeconfig

12.3 启动kube-proxy (工作节点)

cp ~/kubernetes-starter/target/worker-node/kube-proxy.service /lib/systemd/system/ && systemctl daemon-reload && yum -y install conntrack && systemctl enable kube-proxy && systemctl start kube-proxy

检查

journalctl -f -u kube-proxy

13 部署kube-dns (主节点)

kubectl create -f ~/kubernetes-starter/target/services/kube-dns.yaml

验证

kubectl -n kube-system get pods