企业级k8s高可用集群搭建(无坑版,如有坑请联系解决)
2 --> 目录一.服务器规划二.资源准备(系统配置,内核优化,yum源配置)（所有节点操作）三.keepalived配置(master节点操作)四.haproxy配置(master节点操作)五.docker安装(所有节点操作)六.安装kubeadm,kubelet kubectl(所有节点操作)七.k8s集群安装（在具有vip的master上操作）八.安装集群网络(master节点操作)九.其他节点加入集群十.部署dashboard(k8s-master-01)十一.部署ingress（0.30.0）十二.部署metric十三.部署kubernetes dns缓存十四.k8s缩容扩容维护十五.k8s常见操作十六.参考下一篇，关于应用部署实际应用案例 一.服务器规划(基于腾讯云服务器的规划) k8s-master-01 10.206.16.14 master k8s-master-02 10.206.16.15 master k8s-master-03 10.206.16.16 master k8s-node-01 10.206.16.8 node k8s-node-02 10.206.16.9 node k8s-harbor 10.206.16.4 harbor vip 10.206.16.18 (https://cloud.tencent.com/document/product/215/36694 腾讯云vip申请办法) 服务器系统centos8 k8s版本1.16 机器配置4核16G 二.资源准备 1.设置主机的名字 hostnamectl set-hostname k8s-master-01(分别在对应的主机按照规划设置名字) 2.编辑hosts文件(在所有主机操作) cat <<EOF >>/etc/hosts 10.206.16.18 master.k8s.io k8s-vip 10.206.16.14 master01.k8s.io k8s-master-01 10.206.16.15 master02.k8s.io k8s-master-02 10.206.16.16 master03.k8s.io k8s-master-03 10.206.16.8 node01.k8s.io k8s-node-01 10.206.16.9 node02.k8s.io k8s-node-02 10.206.16.4 k8s-harbor EOF 3.关闭防火墙,selinux,swap(在所有主机操作) systemctl stop firewalld && systemctl disable firewalld && setenforce 0 && swapoff -a sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config 4.配置内核参数(所有主机操作) cat >/etc/sysctl.d/k8s.conf <<EOF net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables =1 net.bridge.bridge-nf-call-iptables =1 EOF modprobe br_netfilter sysctl -p /etc/sysctl.d/k8s.conf 5.设置资源配置文件(所有主机操作) echo "* soft nofile 65536" >> /etc/security/limits.conf echo "* hard nofile 65536" >> /etc/security/limits.conf echo "* soft nproc 65536" >> /etc/security/limits.conf echo "* hard nproc 65536" >> /etc/security/limits.conf echo "* soft memlock unlimited" >> /etc/security/limits.conf echo "* hard memlock unlimited" >> /etc/security/limits.conf 6.配置yum源(所有主机操作) yum install -y wget mkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos8_base.repo wget -O /etc/yum.repos.d/epel.repo http://mirrors.cloud.tencent.com/repo/epel-7.repo yum clean all && yum makecache 7.配置kubernetes源(所有主机操作) cat <<EOF >/etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF 8.配置docker源(所有主机操作) wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo 9.安装相关包(所有主机操作) yum install -y conntrack-tools libseccomp libtool-ltdl 三.部署keepalived（在3台master机器操作） 1.安装keepalived(在3台master操作) yum install -y conntrack-tools libseccomp libtool-ltdl 2.配置(另外的两台master配置和上面类似，只需要修改对应的state配置为BACKUP，priority权重值不同即可，配置中的其他字段这里不做说明) k8s-master-01配置: cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id k8s } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 250 nopreempt #设置非抢占模式 preempt_delay 10 #抢占延时10分钟 advert_int 1 #检查间隔默认1s authentication { auth_type PASS auth_pass ceb1b3ec013d66163d6ab11 } unicast_src_ip 10.206.16.14 #设置本机内网ip unicast_peer{ #其他两台master ip 10.206.16.15 10.206.16.16 } virtual_ipaddress { 10.206.16.18 } track_script { check_haproxy } } EOF k8s-master-02配置 cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id k8s } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 nopreempt preempt_delay 10 priority 200 advert_int 1 authentication { auth_type PASS auth_pass ceb1b3ec013d66163d6ab11 } unicast_src_ip 10.206.16.15 #设置本机内网ip unicast_peer{ 10.206.16.14 10.206.16.16 } virtual_ipaddress { 10.206.16.18 } track_script { check_haproxy } } EOF k8s-master-03配置: cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { router_id k8s } vrrp_script check_haproxy { script "killall -0 haproxy" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 150 nopreempt preempt_delay 10 advert_int 1 authentication { auth_type PASS auth_pass ceb1b3ec013d66163d6ab11 } unicast_src_ip 10.206.16.16 unicast_peer{ 10.206.16.14 10.206.16.15 } virtual_ipaddress { 10.206.16.18 } track_script { check_haproxy } } EOF 3.启动检查 设置开机启动 systemctl enable keepalived.service 启动 systemctl start keepalived.service 查看启动状态 systemctl status keepalived.service 启动后查看k8s-master-01网卡信息 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 52:54:00:f2:57:46 brd ff:ff:ff:ff:ff:ff inet 10.206.16.14/20 brd 10.206.31.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet 10.206.16.18/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fef2:5746/64 scope link valid_lft forever preferred_lft forever 尝试停掉k8s-master-01的keepalived服务，查看vip是否能漂移到其他的master，并且重新启动k8s-master-01的keepalived服务，查看vip是否能正常漂移回来，证明配置没有问题。 四.haproxy搭建(三台master上操作) 1.安装haproxy(三台master执行) yum install -y haproxy 2.配置(三台master执行) cat > /etc/haproxy/haproxy.cfg << EOF #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # kubernetes apiserver frontend which proxys to the backends #--------------------------------------------------------------------- frontend kubernetes-apiserver mode tcp bind *:16443 option tcplog default_backend kubernetes-apiserver #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend kubernetes-apiserver mode tcp balance roundrobin server master01.k8s.io 10.206.16.14:6443 check server master02.k8s.io 10.206.16.15:6443 check server master03.k8s.io 10.206.16.16:6443 check #--------------------------------------------------------------------- # collection haproxy statistics message #--------------------------------------------------------------------- listen stats bind *:1080 stats auth admin:awesomePassword stats refresh 5s stats realm HAProxy\ Statistics stats uri /admin?stats EOF 3.启动和检查(三台master执行) # 设置开机启动 systemctl enable haproxy # 开启haproxy systemctl start haproxy # 查看启动状态 systemctl status haproxy 4.检查服务端口是否启动 [root@VM-16-14-centos yum.repos.d]# netstat -lntup|grep haproxy tcp 0 0 0.0.0.0:1080 0.0.0.0:* LISTEN 37567/haproxy tcp 0 0 0.0.0.0:16443 0.0.0.0:* LISTEN 37567/haproxy udp 0 0 0.0.0.0:48413 0.0.0.0:* 37565/haproxy 五.安装docker（所有节点） 1.安装 # step 1: 安装必要的一些系统工具 $ yum install -y yum-utils device-mapper-persistent-data lvm2 # Step 2: 添加软件源信息 $ sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # Step 3: 查找Docker-CE的版本: $ yum list docker-ce.x86_64 --showduplicates | sort -r # Step 4: 安装指定版本的Docker-CE $ yum makecache $ yum install -y docker-ce 2.配置 修改docker的配置文件，目前k8s推荐使用的docker文件驱动是systemd mkdir /etc/docker cat > /etc/docker/daemon.json << EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.override_kernel_check=true" ]， "insecure-registries" : ["10.206.16.4"] } EOF 修改docker的服务配置文件，指定docker的数据目录为外挂的磁盘--graph /data/docker mkdir /data/docker sed -i "s#containerd.sock#containerd.sock --graph /data/docker#g" /lib/systemd/system/docker.service 添加如下执行语句:(如果pod之间无法通信的问题) mkdir -p /etc/systemd/system/docker.service.d/ cat>/etc/systemd/system/docker.service.d/10-docker.conf<<EOF [Service] ExecStartPost=/sbin/iptables --wait -I FORWARD -s 0.0.0.0/0 -j ACCEPT ExecStopPost=/bin/bash -c '/sbin/iptables --wait -D FORWARD -s 0.0.0.0/0 -j ACCEPT &> /dev/null || :' ExecStartPost=/sbin/iptables --wait -I INPUT -i cni0 -j ACCEPT ExecStopPost=/bin/bash -c '/sbin/iptables --wait -D INPUT -i cni0 -j ACCEPT &> /dev/null || :' EOF 3.启动docker $ systemctl daemon-reload $ systemctl start docker.service $ systemctl enable docker.service $ systemctl status docker.service 六.安装kubeadm,kubelet,和kubectl（所有节点操作） 1.安装 yum install -y kubelet-1.16.3 kubeadm-1.16.3 kubectl-1.16.3 systemctl enable kubelet 2.配置kubectl自动补全功能 source <(kubectl completion bash) echo "source <(kubectl completion bash)" >> ~/.bashrc 七.安装k8s集群(在具有vip的k8s-master-01上操作) 1.创建配置文件: mkdir /usr/local/kubernetes/manifests -p cd /usr/local/kubernetes/manifests/ cat > kubeadm-config.yaml <<EOF apiServer: certSANs: - k8s-master-01 - k8s-master-02 - k8s-master-03 - master.k8s.io - 10.206.16.14 - 10.206.16.15 - 10.206.16.16 - 10.206.16.18 - 127.0.0.1 extraArgs: authorization-mode: Node,RBAC timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta1 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: "master.k8s.io:16443" controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.16.3 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.1.0.0/16 scheduler: {} EOF 2.初始化master节点 [root@VM-16-14-centos manifests]# kubeadm init --config kubeadm-config.yaml [init] Using Kubernetes version: v1.16.3 [preflight] Running pre-flight checks [WARNING FileExisting-tc]: tc not found in system path [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 18.09 [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master.k8s.io k8s-master-01 k8s-master-02 k8s-master-03 master.k8s.io] and IPs [10.1.0.1 10.206.16.14 10.206.16.14 10.206.16.15 10.206.16.16 10.206.16.18 127.0.0.1] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master-01 localhost] and IPs [10.206.16.14 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master-01 localhost] and IPs [10.206.16.14 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 36.002615 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master-01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-master-01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: ee3zom.l4xeahsfqcj9uvvz [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of control-plane nodes by copying certificate authorities and service account keys on each node and then running the following as root: kubeadm join master.k8s.io:16443 --token ee3zom.l4xeahsfqcj9uvvz \ --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b \ --control-plane Then you can join any number of worker nodes by running the following on each as root: kubeadm join master.k8s.io:16443 --token ee3zom.l4xeahsfqcj9uvvz \ --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b 3.按照提示配置环境变量 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config 4.查看集群状态 [root@VM-16-14-centos manifests]# kubectl get cs NAME AGE scheduler <unknown> controller-manager <unknown> etcd-0 <unknown> [root@VM-16-14-centos manifests]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-58cc8c89f4-ch8pn 0/1 Pending 0 2m45s coredns-58cc8c89f4-qdz7t 0/1 Pending 0 2m45s etcd-k8s-master-01 1/1 Running 0 113s kube-apiserver-k8s-master-01 1/1 Running 0 99s kube-controller-manager-k8s-master-01 1/1 Running 0 98s kube-proxy-wvp9b 1/1 Running 0 2m45s kube-scheduler-k8s-master-01 1/1 Running 0 2m6s 里处于pending状态的原因是因为还没有安装网络组件 九.安装集群网络(master操作) 1.安装flannel插件 [root@VM-16-14-centos flannel]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created 2.检查 [root@VM-16-14-centos flannel]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-58cc8c89f4-ch8pn 1/1 Running 0 32m coredns-58cc8c89f4-qdz7t 1/1 Running 0 32m etcd-k8s-master-01 1/1 Running 0 31m kube-apiserver-k8s-master-01 1/1 Running 0 31m kube-controller-manager-k8s-master-01 1/1 Running 0 31m kube-flannel-ds-qljzc 1/1 Running 0 63s kube-proxy-wvp9b 1/1 Running 0 32m kube-scheduler-k8s-master-01 1/1 Running 0 32m 十.其他节点加入集群(master,node都需要加入) 1.master加入集群 1.1复制密钥和相关文件（k8s-master-01执行） 建立免登录 ssh-keygen -t rsa ssh-copy-id root@10.206.16.15 ssh-copy-id root@10.206.16.16 复制文件到k8s-master-02 ssh root@10.206.16.15 mkdir -p /etc/kubernetes/pki/etcd scp /etc/kubernetes/admin.conf root@10.206.16.15:/etc/kubernetes scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@10.206.16.15:/etc/kubernetes/pki scp /etc/kubernetes/pki/etcd/ca.* root@10.206.16.15:/etc/kubernetes/pki/etcd 复制文件到k8s-master-03 ssh root@10.206.16.16 mkdir -p /etc/kubernetes/pki/etcd scp /etc/kubernetes/admin.conf root@10.206.16.16:/etc/kubernetes scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@10.206.16.16:/etc/kubernetes/pki scp /etc/kubernetes/pki/etcd/ca.* root@10.206.16.16:/etc/kubernetes/pki/etcd 1.2 master加入集群 分别在其他两个master ,k8s-master-02,k8s-master-03上执行k8s-master-01 init后输出的join命令，如果找不到可以在k8s-master-01执行 kubeadm token create --print-join-command 在k8s-master-02上执行操作,需要带上参数--control-plane表示把master控制节点加入到集群 kubeadm join master.k8s.io:16443 --token 13dqfw.8vteayxksdn03mve --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b --control-plane mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config 在k8s-master-03上执行join命令 kubeadm join master.k8s.io:16443 --token 13dqfw.8vteayxksdn03mve --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b --control-plane mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config 1.3检查master是否加入成功 root@VM-16-14-centos flannel]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master-01 Ready master 74m v1.16.3 k8s-master-02 Ready master 18m v1.16.3 k8s-master-03 Ready master 87s v1.16.3 [root@VM-16-14-centos flannel]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-58cc8c89f4-ch8pn 1/1 Running 0 74m kube-system coredns-58cc8c89f4-qdz7t 1/1 Running 0 74m kube-system etcd-k8s-master-01 1/1 Running 0 73m kube-system etcd-k8s-master-02 1/1 Running 0 19m kube-system etcd-k8s-master-03 1/1 Running 0 93s kube-system kube-apiserver-k8s-master-01 1/1 Running 0 73m kube-system kube-apiserver-k8s-master-02 1/1 Running 0 19m kube-system kube-apiserver-k8s-master-03 1/1 Running 0 94s kube-system kube-controller-manager-k8s-master-01 1/1 Running 1 73m kube-system kube-controller-manager-k8s-master-02 1/1 Running 0 19m kube-system kube-controller-manager-k8s-master-03 1/1 Running 0 94s kube-system kube-flannel-ds-965w9 1/1 Running 0 94s kube-system kube-flannel-ds-qljzc 1/1 Running 0 42m kube-system kube-flannel-ds-vjn8d 1/1 Running 1 19m kube-system kube-proxy-6w9ch 1/1 Running 0 19m kube-system kube-proxy-p4mt8 1/1 Running 0 94s kube-system kube-proxy-wvp9b 1/1 Running 0 74m kube-system kube-scheduler-k8s-master-01 1/1 Running 1 73m kube-system kube-scheduler-k8s-master-02 1/1 Running 0 19m kube-system kube-scheduler-k8s-master-03 1/1 Running 0 94s 2.node加入到集群(分别在两个node上执行) 2.1配置: kubeadm join master.k8s.io:16443 --token hx67nu.7nlxcsvcsa8uy46o --discovery-token-ca-cert-hash sha256:4c0389dec1204d86c9721a08e2bbdb8503e0ff511b7a9584e747425d71a8f99b 2.2检查: [root@VM-16-14-centos flannel]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master-01 Ready master 80m v1.16.3 k8s-master-02 Ready master 24m v1.16.3 k8s-master-03 Ready master 7m11s v1.16.3 k8s-node-01 Ready <none> 2m58s v1.16.3 k8s-node-02 Ready <none> 101s v1.16.3 [root@VM-16-14-centos flannel]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-58cc8c89f4-ch8pn 1/1 Running 0 80m coredns-58cc8c89f4-qdz7t 1/1 Running 0 80m etcd-k8s-master-01 1/1 Running 0 79m etcd-k8s-master-02 1/1 Running 0 24m etcd-k8s-master-03 1/1 Running 0 7m20s kube-apiserver-k8s-master-01 1/1 Running 0 79m kube-apiserver-k8s-master-02 1/1 Running 0 24m kube-apiserver-k8s-master-03 1/1 Running 0 7m21s kube-controller-manager-k8s-master-01 1/1 Running 1 79m kube-controller-manager-k8s-master-02 1/1 Running 0 24m kube-controller-manager-k8s-master-03 1/1 Running 0 7m21s kube-flannel-ds-965w9 1/1 Running 0 7m21s kube-flannel-ds-nvdhl 1/1 Running 0 111s kube-flannel-ds-qljzc 1/1 Running 0 48m kube-flannel-ds-vjn8d 1/1 Running 1 24m kube-flannel-ds-z9zc2 1/1 Running 0 3m8s kube-proxy-6w9ch 1/1 Running 0 24m kube-proxy-fswvz 1/1 Running 0 111s kube-proxy-p4mt8 1/1 Running 0 7m21s kube-proxy-wvp9b 1/1 Running 0 80m kube-proxy-z27lw 1/1 Running 0 3m8s kube-scheduler-k8s-master-01 1/1 Running 1 79m kube-scheduler-k8s-master-02 1/1 Running 0 24m kube-scheduler-k8s-master-03 1/1 Running 0 7m21s 十一.部署dashboard（k8s-master-01执行） 部署最新版本v2.0.0-beta6，下载yaml cd /usr/local/kubernetes/manifests/ mkdir dashboard && cd dashboard wget -c https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta6/aio/deploy/recommended.yaml # 修改service类型为nodeport vim recommended.yaml ... kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: type: NodePort ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard ... [root@k8s-master-01 dashboard]# kubectl apply -f recommended.yaml namespace/kubernetes-dashboard created serviceaccount/kubernetes-dashboard created service/kubernetes-dashboard created secret/kubernetes-dashboard-certs created secret/kubernetes-dashboard-csrf created secret/kubernetes-dashboard-key-holder created configmap/kubernetes-dashboard-settings created role.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created deployment.apps/kubernetes-dashboard created service/dashboard-metrics-scraper created deployment.apps/dashboard-metrics-scraper created [root@k8s-master-01 dashboard]# kubectl get pods -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-76585494d8-62vp9 1/1 Running 0 6m47s kubernetes-dashboard-b65488c4-5t57x 1/1 Running 0 6m48s [root@k8s-master-01 dashboard]# kubectl get svc -n kubernetes-dashboard NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dashboard-metrics-scraper ClusterIP 10.1.207.27 <none> 8000/TCP 7m6s kubernetes-dashboard NodePort 10.1.207.168 <none> 443:30001/TCP 7m7s # 在node上通过https://nodeip:30001访问是否正常,注意在firefox浏览器执行，使用非安全模式进入 2.创建service account并绑定默认cluster-admin管理员集群角色 vim dashboard-adminuser.yaml apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard [root@VM-16-14-centos dashboard]# kubectl apply -f dashboard-adminuser.yaml serviceaccount/admin-user created clusterrolebinding.rbac.authorization.k8s.io/admin-user created 获取token,dashboard通过这个token进入系统 [root@VM-16-14-centos dashboard]# kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') Name: admin-user-token-p7wgc Namespace: kubernetes-dashboard Labels: <none> Annotations: kubernetes.io/service-account.name: admin-user kubernetes.io/service-account.uid: 0e9f5406-3c26-4141-a233-ff4eaa841401 Type: kubernetes.io/service-account-token Data ==== namespace: 20 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6InBDdkJ1VWZFZENYbEd3ZGVrc3FldlhXWG94QU0ySjN1M1Y4ZVRJOUZPd1UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXA3d2djIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwZTlmNTQwNi0zYzI2LTQxNDEtYTIzMy1mZjRlYWE4NDE0MDEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.jcCnl8hHWrZcFtd5H17gJdoaDiFsPUos_4oNYQexiXSjFDgy972Bk1qgYV-zHZhu7o_UZyESMRLTlzRFl3W5Eqbhq9fouD0j0DH_qnTGewNTEuByQj5n6uPLloPG5VNCOs1y3TINVj8LdG5q_n6DWfozfn76eNhU9eAnSJZVZ97dGKy_LDykpM9QtJQQkpaF9jSnDPCeoSnSd_1ud1FoQlNS3PAenB54khOmL5gbD6Pf4uJOVUjzxoHk_--gKDW7juVAsaDPbbGftuiM1mIfQ3K02VoNMiG1VB2hlzJ5kWeUn7wpqZpmngzrqBtVj5DJWSpnHAZZef_FFCakKMp5TA ca.crt: 1025 bytes 十二.部署ingress控制器 mandatory.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: #为了让pod获取真实ip compute-full-forwarded-for: 'true' use-forwarded-headers: 'true' --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: DaemonSet metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true containers: - name: nginx-ingress-controller image: lizhenliang/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 --- 十三.部署metric(v0.3.6) mandatory.yaml ## ServiceAccount apiVersion: v1 kind: ServiceAccount metadata: name: metrics-server namespace: kube-system --- ## ClusterRole aggregated-metrics-reader apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:aggregated-metrics-reader labels: rbac.authorization.k8s.io/aggregate-to-view: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-admin: "true" rules: - apiGroups: ["metrics.k8s.io"] resources: ["pods","nodes"] verbs: ["get","list","watch"] --- ## ClusterRole metrics-server apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:metrics-server rules: - apiGroups: [""] resources: ["pods","nodes","nodes/stats","namespaces","configmaps"] verbs: ["get","list","watch"] --- ## ClusterRoleBinding auth-delegator apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- ## RoleBinding metrics-server-auth-reader apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- ## ClusterRoleBinding system:metrics-server apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system ## APIService --- apiVersion: apiregistration.k8s.io/v1beta1 kind: APIService metadata: name: v1beta1.metrics.k8s.io spec: service: name: metrics-server namespace: kube-system group: metrics.k8s.io version: v1beta1 insecureSkipTLSVerify: true groupPriorityMinimum: 100 versionPriority: 100 ## Service --- apiVersion: v1 kind: Service metadata: name: metrics-server namespace: kube-system labels: kubernetes.io/name: "Metrics-server" kubernetes.io/cluster-service: "true" spec: selector: k8s-app: metrics-server ports: - port: 443 targetPort: 4443 --- ## Deployment apiVersion: apps/v1 kind: Deployment metadata: name: metrics-server namespace: kube-system labels: k8s-app: metrics-server spec: selector: matchLabels: k8s-app: metrics-server template: metadata: name: metrics-server labels: k8s-app: metrics-server spec: hostNetwork: true serviceAccountName: metrics-server containers: - name: metrics-server ## 修改镜像源地址 image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6 imagePullPolicy: IfNotPresent args: - --cert-dir=/tmp - --secure-port=4443 - --kubelet-insecure-tls ## 增加 - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname ## 增加 ports: - name: main-port containerPort: 4443 protocol: TCP securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 resources: limits: memory: 1Gi cpu: 1000m requests: memory: 1Gi cpu: 1000m volumeMounts: - name: tmp-dir mountPath: /tmp - name: localtime readOnly: true mountPath: /etc/localtime volumes: - name: tmp-dir emptyDir: {} - name: localtime hostPath: type: File path: /etc/localtime nodeSelector: kubernetes.io/os: linux kubernetes.io/arch: "amd64" 十四.安装kubernetes dns缓存，避免延迟问题 kubectlapply-fhttps://github.com/feiskyer/kubernetes-handbook/raw/master/examples/nodelocaldns/nodelocaldns-kubenet.yaml (参考:https://mp.weixin.qq.com/s/t7nt87JPJnWEVCNBS-sBpw) 十五. 集群的扩容，缩容 1.集群扩容 默认情况下加入集群的token是24小时过期，24小时后如果是想要新的node加入到集群，需要重新生成一个token，命令如下# 显示获取token列表$ kubeadm token list# 生成新的token$ kubeadm token create 除token外，join命令还需要一个sha256的值，通过以下方法计算openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 用上面输出的token和sha256的值或者是利用kubeadm token create --print-join-command拼接join命令即可 2.集群的缩容 kubectl cordon <node name> #设置为不可调度 kubectl drain <node name> --delete-local-data --force --ignore-daemonsets 驱逐节点上的pod kubectl delete node <node name> 3.初始化重新加入 需要把原来的配置清空 kubeadm reset systemctl stop kubelet systemctl stop docker rm -rf /var/lib/cni/ rm -rf /var/lib/kubelet/* rm -rf /etc/cni/ ifconfig cni0 down ifconfig flannel.1 down ifconfig docker0 down ip link delete cni0 ip link delete flannel.1 systemctl start docker
