离线安装k8s 1.9.0

说明
本文参考 https://segmentfault.com/a/1190000012755243。在前 文基础上整理、增加说明，避坑。

踩过的坑： 安装k8s 1.9.0 实践：问题集锦

环境说明

环境信息（采用一个master节点+两个node节点) 192.168.1.137 tensorflow0 node 192.168.1.138 tensorflow1 master 192.168.1.139 tensorflow2 node 

操作系统版本：

[root@tensorflow1 ~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) 

内核版本： 

[root@tensorflow1 ~]# cat /proc/version Linux version 3.10.0-693.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017 

软件版本：

kubernetes v1.9.0 docker：17.03.2-ce kubeadm:v1.9.0 kube-apiserver:v1.9.0 kube-controller-manager:v1.9.0 kube-scheduler:v1.9.0 k8s-dns-sidecar:1.14.7 k8s-dns-kube-dns:1.14.7 k8s-dns-dnsmasq-nanny:1.14.7 kube-proxy:v1.9.0 etcd:3.1.10 pause :3.0 flannel:v0.9.1 kubernetes-dashboard:v1.8.1 

采用kubeadm安装
kubeadm为kubernetes官方推荐的自动化部署工具，他将kubernetes的组件以pod的形式部署在master和node节点上，并自动完成证书认证等操作。
因为kubeadm默认要从google的镜像仓库下载镜像，但目前国内无法访问google镜像仓库，所以这里我提交将镜像下好了，只需要将离线包的镜像导入到节点中就可以了。
开始安装
下载
链接: https://pan.baidu.com/s/1c2O1gIW 密码: 9s92

比对md5解压离线包

MD5 (k8s_images.tar.bz2) = b60ad6a638eda472b8ddcfa9006315ee 

解压下载下来的离线包

tar -xjvf k8s_images.tar.bz2

所有节点操作

环境设置

绑定hosts
将节点ip和host写入hosts文件

[root@tensorflow1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 

192.168.1.137 tensorflow0 192.168.1.138 tensorflow1 192.168.1.139 tensorflow2 

关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

关闭selinux

修改vi /etc/selinux/config 文件，将SELINUX改为disabled SELINUX=disabled setenforce 0 关闭swap swapoff -a 设置永久关闭swap 

修改/etc/fstab中内容，将swap那一行用#注释掉。

配置系统路由参数,防止kubeadm报路由警告

echo " net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 " >> /etc/sysctl.conf sysctl -p 

安装docker
安装docker-ce17.03 (kubeadmv1.9最大支持docker-ce17.03)

rpm -ihv docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm rpm -ivh docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm 

启动docker-ce

systemctl start docker && systemctl enable docker 

检查docker服务

systemctl status docker active (running)则正常 

安装k8s
导入镜像

docker load <etcd-amd64_v3.1.10.tar docker load <flannel\:v0.9.1-amd64.tar docker load <k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar docker load <k8s-dns-kube-dns-amd64_1.14.7.tar docker load <k8s-dns-sidecar-amd64_1.14.7.tar docker load <kube-apiserver-amd64_v1.9.0.tar docker load <kube-controller-manager-amd64_v1.9.0.tar docker load <kube-scheduler-amd64_v1.9.0.tar docker load <kube-proxy-amd64_v1.9.0.tar docker load <pause-amd64_3.0.tar docker load <kubernetes-dashboard_v1.8.1.tar 

注意kubernetes-dashboard_v1.8.1.tar与其他包不在同一个目录下，在上一级目录中

安装kubelet kubeadm kubectl包

rpm -ivh socat-1.7.3.2-2.el7.x86_64.rpm rpm -ivh kubernetes-cni-0.6.0-0.x86_64.rpm kubelet-1.9.9-9.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm rpm -ivh kubectl-1.9.0-0.x86_64.rpm rpm -ivh kubeadm-1.9.0-0.x86_64.rpm 

修改kublet配置文件

查看docker cgroup driver:

docker info|grep Cgroup 

有systemd和cgroupfs两种，把kubelet service配置改成与docker一致

vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 

修改 Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"。修改 systemd为cgroupfs

启动kubelet

systemctl enable kubelet && sudo systemctl start kubelet 

检查kubelet服务

systemctl status kubelet 

kubelet启动后 ，不停重启、ca文件不存在是正常现象，在后续步骤 kubeadm init执行后会生成ca文件，就会正常运行。

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally. 

master节点操作
开始初始化master
kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16
kubernetes默认支持多重网络插件如flannel、weave、calico，这里使用flanne，就必须要设置--pod-network-cidr参数，10.244.0.0/16是kube-flannel.yml里面配置的默认网段，如果需要修改的话，需要把kubeadm init的--pod-network-cidr参数和后面的kube-flannel.yml里面修改成一样的网段就可以了。

将kubeadm join xxx保存下来，等下node节点加入集群需要使用

eg: kubeadm join --token 5ce44e.47b6dc4e4b66980f 192.168.1.138:6443 --discovery-token-ca-cert-hash sha256:9d7eac82d66744405c783de5403e1f2bb7191b4c1b350d721b7b8570c62ff83a 

如果忘记了，可以在master上通过kubeadmin token list得到

kubeadmin token list 

默认token 24小时就会过期，后续的机器要加入集群需要重新生成token

kubeadm token create sha256获取方式 master节点执行: openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 

按照上面提示，此时root用户还不能使用kubelet控制集群需要，配置下环境变量

对于非root用户

mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config 对于root用户 

可以直接放到~/.bash_profile

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile source一下环境变量 source ~/.bash_profile kubectl version测试 kubectl version Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} 

安装网络，可以使用flannel、calico、weave、macvlan这里我们用flannel。直接使用离线包里面的。

若要修改网段，需要kubeadm --pod-network-cidr=和这里同步

vi kube-flannel.yml 修改network项 "Network": "10.244.0.0/16", 

执行

kubectl create -f kube-flannel.yml 

node节点操作
使用刚刚kubeadm后的kubeadm join

kubeadm join --token 5ce44e.47b6dc4e4b66980f 192.168.1.138:6443 --discovery-token-ca-cert-hash sha256:9d7eac82d66744405c783de5403e1f2bb7191b4c1b350d721b7b8570c62ff83a 

在master节点上确认一下

[root@tensorflow1 hadoop]# kubectl get nodes NAME STATUS ROLES AGE VERSION tensorflow0 Ready <none> 1d v1.9.0 tensorflow1 Ready master 1d v1.9.0 tensorflow2 Ready <none> 1d v1.9.0 

kubernetes会在每个node节点创建flannel和kube-proxy的pod

[root@tensorflow1 hadoop]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system etcd-tensorflow1 1/1 Running 0 1d kube-system kube-apiserver-tensorflow1 1/1 Running 0 1d kube-system kube-controller-manager-tensorflow1 1/1 Running 0 1d kube-system kube-dns-6f4fd4bdf-59ttf 3/3 Running 0 1d kube-system kube-flannel-ds-fb75p 1/1 Running 0 1d kube-system kube-flannel-ds-ppm2t 1/1 Running 0 1d kube-system kube-flannel-ds-w54wh 1/1 Running 0 1d kube-system kube-proxy-4lftj 1/1 Running 0 1d kube-system kube-proxy-cj4st 1/1 Running 0 1d kube-system kube-proxy-kd7vb 1/1 Running 0 1d kube-system kube-scheduler-tensorflow1 1/1 Running 0 1d 

至此kubernetes基本集群安装完成。

--后续补充部署dashboard内容--

安装nvidia-gpu组件
这里是为了给容器使用gpu，需要安装组件，否则不用安装。

编辑/etc/docker/daemon.json

cat /etc/docker/daemon.json { "runtimes": { "nvidia": { "path": "/usr/bin/nvidia-container-runtime", "runtimeArgs": [] } }, "default-runtime": "nvidia" } 

重启docker

systemctl restart docker 

编辑kubelet配置文件

资源则需要增加一行 Environment="KUBELET_EXTRA_ARGS=--feature-gates=DevicePlugins=true"  注意要加在 ExecStart= 之前

重启kubelet

systemctl daemon-reload && systemctl restart kubelet 

根据gpu型号下载相应的gpu插件镜像，挂载镜像

docker load < # docker images REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE nvidia/k8s-device-plugin 1.9 3325c3b04513 2 weeks ago 63 MB 

通过yaml文件启动

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml 文件内容如下： [root@tensorflow1 tf_gpu]# cat nvidia-device-plugin.yml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: nvidia-device-plugin-daemonset namespace: kube-system spec: template: metadata: # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler # reserves resources for critical add-on pods so that they can be rescheduled after # a failure. This annotation works in tandem with the toleration below. annotations: scheduler.alpha.kubernetes.io/critical-pod: "" labels: name: nvidia-device-plugin-ds spec: tolerations: # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode. # This, along with the annotation above marks this pod as a critical add-on. - key: CriticalAddonsOnly operator: Exists containers: - image: nvidia/k8s-device-plugin:1.9 name: nvidia-device-plugin-ctr securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins volumes: - name: device-plugin hostPath: path: /var/lib/kubelet/device-plugins systemctl daemon-reload && systemctl restart kubelet 

本文转自CSDN-离线安装k8s 1.9.0