kubernetes 1.9.0 kubeadm方式安装

1、安装rpm包

yum localinstall -y kubeadm-1.9.0-0.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm kubelet-1.9.0-0.x86_64.rpm kubernetes-cni-0.6.0-0.x86_64.rpm

2、修改内核参数

修改 /etc/sysctl.conf，添加以下内容

net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1

修改后，及时生效

sysctl -p

3、修改kubelet配置文件

kubelet和docker 的cgroup driver 有2种方式:cgroupfs和systemd.注意保持 2个应用的driver保持一致。

3.1 docker是cgroupfs的，修改kubelet

vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

#修改systemd为cgroupfs
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

#新加一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=foxchan/google_containers/pause-amd64:3.0"

修改完成后

systemctl daemon-reload

3.2 docker是systemd的，kubelet无需修改

docker 启动命令添加如下内容，可以修改cgroup driver为systemd

--exec-opt native.cgroupdriver=systemd

4、安装 kubernetes

kubeadm init --kubernetes-version=1.9.0 --token-ttl 0

参数说明

• --kubernetes-version 最好指定版本，否则会请求 https://storage.googleapis.com/kubernetes-release/release/stable-1.9.txt ，如果没"翻""墙"，就超时报错
• --token-ttl 默认的token有效期24小时， 设置为0表示永不过期

因为镜像要"翻""墙"，可以预先下载镜像，然后更改tag

镜像列表

• gcr.io/google_containers/kube-apiserver-amd64:v1.9.0
• gcr.io/google_containers/kube-controller-manager-amd64:v1.9.0
• gcr.io/google_containers/kube-scheduler-amd64:v1.9.0
• gcr.io/google_containers/pause-amd64:3.0
• gcr.io/google_containers/etcd-amd64:3.1.10
• gcr.io/google_containers/kube-proxy-amd64:v1.9.0
• gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
• gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7
• gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7
• quay.io/calico/node:v3.0.1
• quay.io/coreos/etcd:v3.2.4
• quay.io/calico/cni:v2.0.0
• quay.io/calico/kube-controllers:v1.0.1

如果下载自己的或者dockerhub的镜像。可以利用脚本，批量替换镜像imagename

docker images | sed 's/foxchan/gcr.io\/google_containers/'| awk '{print "docker tag "$3" "$1":"$2}'

安装信息

[root@kvm-gs242024 ~]# kubeadm init --kubernetes-version=1.9.0 --token-ttl 0 --ignore-preflight-errors=all
[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kvm-gs024 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.24]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[apiclient] All control plane components are healthy after 78.502690 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node kvm-gs242024 as master by adding a label and a taint
[markmaster] Master kvm-gs242024 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 1ac970.704ce2d03cc45382
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 1ac970.704ce2d03cc45382 192.168.0.24:6443 --discovery-token-ca-cert-hash sha256:f70f07be83a7b2af2c41752b00def4389e3019006b3be643fe1ccf1c53368043

安装后提示如何管理集群和添加node

token要记得保存，当前版本 token 无法通过命令找回，否则无法添加node

master节点验证

[root@kvm-master ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}

node节点验证

[root@kvm-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
kvm-node1 Ready 1h v1.9.0
kvm-master NotReady master 18h v1.9.0
kvm-node2 Ready 6m v1.9.0

命令行验证

curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://k8smaster:6443

管理集群使用kubectl的报错

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

https://github.com/kubernetes/kubernetes/issues/48378

解决方式： export KUBECONFIG=/etc/kubernetes/kubelet.conf 这是普通用户，没有权限，会有报错 Error from server (Forbidden): daemonsets.extensions is forbidden: User "system:node:kvm-master" cannot list daemonsets.extensions in the namespace "default"
export KUBECONFIG=/etc/kubernetes/admin.conf 管理用户

###由于安全原因，默认情况下pod不会被schedule到master节点上，可以通过下面命令解除这种限制：

kubectl taint nodes --all node-role.kubernetes.io/master-

calico网络安装

先装rbac
kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/rbac.yaml

下载calico.yaml
https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml

如果使用calico自带etcd，注意保证calico etcd的稳定
如果使用自有etcd集群，需要修改yaml里的etcd_endpoints

默认安装时，如果节点是多网卡会报错，导致网络不成功，calico 总是随机绑定网卡，导致注册失败

calico 报错日志

Skipping datastore connection test
IPv4 address 10.96.0.1 discovered on interface kube-ipvs0
No AS number configured on node resource, using global value

需要修改calico.yaml,注意顺序

        - name: IP
          value: "autodetect"
        - name: IP_AUTODETECTION_METHOD
          value: "can-reach=192.168.1.1"

IP_AUTODETECTION_METHOD 参数说明

使用通过ip访问的interface
can-reach=192.168.1.1

使用通过域名访问的interface
can-reach=www.baidu.com

使用指定的interface
interface=ethx

安装dashboard

下载yaml

wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

下载完成后修改imagename

name: kubernetes-dashboard
        image: foxchan/google_containers/kubernetes-dashboard-amd64:v1.8.0

kubectl proxy --address=masterip --accept-hosts='^*$'

访问

http://masterip:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

安装heapster

下载yaml

wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml

下载完成后，修改image为自己的

#grafana.yaml
      - name: grafana
        image: gcr.io/google_containers/heapster-grafana-amd64:v4.4.3

#heapster.yaml
      - name: heapster
        image: gcr.io/google_containers/heapster-amd64:v1.4.2

#influxdb.yaml
      - name: influxdb
        image: gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3

下载镜像

docker pull foxchan/heapster-grafana-amd64:v4.4.3
docker pull foxchan/heapster-amd64:v1.4.2
docker pull foxchan/heapster-influxdb-amd64:v1.3.3

如果成功，dashboard页面 会有图形

将skydns替换为coredns

下载脚本

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

利用脚本 修改coredns.yaml.sed配置文件

deploy.sh clusterip|kubectl apply

我的clusterip是10.96.0.0/12

./deploy.sh 10.96.0.0/12|kubectl apply -f

请确保coredns正常运行，然后就可以删除skydns

kubectl delete --namespace=kube-system deployment kube-dns

另外 kubeadm 也添加了替换dns功能，需要kubeadm 1.9版本

参数如下：

kubeadm init --feature-gates=CoreDNS=true

总结

总的来说1.9 和1.8 没什么大的变化，以下是我关注的

• Bootstrap Tokens 从alpha 变为beta。
• etcd 版本从3.0 升级到3.1，因为etcd 版本数据是不兼容的，如果要升级etcd，需要提前备份数据。
• kube-proxy 的ipvs 从alpha变为beta
• coredns 比较有前景，可以持续关注