使用Kubespray 部署kubernetes 高可用集群
Requirements Ansible v2.4 及以上版本，安装python-netaddr到运行Ansible commands的机器 Jinja 2.9 及以上版本，运行Ansible Playbooks 目标servers 必须可以访问外网，可以pull docker images 目标servers 配置允许 IPv4 forwarding 将公钥复制到所有机器 关闭防火墙 请提前安装好docker,因为k8s不支持最新版docker,具体适配哪些docker版本，请看k8s上的changelog 0、环境 主机名 IP master1 172.16.105.21 master2 172.16.105.22 master3 172.16.105.23 node1 172.16.105.24 node2 172.16.105.25 ansible-client 172.16.105.20 1、安装ansible和依赖 在172.16.105.20安装ansible # 安装 python 及 epel yum install -y epel-release python-pip python34 python34-pip # 安装 ansible yum install -y ansible pip install netaddr pip install --upgrade jinja2 2、建立公私钥，分发各服务器 在ansible-clinet机器生成免密密钥对 ssh-keygen -t rsa -P '' 将生成的公钥(id_rsa.pub)传到其他节点，这样ansible-client可以免密登陆其他机器 cat id_rsa.pub >> ~/.ssh/authorized_keys 3、下载kuberspay源码 cd /usr/local/src/ wget https://github.com/kubernetes-incubator/kubespray/archive/v2.3.0.tar.gz 本版本所包含的组件版本 Kubernetes v1.8.1 Docker 1.13.1 etcd v3.2.4 Rkt v1.21.0 (optional) Calico v2.5.0 Weave 2.0.4 Flannel v0.8.0 3.1 禁用docker yum仓和docker安装 vim roles/docker/tasks/main.yml --- - name: gather os specific variables include_vars: "{{ item }}" with_first_found: - files: - "{{ ansible_distribution|lower }}-{{ ansible_distribution_version|lower|replace('/', '_') }}.yml" - "{{ ansible_distribution|lower }}-{{ ansible_distribution_release }}.yml" - "{{ ansible_distribution|lower }}-{{ ansible_distribution_major_version|lower|replace('/', '_') }}.yml" - "{{ ansible_distribution|lower }}.yml" - "{{ ansible_os_family|lower }}.yml" - defaults.yml paths: - ../vars skip: true tags: - facts - include: set_facts_dns.yml when: dns_mode != 'none' and resolvconf_mode == 'docker_dns' tags: - facts - name: check for minimum kernel version fail: msg: > docker requires a minimum kernel version of {{ docker_kernel_min_version }} on {{ ansible_distribution }}-{{ ansible_distribution_version }} when: (not ansible_os_family in ["CoreOS", "Container Linux by CoreOS"]) and (ansible_kernel|version_compare(docker_kernel_min_version, "<")) tags: - facts #禁用docker仓库，已经使用清华源 #- name: ensure docker repository public key is installed # action: "{{ docker_repo_key_info.pkg_key }}" # args: # id: "{{item}}" # keyserver: "{{docker_repo_key_info.keyserver}}" # state: present # register: keyserver_task_result # until: keyserver_task_result|succeeded # retries: 4 # delay: "{{ retry_stagger | random + 3 }}" # environment: "{{ proxy_env }}" # with_items: "{{ docker_repo_key_info.repo_keys }}" # when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) #- name: ensure docker repository is enabled # action: "{{ docker_repo_info.pkg_repo }}" # args: # repo: "{{item}}" # state: present # with_items: "{{ docker_repo_info.repos }}" # when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and(docker_repo_info.repos|length > 0) #- name: Configure docker repository on RedHat/CentOS # template: # src: "rh_docker.repo.j2" # dest: "/etc/yum.repos.d/docker.repo" # when: ansible_distribution in ["CentOS","RedHat"] and not is_atomic #- name: ensure docker packages are installed # action: "{{ docker_package_info.pkg_mgr }}" # args: # pkg: "{{item.name}}" # force: "{{item.force|default(omit)}}" # state: present # register: docker_task_result # until: docker_task_result|succeeded # retries: 4 # delay: "{{ retry_stagger | random + 3 }}" # environment: "{{ proxy_env }}" # with_items: "{{ docker_package_info.pkgs }}" # notify: restart docker # when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and (docker_package_info.pkgs|length > 0) #对于docker的版本检测进行了保留 - name: check minimum docker version for docker_dns mode. You need at least docker version >= 1.12 for resolvconf_mode=docker_dns command: "docker version -f '{{ '{{' }}.Client.Version{{ '}}' }}'" register: docker_version failed_when: docker_version.stdout|version_compare('1.12', '<') changed_when: false when: dns_mode != 'none' and resolvconf_mode == 'docker_dns' #对于docker的systemd配置，可以根据自己需求修改，但是注意会覆盖原来的 - name: Set docker systemd config include: systemd.yml - name: ensure docker service is started and enabled service: name: "{{ item }}" enabled: yes state: started with_items: - docker 4、替换镜像 因为长城的原因，需要的镜像在安装的时候无法获取，所以需要改下源码，下载自己私有仓的镜像 脚本内容如下： gcr_image_files=( ./kubespray/roles/download/defaults/main.yml ./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2 ./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml ) for file in ${gcr_image_files[@]} ; do sed -i 's/gcr.io/docker.emarbox.com/g' $file done 镜像列表，最好提前下载到私有仓,毕竟下载会很慢 gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 gcr.io/google_containers/pause-amd64:3.0 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3 nginx:1.11.4-alpine busybox:latest quay.io/coreos/hyperkube:v1.8.1_coreos.0 quay.io/coreos/etcd:v3.2.4 quay.io/calico/ctl:v1.5.0 quay.io/calico/node:v2.5.0 quay.io/calico/routereflector:v0.4.0 quay.io/calico/cni:v1.10.0 5、配置文件内容 可以对auth的密码进行修改，网络插件默认calico，可替换成weave或flannel，还可以配置是否安装helm和efk,以及修改安装路径 more kubespray/kubespray-2.3.0/inventory/group_vars/k8s-cluster.yml #6、生成自己的集群配置 因为kubespray自带的python脚本是Python3 ，所以要安装Python3 yum install -y python-pip python34 python34-pip # 定义集群IP IP=( 172.16.105.21 172.16.105.22 172.16.105.23 ) # 利用kubespray自带的python脚本生成配置 CONFIG_FILE=./kubespray/inventory/inventory.cfg python3 ./kubespray/contrib/inventory_builder/inventory.py ${IP[*]} 查看配置 cat ./kubespray/inventory/inventory.cfg [all] node1 ansible_host=172.16.105.21 ip=172.16.105.21 node2 ansible_host=172.16.105.22 ip=172.16.105.22 node3 ansible_host=172.16.105.23 ip=172.16.105.23 [kube-master] node1 node2 node3 [kube-node] node1 node2 node3 [etcd] node1 node2 node3 [k8s-cluster:children] kube-node kube-master [calico-rr] [vault] node1 node2 node3 7、安装集群 ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v 镜像地址 kubespray/roles/download/tasks/download_container.yml 8、问题 8.1 从1.8 开始，kubelet 会检测机器是否有swap,如果启用swap，kubelet会无法启动，需要手动添加参数。 去如下目录，修改kubelet参数 /usr/local/src/kubespray/kubespray-2.3.0/roles/kubernetes/node/defaults ### fail with swap on (default true) kubelet_fail_swap_on: false 8.2 注意机器主机名，要符合k8s的规范 9、安装失败如何清理 rm -rf /etc/kubernetes/ rm -rf /var/lib/kubelet rm -rf /var/lib/etcd rm -rf /usr/local/bin/kubectl rm -rf /etc/systemd/system/calico-node.service rm -rf /etc/systemd/system/kubelet.service systemctl stop etcd.service systemctl disable etcd.service systemctl stop calico-node.service systemctl disable calico-node.service docker stop $(docker ps -q) docker rm $(docker ps -a -q) systemctl restart docker 10、安装完成 [root@node2 .kube]# kubectl get nodes NAME STATUS ROLES AGE VERSION node1 Ready master,node 9m v1.8.1+coreos.0 node2 Ready master,node 9m v1.8.1+coreos.0 node3 Ready master,node 9m v1.8.1+coreos.0 11、扩展集群node 把需要添加的node 写入配置文件，然后执行ansible 以添加node4举例 修改 inventory.cfg [all] node1 ansible_host=172.16.105.21 ip=172.16.105.21 node2 ansible_host=172.16.105.22 ip=172.16.105.22 node3 ansible_host=172.16.105.23 ip=172.16.105.23 node4 ansible_host=172.16.105.37 ip=172.16.105.37 [kube-master] node1 node2 node3 [kube-node] node1 node2 node3 node4 [etcd] node1 node2 node3 [k8s-cluster:children] kube-node kube-master [calico-rr] [vault] node1 node2 node3 ansible-playbook -i inventory/inventory.cfg scale.yml -b -v \ --private-key=~/.ssh/private_key 后感 了解ansible，也就能自己随便搞kubespray,这个东西比较透明，不像kubeadm,封装很严，不知道具体流程，出问题也不知道怎么处理。 本文转自银狐博客51CTO博客，原文链接http://blog.51cto.com/foxhound/2044766如需转载请自行联系原作者 战狐
