Requirements
- Ansible v2.4 及以上版本，安装python-netaddr到运行Ansible commands的机器
- Jinja 2.9 及以上版本，运行Ansible Playbooks
- 目标servers 必须可以访问外网，可以pull docker images
- 目标servers 配置允许 IPv4 forwarding
- 将公钥复制到所有机器
- 关闭防火墙
- 请提前安装好docker,因为k8s不支持最新版docker,具体适配哪些docker版本，请看k8s上的changelog
0、环境
| 主机名 |
IP |
| master1 |
172.16.105.21 |
| master2 |
172.16.105.22 |
| master3 |
172.16.105.23 |
| node1 |
172.16.105.24 |
| node2 |
172.16.105.25 |
| ansible-client |
172.16.105.20 |
1、安装ansible和依赖
在172.16.105.20安装ansible
# 安装 python 及 epel
yum install -y epel-release python-pip python34 python34-pip
# 安装 ansible
yum install -y ansible
pip install netaddr
pip install --upgrade jinja2
2、建立公私钥，分发各服务器
在ansible-clinet机器生成免密密钥对
ssh-keygen -t rsa -P ''
将生成的公钥(id_rsa.pub)传到其他节点，这样ansible-client可以免密登陆其他机器
cat id_rsa.pub >> ~/.ssh/authorized_keys
3、下载kuberspay源码
cd /usr/local/src/
wget https://github.com/kubernetes-incubator/kubespray/archive/v2.3.0.tar.gz
本版本所包含的组件版本
- Kubernetes v1.8.1
- Docker 1.13.1
- etcd v3.2.4
- Rkt v1.21.0 (optional)
- Calico v2.5.0
- Weave 2.0.4
- Flannel v0.8.0
3.1 禁用docker yum仓和docker安装
vim roles/docker/tasks/main.yml
---
- name: gather os specific variables
include_vars: "{{ item }}"
with_first_found:
- files:
- "{{ ansible_distribution|lower }}-{{ ansible_distribution_version|lower|replace('/', '_') }}.yml"
- "{{ ansible_distribution|lower }}-{{ ansible_distribution_release }}.yml"
- "{{ ansible_distribution|lower }}-{{ ansible_distribution_major_version|lower|replace('/', '_') }}.yml"
- "{{ ansible_distribution|lower }}.yml"
- "{{ ansible_os_family|lower }}.yml"
- defaults.yml
paths:
- ../vars
skip: true
tags:
- facts
- include: set_facts_dns.yml
when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'
tags:
- facts
- name: check for minimum kernel version
fail:
msg: >
docker requires a minimum kernel version of
{{ docker_kernel_min_version }} on
{{ ansible_distribution }}-{{ ansible_distribution_version }}
when: (not ansible_os_family in ["CoreOS", "Container Linux by CoreOS"]) and (ansible_kernel|version_compare(docker_kernel_min_version, "<"))
tags:
- facts
#禁用docker仓库，已经使用清华源
#- name: ensure docker repository public key is installed
# action: "{{ docker_repo_key_info.pkg_key }}"
# args:
# id: "{{item}}"
# keyserver: "{{docker_repo_key_info.keyserver}}"
# state: present
# register: keyserver_task_result
# until: keyserver_task_result|succeeded
# retries: 4
# delay: "{{ retry_stagger | random + 3 }}"
# environment: "{{ proxy_env }}"
# with_items: "{{ docker_repo_key_info.repo_keys }}"
# when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic)
#- name: ensure docker repository is enabled
# action: "{{ docker_repo_info.pkg_repo }}"
# args:
# repo: "{{item}}"
# state: present
# with_items: "{{ docker_repo_info.repos }}"
# when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and(docker_repo_info.repos|length > 0)
#- name: Configure docker repository on RedHat/CentOS
# template:
# src: "rh_docker.repo.j2"
# dest: "/etc/yum.repos.d/docker.repo"
# when: ansible_distribution in ["CentOS","RedHat"] and not is_atomic
#- name: ensure docker packages are installed
# action: "{{ docker_package_info.pkg_mgr }}"
# args:
# pkg: "{{item.name}}"
# force: "{{item.force|default(omit)}}"
# state: present
# register: docker_task_result
# until: docker_task_result|succeeded
# retries: 4
# delay: "{{ retry_stagger | random + 3 }}"
# environment: "{{ proxy_env }}"
# with_items: "{{ docker_package_info.pkgs }}"
# notify: restart docker
# when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and (docker_package_info.pkgs|length > 0)
#对于docker的版本检测进行了保留
- name: check minimum docker version for docker_dns mode. You need at least docker version >= 1.12 for resolvconf_mode=docker_dns
command: "docker version -f '{{ '{{' }}.Client.Version{{ '}}' }}'"
register: docker_version
failed_when: docker_version.stdout|version_compare('1.12', '<')
changed_when: false
when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'
#对于docker的systemd配置，可以根据自己需求修改，但是注意会覆盖原来的
- name: Set docker systemd config
include: systemd.yml
- name: ensure docker service is started and enabled
service:
name: "{{ item }}"
enabled: yes
state: started
with_items:
- docker
4、替换镜像
因为长城的原因，需要的镜像在安装的时候无法获取，所以需要改下源码，下载自己私有仓的镜像
脚本内容如下：
gcr_image_files=(
./kubespray/roles/download/defaults/main.yml
./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml
)
for file in ${gcr_image_files[@]} ; do
sed -i 's/gcr.io/docker.emarbox.com/g' $file
done
镜像列表，最好提前下载到私有仓,毕竟下载会很慢
gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3
nginx:1.11.4-alpine
busybox:latest
quay.io/coreos/hyperkube:v1.8.1_coreos.0
quay.io/coreos/etcd:v3.2.4
quay.io/calico/ctl:v1.5.0
quay.io/calico/node:v2.5.0
quay.io/calico/routereflector:v0.4.0
quay.io/calico/cni:v1.10.0
5、配置文件内容
可以对auth的密码进行修改，网络插件默认calico，可替换成weave或flannel，还可以配置是否安装helm和efk,以及修改安装路径
more kubespray/kubespray-2.3.0/inventory/group_vars/k8s-cluster.yml
#6、生成自己的集群配置
因为kubespray自带的python脚本是Python3 ，所以要安装Python3
yum install -y python-pip python34 python34-pip
IP=(
172.16.105.21
172.16.105.22
172.16.105.23
)
CONFIG_FILE=./kubespray/inventory/inventory.cfg python3 ./kubespray/contrib/inventory_builder/inventory.py ${IP[*]}
查看配置
cat ./kubespray/inventory/inventory.cfg
[all]
node1 ansible_host=172.16.105.21 ip=172.16.105.21
node2 ansible_host=172.16.105.22 ip=172.16.105.22
node3 ansible_host=172.16.105.23 ip=172.16.105.23
[kube-master]
node1
node2
node3
[kube-node]
node1
node2
node3
[etcd]
node1
node2
node3
[k8s-cluster:children]
kube-node
kube-master
[calico-rr]
[vault]
node1
node2
node3
7、安装集群
ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v
镜像地址
kubespray/roles/download/tasks/download_container.yml
8、问题
8.1
从1.8 开始，kubelet 会检测机器是否有swap,如果启用swap，kubelet会无法启动，需要手动添加参数。
去如下目录，修改kubelet参数
/usr/local/src/kubespray/kubespray-2.3.0/roles/kubernetes/node/defaults
kubelet_fail_swap_on: false
8.2
注意机器主机名，要符合k8s的规范
9、安装失败如何清理
rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd
rm -rf /usr/local/bin/kubectl
rm -rf /etc/systemd/system/calico-node.service
rm -rf /etc/systemd/system/kubelet.service
systemctl stop etcd.service
systemctl disable etcd.service
systemctl stop calico-node.service
systemctl disable calico-node.service
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)
systemctl restart docker
10、安装完成
[root@node2 .kube]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master,node 9m v1.8.1+coreos.0
node2 Ready master,node 9m v1.8.1+coreos.0
node3 Ready master,node 9m v1.8.1+coreos.0
11、扩展集群node
把需要添加的node 写入配置文件，然后执行ansible
以添加node4举例
修改 inventory.cfg
[all]
node1 ansible_host=172.16.105.21 ip=172.16.105.21
node2 ansible_host=172.16.105.22 ip=172.16.105.22
node3 ansible_host=172.16.105.23 ip=172.16.105.23
node4 ansible_host=172.16.105.37 ip=172.16.105.37
[kube-master]
node1
node2
node3
[kube-node]
node1
node2
node3
node4
[etcd]
node1
node2
node3
[k8s-cluster:children]
kube-node
kube-master
[calico-rr]
[vault]
node1
node2
node3
ansible-playbook -i inventory/inventory.cfg scale.yml -b -v \
--private-key=~/.ssh/private_key
后感
了解ansible，也就能自己随便搞kubespray,这个东西比较透明，不像kubeadm,封装很严，不知道具体流程，出问题也不知道怎么处理。
本文转自银狐博客51CTO博客，原文链接http://blog.51cto.com/foxhound/2044766如需转载请自行联系原作者
战狐
