您现在的位置是：首页 > 文章详情

k8s环境部署及使用方式

日期：2018-12-14点击：399收藏

安装k8s+docker集群环境（基于centos7系统）

集群机器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67

1.配置yum源
[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release] name=virt7-docker-common-release baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/ gpgcheck=0

加载安装包：
[centos-master]:yum repolist

virt7-docker-common-release virt7-docker-common-release 15

2.安装集群必要软件—-etcd/flannel/kubernetes
Etcd服务在k8s集群中用于配置共享和服务发现。

Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。

[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel

3.如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：

echo "192.168.121.9 centos-master 192.168.121.65 centos-minion-1 192.168.121.66 centos-minion-2 192.168.121.67 centos-minion-3" >> /etc/hosts

4.修改配置master节点的kubernetes配置文件/etc/kubernetes/config

#表示错误日志记录到文件还是输出到stderr KUBE_LOGTOSTDERR="--logtostderr=true" #日志等级 KUBE_LOG_LEVEL="--v=0" #允许运行特权容器 KUBE_ALLOW_PRIV="--allow-privileged=false" #apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9 KUBE_MASTER="--master=http://192.168.121.9:8080"

5.k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则，并需关闭selinux。为确保k8s集群的正常运行，我们可以直接关闭iptables与seliinux服务。

setenforce 0 systemctl stop firewalld.service systemctl stop firewalld.service

6.修改配置master节点的etcd配置文件/etc/etcd/etcd.conf
etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：

# [member] #etcd名称 ETCD_NAME=default #etcd数据存储位置 ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #监听的端口 ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #[cluster] #集群监听的端口 ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

7.修改配置master节点的apiserver配置文件/etc/kubernetes/apiserver

#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0 KUBE_API_ADDRESS="--address=0.0.0.0" #apiserver的监听端口，默认8080 KUBE_API_PORT="--port=8080" #minion上kubelet监听的端口，默认10250 KUBELET_PORT="--kubelet-port=10250" #etcd服务地址，前面已经启动了etcd服务,端口为2379 KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" #kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配 KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" #需要额外添加的配置项，简单地启用一个集群无需配置 KUBE_API_ARGS=""

8.启动并且配置etcd的网段，此网段一定是未被使用的

systemctl start etcd etcdctl mkdir /kube-centos/network etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

9.修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口 FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379" #服务范围 FLANNEL_ETCD_PREFIX="/kube-centos/network" #其他 FLANNEL_OPTIONS=""

10.启动k8s集群

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done

以上就是k8s，master节点的安装及配置
接下来，配置nodes节点

11.修改配置nodes节点kubelet的配置文件/etc/kubernetes/kubelet

#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0 KUBELET_ADDRESS="--address=0.0.0.0" #监听的端口 KUBELET_PORT="--port=10250" #apiserver的访问地址及端口 KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080" #额外增加的参数 KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"

12.修改配置nodes节点flanneld的配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口 FLANNEL_ETCD="http://192.168.121.9:2379" #etcd服务范围 FLANNEL_ETCD_KEY="/kube-centos/network"

13.启动k8s集群服务

for SERVICES in kube-proxy kubelet flanneld docker; do systemctl restart $SERVICES systemctl enable $SERVICES systemctl status $SERVICES done

14.设置kubectl的配置文件

kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080 kubectl config set-context default-context --cluster=default-cluster --user=default-admin kubectl config use-context default-context

15.检查集群状态
[centos-master]:kubectl get nodes

NAME STATUS AGE VERSION centos-minion-1 Ready 3d v1.5.0+fff5156 centos-minion-2 Ready 3d v1.5.0+fff5156 centos-minion-3 Ready 3d v1.5.0+fff5156

至此，集群构建完毕

搭建私有库

私有库用于系统内部存储成品镜像，能够快速进行下载及被k8s调度。

1.下载并启动私有库

[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry #--name 表示启动的容器后名称，此处为registry #-v 表示挂载路径 格式为宿主机路径:容器内路径 #-p 表示映射端口 格式为宿主机端口:容器内端口 #-itd docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry

2.创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务

[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker --docker-email=lienhua@zhongchuangsanyou.com [centos-master]:kubectl get secret NAME TYPE DATA AGE registrykey kubernetes.io/dockercfg 1 6s

此时登录时会提示认证错误

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn Flag --email has been deprecated, will be removed in 1.13. Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized

这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用tls。这样我们就需要为Registry配置tls所需的key和crt文件了

3.配置nginx反向代理
[centos-master]: cat registry.evehicle.cn.conf

# For versions of nginx > 1.3.9 that include chunked transfer encoding support # Replace with appropriate values where necessary upstream docker-registry { server 192.168.121.9:5000; #server 10.44.170.95:5000; } # uncomment if you want a 301 redirect for users attempting to connect # on port 80 # NOTE: docker client will still fail. This is just for convenience # server { # listen *:80; # server_name my.docker.registry.com; # return 301 https://$server_name$request_uri; # } server { listen 443; server_name registry.evehicle.cn; ssl on; ssl_certificate ssl/registry.evehicle.cn.crt; ssl_certificate_key ssl/registry.evehicle.cn.key; client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486) chunked_transfer_encoding on; location / { auth_basic "Restricted"; auth_basic_user_file passwd; add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always; proxy_pass http://docker-registry; proxy_set_header Host $http_host; # required for docker client's sake proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 900; } location /_ping { auth_basic off; include docker-registry.conf; } location /v1/_ping { auth_basic off; include docker-registry.conf; } location /v2/_ping { auth_basic off; include docker-registry.conf; } }

将key及crt证书文件放到../ssl目录下。使用htpasswd生成密码放于./上一级目录

 htpasswd -bcm passwd docker docker #-c：创建一个加密文件 #-m：md5加密，默认可不填写 #-b：表示用户名密码在命令行中一并输入，不用分别填写

4.再次登录

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn Login Succeeded 表示成功，此时再pull\push既在私有库中进行

构建服务

docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：
1.从docker的公有库里下载centos7的原生镜像

[centos-master]:docker pull centos Using default tag: latest Trying to pull repository docker.io/library/centos ... latest: Pulling from docker.io/library/centos d9aaf4d82f24: Downloading [> ] 540 kB/73.39 MB d9aaf4d82f24: Pulling fs layer Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e Status: Downloaded newer image for centos:latest

2.编写Dockerfile制造apache基础镜像

######httpd#### FROM centos MAINTAINER lienhua lienhua@zhongchuangsanyou.com RUN yum -y install epel-release RUN yum -y install httpd php php-mysql php-memcache* php-mbstring ADD httpd.conf /etc/httpd/conf/httpd.conf EXPOSE 80 CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]

其中httpd.conf文件需要在当前目录下真实存在，此处其内容为

ServerRoot "/etc/httpd" Listen 80 Listen 8080 Include conf.modules.d/*.conf Include zcsy/*.conf User apache Group apache ServerAdmin root@localhost <Directory /> AllowOverride none Require all denied </Directory> DocumentRoot "/var/www/html" <Directory "/var/www"> AllowOverride None Require all granted </Directory> <Directory "/var/www/html"> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> <IfModule dir_module> DirectoryIndex index.html </IfModule> <Files ".ht*"> Require all denied </Files> ErrorLog "logs/error_log" LogLevel warn <IfModule log_config_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common <IfModule logio_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio </IfModule> CustomLog "logs/access_log" combined </IfModule> <IfModule alias_module> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" </IfModule> <Directory "/var/www/cgi-bin"> AllowOverride None Options None Require all granted </Directory> <IfModule mime_module> TypesConfig /etc/mime.types AddType application/x-compress .Z AddType application/x-gzip .gz .tgz AddType application/x-httpd-php .php AddType text/html .shtml AddOutputFilter INCLUDES .shtml </IfModule> AddDefaultCharset UTF-8 <IfModule mime_magic_module> MIMEMagicFile conf/magic </IfModule> EnableSendfile off EnableMMAP off IncludeOptional conf.d/*.conf

执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）

3.将制作好的镜像上传到私有库

docker push registry.evehicle.cn/httpd

4.编写启动apache服务的yaml文件

[centos-master]:cat 13-rc-httpd.yaml

apiVersion: v1 kind: ReplicationController metadata: name: 13-rc-httpd labels: name: 13-rc-httpd spec: replicas: 2 selector: name: 13-rc-httpd template: metadata: labels: name: 13-rc-httpd spec: containers: - name: 13-rc-httpd image: registry.evehicle.cn/httpd env: - name: LANG value: en_US.UTF-8 ports: - containerPort: 80 hostPort: 80 volumeMounts: - name: time mountPath: /etc/localtime - name: zcsy mountPath: /etc/httpd/zcsy - name: deploy mountPath: /docker/httpd/deploy - name: log mountPath: /var/log/httpd volumes: - name: time hostPath: path: /etc/localtime - name: zcsy hostPath: path: /docker/httpd/zcsy - name: deploy hostPath: path: /docker/httpd/deploy - name: log hostPath: path: /docker/httpd/log nodeSelector: slave: "13" imagePullSecrets: - name: registrykey

5.给其中一个node加上标签为“13”

kubectl label nodes centos-minion-1 slave=13

6.此时拥有标签“13”的nodes应具备的条件

/docker/httpd/zcsy下需要有官网的配置文件

<VirtualHost *:80> ServerName www.evehicle.cn DocumentRoot /var/deploy/wordpress/ RewriteEngine on RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^.*\.(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$ RewriteCond %{REQUEST_URI} !^/server-status$ RewriteRule . /index.php [QSA,PT,L] </VirtualHost> <Directory /var/deploy/wordpress/> Options FollowSymLinks AllowOverride All Require all granted </Directory>

以及/docker/httpd/deploy下需要有官网的代码

7.运行yaml文件启动容器

[centos-master]: kuberctl create -f 13-rc-httpd.yaml

8.查看服务

[centos-master]: kuberctl get rc

NAME DESIRED CURRENT AGE 13-rc-httpd 2 2 168d

9.程序中涉及的mysql\redis\memcache等服务也需使用容器运行起来

[centos-master]: docker pull redis
[centos-master]: docker tag registry.evehicle.cn/redis redis
[centos-master]: docker push registry.evehicle.cn/redis
[centos-master]: kubectl create -f rc-redis.yaml
[centos-master]: cat rc-redis.yaml

apiVersion: v1 kind: ReplicationController metadata: name: redis labels: name: redis spec: replicas: 2 selector: name: redis template: metadata: labels: name: redis spec: containers: - name: redis image: registry.evehicle.cn/redis ports: - containerPort: 6379 hostPort: 6379 volumeMounts: - name: data mountPath: /data - name: time mountPath: /etc/localtime volumes: - name: data hostPath: path: /docker/redis/6379 - name: time hostPath: path: /etc/localtime nodeSelector: slave: "13" imagePullSecrets: - name: registrykey

启动memcache
[centos-master]: docker pull memcache
[centos-master]: docker tag registry.evehicle.cn/memcached memcache
[centos-master]: docker push registry.evehicle.cn/memcached
[centos-master]: kubectl create -f rc-memcached.yaml
[centos-master]: cat rc-memcached.yaml

apiVersion: v1 kind: ReplicationController metadata: name: memcached labels: name: memcached spec: replicas: 3 selector: name: memcached template: metadata: labels: name: memcached spec: containers: - name: memcached image: registry.evehicle.cn/memcached ports: - containerPort: 11211 hostPort: 11211 #nodeSelector: # slave: "13" imagePullSecrets: - name: registrykey

制造mysql镜像
[centos-master]: cat Dockerfile

FROM alpine COPY startup.sh /startup.sh RUN addgroup mysql && \ adduser -H -D -s /bin/false -G mysql mysql && \ apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \ mkdir /data && \ chown -R mysql:mysql /data /etc/mysql && \ chmod 755 /startup.sh \ ; WORKDIR /data VOLUME /data VOLUME /etc/mysql EXPOSE 3306 CMD ["/startup.sh"]

启动mysql（建议mysql在宿主机启动）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml

apiVersion: v1 kind: ReplicationController metadata: name: 13-rc-mysql labels: name: 13-rc-mysql spec: replicas: 2 selector: name: 13-rc-mysql template: metadata: labels: name: 13-rc-mysql spec: containers: - name: 13-rc-mysql image: registry.evehicle.cn/mysql env: - name: MYSQL_DATABASE value: admin - name: MYSQL_USER value: tony - name: MYSQL_PASSWORD value: 456 - name: MYSQL_ROOT_PASSWORD value: 123 ports: - containerPort: 3306 hostPort: 3306 volumeMounts: - name: time mountPath: /etc/localtime - name: data mountPath: /data - name: etc mountPath: /etc/mysql - name: run mountPath: /run/mysqld volumes: - name: time hostPath: path: /etc/localtime - name: data hostPath: path: /docker/mysql/data - name: etc hostPath: path: /docker/mysql/etc - name: run hostPath: path: /docker/mysql/run nodeSelector: slave: "13" imagePullSecrets: - name: registrykey