亚马逊k8s开局系列-基础组件部署
2 --> 介绍： 到这里基本上集群就已经跑起来了，接下来就要把运行业务系统的程序部署上去。这里会介绍zookeeper、etcd、kafka的部署，会讲到服务为什么需要用Deployment和StatefulSet。这两个东西分别是如何运用的。 zookeeper： 要制作zookeeper的镜像需要了解zk集群运行时的配置方法和原理，其中配置文件有个地方需要处理。下载zookeeper安装包，解压到一个目录中。# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/zookeeper/data# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#这里是添加集群节点的配置，下面可以看到zookeeper-0这样的名称。这个是在k8s中运行时的service名称。分别对应到不同的pod。server.0=zookeeper-0:2888:3888server.1=zookeeper-1:2888:3888server.2=zookeeper-2:2888:3888还有一个地方就是myid文件，通过文件内容中的数字来区分是那个节点。对应配置文件的server.0-server.2。 这个地方我决定在启动脚本里面去处理打开bin/zkServer.sh添加以下内容hostname | grep -oE [0-9] > /home/zookeeper/data/myid这样当我们使用StatefulSet方式部署服务时每个容器的hostname就会是服务名称-0这样的格式了。 Dockerfile文件FROM openjdk:11-jre-slimRUN mkdir /home/zookeeper/COPY zookeeper/ /home/zookeeper/CMD /home/zookeeper/bin/zkServer.sh start-foreground执行docker build -t docker.xxxx.net/zookeeper:v3.6.3 .执行docker push docker.xxxx.net/zookeeper:v3.6.3把镜像推送到镜像仓库。记得推送前需要登录镜像仓库，这里的docker.xxxx.net请修改成你自己的域名。 在rancher中添加镜像仓库凭证 部署zookeeper: 文件中的镜像地址请自行修改。apiVersion: apps/v1kind: StatefulSetmetadata: name: zookeeperspec: replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: app: zookeeper serviceName: zookeeper-svc template: metadata: labels: app: zookeeper spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - zookeeper topologyKey: kubernetes.io/hostname containers: - env: - name: JAVA_HOME value: /usr/local/openjdk-11 - name: LANG value: C.UTF-8 image: docker.xxxx.net/zookeeper:v3.6.3 imagePullPolicy: IfNotPresent name: zookeeper resources: requests: cpu: 500m memory: 512Mi volumeMounts: - mountPath: /home/zookeeper/data name: zookeeper - mountPath: /etc/localtime name: volume-localtime imagePullSecrets: - name: docker dnsPolicy: ClusterFirst restartPolicy: Always volumes: - hostPath: path: /etc/localtime type: "" name: volume-localtime volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: zookeeper spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumeMode: Filesystem---apiVersion: v1kind: Servicemetadata: name: zookeeper-0spec: ports: - name: cli port: 2181 protocol: TCP targetPort: 2181 - name: pri port: 2888 protocol: TCP targetPort: 2888 - name: clu port: 3888 protocol: TCP targetPort: 3888 selector: app: zookeeper statefulset.kubernetes.io/pod-name: zookeeper-0 type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: zookeeper-1spec: ports: - name: cli port: 2181 protocol: TCP targetPort: 2181 - name: pri port: 2888 protocol: TCP targetPort: 2888 - name: clu port: 3888 protocol: TCP targetPort: 3888 selector: app: zookeeper statefulset.kubernetes.io/pod-name: zookeeper-1 type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: zookeeper-2spec: ports: - name: cli port: 2181 protocol: TCP targetPort: 2181 - name: pri port: 2888 protocol: TCP targetPort: 2888 - name: clu port: 3888 protocol: TCP targetPort: 3888 selector: app: zookeeper statefulset.kubernetes.io/pod-name: zookeeper-2 type: ClusterIP 等到一段时间以后分别进入到容器中执行命令查看状态 etcd: 下载etcd包https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gztar -xf etcd-v3.5.0-linux-amd64.tar.gzmv etcd-v3.5.0-linux-amd64 etcd 创建etcd启动脚本cat etcd/start.sh#!/bin/shlocalIP=`grep $HOSTNAME /etc/hosts|grep -oE "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}"`/opt/etcd/etcd --name $HOSTNAME --initial-advertise-peer-urls http://$localIP:2380 --listen-peer-urls http://$localIP:2380 --listen-client-urls http://$localIP:2379 --advertise-client-urls http://$localIP:2379 --initial-cluster-token etcd-im-cluster --initial-cluster "etcd-0=http://etcd-0:2380,etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380" --initial-cluster-state new --data-dir=/opt/etcd/data --logger=zap这里使用的http方式，要制作https的镜像需要创建证书修改配置等请自行处理。chmod 755 etcd/start.sh Dockerfile文件 FROM alpine:latestVOLUME ["/var/cache"]ENV LANG=zh_CN.utf8 LANGUAGE=zh_CN:zh LC_ALL=zh_CN.utf8RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g" /etc/apk/repositories &&\ apk add --no-cache busybox-extras vim curl tzdata && \ cp -rf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo "export LANG=zh_CN.utf8" >> /etc/profile && . /etc/profile && \ echo "export LANG=$LANG" > /etc/profile.d/locale.shCOPY etcd /opt/etcdCMD /opt/etcd/start.sh 参照zookeeper的方式执行docker build 和 docker push命令 部署etcd： 文件中的镜像地址请自行修改。apiVersion: apps/v1kind: StatefulSetmetadata: name: etcdspec: replicas: 3 selector: matchLabels: app: etcd serviceName: etcd-svc template: metadata: labels: app: etcd spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - etcd topologyKey: kubernetes.io/hostname containers: - env: - name: LANG value: zh_CN.utf8 - name: LANGUAGE value: zh_CN:zh - name: LC_ALL value: zh_CN.utf8 - name: aliyun_logs_wos-app value: stdout image: docker.xxxx.net/etcd:v3.4.16 imagePullPolicy: Always name: etcd resources: limits: memory: 1Gi requests: cpu: 250m memory: 512Mi volumeMounts: - mountPath: /etc/localtime name: volume-localtime - mountPath: /opt/etcd/data name: etcd dnsPolicy: ClusterFirst imagePullSecrets: - name: docker restartPolicy: Always volumes: - hostPath: path: /etc/localtime type: "" name: volume-localtime volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: etcd spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumeMode: Filesystem---apiVersion: v1kind: Servicemetadata: name: etcd-0spec: clusterIP: None ports: - name: cli port: 2379 protocol: TCP targetPort: 2379 - name: pri port: 2380 protocol: TCP targetPort: 2380 selector: app: etcd statefulset.kubernetes.io/pod-name: etcd-0 sessionAffinity: None type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: etcd-1spec: clusterIP: None ports: - name: cli port: 2379 protocol: TCP targetPort: 2379 - name: pri port: 2380 protocol: TCP targetPort: 2380 selector: app: etcd statefulset.kubernetes.io/pod-name: etcd-1 sessionAffinity: None type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: etcd-2spec: clusterIP: None ports: - name: cli port: 2379 protocol: TCP targetPort: 2379 - name: pri port: 2380 protocol: TCP targetPort: 2380 selector: app: etcd statefulset.kubernetes.io/pod-name: etcd-2 sessionAffinity: None type: ClusterIP等容器正常以后，进入到命令行使用命令/opt/etcd/etcdctl --endpoints=etcd-0:2379,etcd-1:2379,etcd-2:2379 endpoint health检查运行情况 kafka部署： 自行网上下载kafka安装包，修改配置文件: cat config/server.propertiesbroker.id=listeners=PLAINTEXT://port=19092auto.create.topics.enable=truenum.network.threads=4num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/opt/kafka/kafkalogs/num.partitions=1num.recovery.threads.per.data.dir=1offsets.topic.replication.factor=1transaction.state.log.replication.factor=1transaction.state.log.min.isr=1log.retention.hours=24log.segment.bytes=1073741824log.retention.check.interval.ms=300000zookeeper.connect=zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181zookeeper.connection.timeout.ms=6000group.initial.rebalance.delay.ms=0message.max.byte=5242880default.replication.factor=2replica.fetch.max.bytes=5242880log.cleaner.enable=truedelete.topic.enable=truelog.cleanup.policy = delete 修改启动文件bin/kafka-server-start.sh: rm -rf /opt/kafka/kafkalogs/lost+foundMyid=$(hostname | grep -oE [0-9])sed -i "/broker.id/c broker.id=$Myid" /home/kafka/config/server.propertiessed -i "/listeners=PLAINTEXT/c listeners=PLAINTEXT:\/\/$HOSTNAME:19092" /home/kafka/config/server.properties 放到文件这个位置Dockerfile: FROM openjdk:11-jre-slimRUN mkdir /home/kafka/COPY kafka/ /home/kafka/CMD /home/kafka/bin/kafka-server-start.sh -daemon /home/kafka/config/server.properties 然后使用docker build和docker push命令制作和上传镜像 部署kafka： 文件中的镜像地址请自行修改。apiVersion: apps/v1kind: StatefulSetmetadata: labels: app: kafka name: kafkaspec: replicas: 3 revisionHistoryLimit: 10 selector: matchLabels: app: kafka serviceName: kafka-svc template: metadata: labels: app: kafka spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - kafka topologyKey: kubernetes.io/hostname containers: - env: - name: JAVA_HOME value: /usr/local/openjdk-11 - name: LANG value: C.UTF-8 - name: JMX_PORT value: "9988" image: docker.xxxx.net/kafka:v2.11-2.4.1 imagePullPolicy: Always name: kafka resources: requests: cpu: "1" memory: 1224Mi volumeMounts: - mountPath: /opt/kafka/kafkalogs/ name: kafka - mountPath: /etc/localtime name: volume-localtime dnsPolicy: ClusterFirst restartPolicy: Always imagePullSecrets: - name: docker volumes: - hostPath: path: /etc/localtime type: "" name: volume-localtime volumeClaimTemplates: - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: kafka namespace: wos-uat spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumeMode: Filesystem---apiVersion: v1kind: Servicemetadata: name: kafka-0spec: ports: - name: client port: 19092 protocol: TCP targetPort: 19092 selector: app: kafka statefulset.kubernetes.io/pod-name: kafka-0 type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: kafka-1spec: ports: - name: client port: 19092 protocol: TCP targetPort: 19092 selector: app: kafka statefulset.kubernetes.io/pod-name: kafka-1 type: ClusterIP---apiVersion: v1kind: Servicemetadata: name: kafka-2spec: ports: - name: client port: 19092 protocol: TCP targetPort: 19092 selector: app: kafka statefulset.kubernetes.io/pod-name: kafka-2 type: ClusterIP 等运行完成以后进行健康检查。为防止磁盘空间不回收除了上面配置文件的log.cleaner.enable=truedelete.topic.enable=truelog.cleanup.policy = delete 三个选项以外还需要手动配置一下策略bin/kafka-configs.sh --zookeeper zookeeper-0:2181 --entity-type topics --entity-name __consumer_offsets --describe 查看策略默认是compot，改成deletebin/kafka-configs.sh --zookeeper zookeeper-0:2181 --entity-type topics --entity-name __consumer_offsets --alter --add-config 'cleanup.policy=delete' Deployment和StatefulSet 很多时候不知道Deployment和StatefulSet的区别在哪里。上面创建的三个服务推荐都使用StatefulSet创建，因为里面有共享存储，而这些程序使用共享存储的方式都是独占里面会有锁文件。如果使用Deployment在程序出现问题时可能会导致文件抢占新的pod启动不了，同时集群中创建service的时候为每个pod创建一个service，Deployment的pod名称是会变化的定位不到pod中会导致集群出现问题。Deployment适合用于无磁盘文件锁、状态可以中断的应用，比如我们的java微服务程序。 具体就介绍这么多，其它的基础镜像自行制作处理。希望对大家有所启发。
