在Kubernetes上搭建RabbitMQ Cluster
为了在Kubernetes上搭建RabbitMQ3.7.X Cluster,踩爆无数坑,官方整合了第三方开源项目但没有完整demo,网上的post都是RabbitMQ 3.6.X旧版的部署方案,几经周折,最终弄明白在Kubernetes集群下,基于Kubernetes Discovery,使用hostname方式部署RabbitMQ3.7.X Cluster,总结如下:
1. IP模式
rabbitmq-peer-discovery-k8s是RabbitMQ官方基于第三方开源项目rabbitmq-autocluster开发,对3.7.X版本提供的Kubernetes下的同行发现插件,但官方只提供了一个基于IP模式的demo
kind: Service apiVersion: v1 metadata: namespace: test-rabbitmq name: rabbitmq labels: app: rabbitmq type: LoadBalancer spec: type: NodePort ports: - name: http protocol: TCP port: 15672 targetPort: 15672 nodePort: 31672 - name: amqp protocol: TCP port: 5672 targetPort: 5672 nodePort: 30672 selector: app: rabbitmq --- apiVersion: v1 kind: ConfigMap metadata: name: rabbitmq-config namespace: test-rabbitmq data: enabled_plugins: | [rabbitmq_management,rabbitmq_peer_discovery_k8s]. rabbitmq.conf: | ## Cluster formation. See http://www.rabbitmq.com/cluster-formation.html to learn more. cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s cluster_formation.k8s.host = kubernetes.default.svc.cluster.local ## Should RabbitMQ node name be computed from the pod's hostname or IP address? ## IP addresses are not stable, so using [stable] hostnames is recommended when possible. ## Set to "hostname" to use pod hostnames. ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME ## environment variable. cluster_formation.k8s.address_type = ip ## How often should node cleanup checks run? cluster_formation.node_cleanup.interval = 30 ## Set to false if automatic removal of unknown/absent nodes ## is desired. This can be dangerous, see ## * http://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup ## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ cluster_formation.node_cleanup.only_log_warning = true cluster_partition_handling = autoheal ## See http://www.rabbitmq.com/ha.html#master-migration-data-locality queue_master_locator=min-masters ## See http://www.rabbitmq.com/access-control.html#loopback-users loopback_users.guest = false --- apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: rabbitmq namespace: test-rabbitmq spec: serviceName: rabbitmq replicas: 3 template: metadata: labels: app: rabbitmq spec: serviceAccountName: rabbitmq terminationGracePeriodSeconds: 10 containers: - name: rabbitmq-k8s image: rabbitmq:3.7 volumeMounts: - name: config-volume mountPath: /etc/rabbitmq ports: - name: http protocol: TCP containerPort: 15672 - name: amqp protocol: TCP containerPort: 5672 livenessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 60 periodSeconds: 60 timeoutSeconds: 10 readinessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 20 periodSeconds: 60 timeoutSeconds: 10 imagePullPolicy: Always env: - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP - name: RABBITMQ_USE_LONGNAME value: "true" # See a note on cluster_formation.k8s.address_type in the config file section - name: RABBITMQ_NODENAME value: "rabbit@$(MY_POD_IP)" - name: K8S_SERVICE_NAME value: "rabbitmq" - name: RABBITMQ_ERLANG_COOKIE value: "mycookie" volumes: - name: config-volume configMap: name: rabbitmq-config items: - key: rabbitmq.conf path: rabbitmq.conf - key: enabled_plugins path: enabled_plugins
在ConfigMap配置项中,指明 cluster_formation.k8s.address_type = ip,也就是说RabbitMQ Node的命名和访问地址是以IP地址作为区分,如rabbit@172.0.5.1
但这样的配置会产生比较大的问题,如果我们使用pv和pvc去做数据的持久化,那么每个节点的配置和数据存储都会放在rabbit@172.0.5.1这样的文件夹下,而Kubernetes集群中,Pod的IP都是不稳定的,当有RabbitMQ Node的Pod挂掉后,重新创建的Pod IP可能会变,这就会使得节点的配置和数据全部丢失。
所以我们更希望RabbitMQ Node的命名是以一定规则编写的相对稳定的名称,如rabbit@rabbit-0,这就需要修改 cluster_formation.k8s.address_type = hostname,以启用hostname模式。
但直接修改address_type 并不能满足要求,注释部分也描述了“Set to hostname to use pod hostnames. When this value is changed, so should the variable used to set the RABBITMQ_NODENAME”。那么RABBITMQ_NODENAME该如何设置,就必须先要了解如何用hostname访问pod
2. Pod与Service的DNS
Kubernetes官方讲述了如何用hostname访问service和pod:dns-pod-service,
其中对于service,可以直接使用my-svc.my-namespace.svc.cluster.local进行访问;而对于pod,则需使用pod-ip-address.my-namespace.pod.cluster.local进行访问,但这里却仍显式的应用到了pod的ip。我们希望脱离ip对pod进行访问,很不幸的是,pod确实无法直接通过hostname访问,不过却有个曲线救国的方案。
apiVersion: v1 kind: Service metadata: name: default-subdomain # 和pod的subdomain相同 spec: selector: name: busybox clusterIP: None # clusterIP: None表示这是一个headless service ports: - name: foo # 没啥用 port: 1234 targetPort: 1234 --- apiVersion: v1 kind: Pod metadata: name: busybox1 labels: name: busybox spec: hostname: busybox-1 # 默认使用metadata.name作为hostname,也可指定设置 subdomain: default-subdomain containers: - image: busybox command: - sleep - "3600" name: busybox
如上面代码所示,我们需要一个headless service来作为中介,这样就可以使用busybox-1.default-subdomain.default.svc.cluster.local来访问pod了(hostname.subdomain.my-namespace.svc.cluster.local)
3. Statefulset 与Headless Service
了解了如何用hostname访问Pod还不足以解决问题,在RabbitMQ的配置中,我们使用的是StatefulSet,那么StatefulSet如何用Headless Service去做Pod的hostname访问呢?
Kubernetes(StatefulSets在1.9版本后已经是一个稳定功能)官方也给出了详细的说明:statefulset
Demo和注释如下:
apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: - port: 80 name: web clusterIP: None # 是一个headless service selector: app: nginx --- apiVersion: apps/v1 # 需要注意如果是apps/v1,.spec.selector.matchLabels和.spec.template.metadata.labels要相同;如果是apps/v1beta,可以省略.spec.selector.matchLabels kind: StatefulSet metadata: name: web spec: selector: matchLabels: app: nginx # 需要与 .spec.template.metadata.labels 相同,但无需与headless service name相同 serviceName: "nginx" # 需要与headless service name相同 replicas: 3 template: metadata: labels: app: nginx # 需要与 .spec.selector.matchLabels 相同,但无需与headless service name相同 spec: terminationGracePeriodSeconds: 10 containers: - name: nginx image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web
需要特别注意的是,网上很多例子的StatefulSet用的apps/v1beta
4. hostname模式
在我查找的众多资料中,在Kubernetes中
讲RabbitMQ 3.6.X部署的,https://www.kubernetes.org.cn/2629.html 这篇讲的比较清楚
讲RabbitMQ 3.7.X部署的,https://habr.com/company/eastbanctech/blog/419817 这篇俄文的Post讲的比较清楚,但它也是用的apps/v1beta,同时有大量的重复配置,不知道哪些可用哪些无用,还有一个最致命的问题是按照它的配置部署后,readinessProbe老报错,说DNS解析出现问题。几经折腾,才明白因为用Headless Service去做Pod的hostname访问,需要等Pod和Service都启动后才能访问,而readiness探针还没等DNS正常就去探查服务是否可用,所以才会误认为服务不可达,最终无法启动Pod。解决办法是给Headless Service设置publishNotReadyAddresses: true
我的配置文件如下所示:
apiVersion: v1 kind: Namespace metadata: name: rabbitmq --- apiVersion: v1 kind: ServiceAccount metadata: name: rabbitmq namespace: rabbitmq --- kind: Role apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: endpoint-reader namespace: rabbitmq rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: endpoint-reader namespace: rabbitmq subjects: - kind: ServiceAccount name: rabbitmq roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: endpoint-reader --- apiVersion: v1 kind: PersistentVolume metadata: name: rabbitmq-data labels: release: rabbitmq-data namespace: rabbitmq spec: capacity: storage: 10Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain nfs: path: /rabbit server: xxxxx # nas地址 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rabbitmq-data-claim namespace: rabbitmq spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi selector: matchLabels: release: rabbitmq-data --- # headless service 用于使用hostname访问pod kind: Service apiVersion: v1 metadata: name: rabbitmq-headless namespace: rabbitmq spec: clusterIP: None # publishNotReadyAddresses, when set to true, indicates that DNS implementations must publish the notReadyAddresses of subsets for the Endpoints associated with the Service. The default value is false. The primary use case for setting this field is to use a StatefulSet's Headless Service to propagate SRV records for its Pods without respect to their readiness for purpose of peer discovery. This field will replace the service.alpha.kubernetes.io/tolerate-unready-endpoints when that annotation is deprecated and all clients have been converted to use this field. # 由于使用DNS访问Pod需Pod和Headless service启动之后才能访问,publishNotReadyAddresses设置成true,防止readinessProbe在服务没启动时找不到DNS publishNotReadyAddresses: true ports: - name: amqp port: 5672 - name: http port: 15672 selector: app: rabbitmq --- # 用于暴露dashboard到外网 kind: Service apiVersion: v1 metadata: namespace: rabbitmq name: rabbitmq-service spec: type: NodePort ports: - name: http protocol: TCP port: 15672 targetPort: 15672 nodePort: 15672 # 注意k8s默认情况下,nodeport要在30000~32767之间,可以自行修改 - name: amqp protocol: TCP port: 5672 targetPort: 5672 # 注意如果你想在外网下访问mq,需要增配nodeport selector: app: rabbitmq --- apiVersion: v1 kind: ConfigMap metadata: name: rabbitmq-config namespace: rabbitmq data: enabled_plugins: | [rabbitmq_management,rabbitmq_peer_discovery_k8s]. rabbitmq.conf: | cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s cluster_formation.k8s.host = kubernetes.default.svc.cluster.local cluster_formation.k8s.address_type = hostname cluster_formation.node_cleanup.interval = 10 cluster_formation.node_cleanup.only_log_warning = true cluster_partition_handling = autoheal queue_master_locator=min-masters loopback_users.guest = false cluster_formation.randomized_startup_delay_range.min = 0 cluster_formation.randomized_startup_delay_range.max = 2 # 必须设置service_name,否则Pod无法正常启动,这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量 cluster_formation.k8s.service_name = rabbitmq-headless # 必须设置hostname_suffix,否则节点不能成为集群 cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local # 内存上限 vm_memory_high_watermark.absolute = 1.6GB # 硬盘上限 disk_free_limit.absolute = 2GB --- # 使用apps/v1版本代替apps/v1beta apiVersion: apps/v1 kind: StatefulSet metadata: name: rabbitmq namespace: rabbitmq spec: serviceName: rabbitmq-headless # 必须与headless service的name相同,用于hostname传播访问pod selector: matchLabels: app: rabbitmq # 在apps/v1中,需与 .spec.template.metadata.label 相同,用于hostname传播访问pod,而在apps/v1beta中无需这样做 replicas: 3 template: metadata: labels: app: rabbitmq # 在apps/v1中,需与 .spec.selector.matchLabels 相同 # 设置podAntiAffinity annotations: scheduler.alpha.kubernetes.io/affinity: > { "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": [{ "labelSelector": { "matchExpressions": [{ "key": "app", "operator": "In", "values": ["rabbitmq"] }] }, "topologyKey": "kubernetes.io/hostname" }] } } spec: serviceAccountName: rabbitmq terminationGracePeriodSeconds: 10 containers: - name: rabbitmq image: registry-vpc.cn-shenzhen.aliyuncs.com/heygears/rabbitmq:3.7 resources: limits: cpu: 0.5 memory: 2Gi requests: cpu: 0.3 memory: 2Gi volumeMounts: - name: config-volume mountPath: /etc/rabbitmq - name: rabbitmq-data mountPath: /var/lib/rabbitmq/mnesia ports: - name: http protocol: TCP containerPort: 15672 - name: amqp protocol: TCP containerPort: 5672 livenessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 60 periodSeconds: 60 timeoutSeconds: 5 readinessProbe: exec: command: ["rabbitmqctl", "status"] initialDelaySeconds: 20 periodSeconds: 60 timeoutSeconds: 5 imagePullPolicy: Always env: - name: HOSTNAME valueFrom: fieldRef: fieldPath: metadata.name - name: RABBITMQ_USE_LONGNAME value: "true" - name: RABBITMQ_NODENAME value: "rabbit@$(HOSTNAME).rabbitmq-headless.rabbitmq.svc.cluster.local" # 若在ConfigMap中设置了service_name,则此处无需再次设置 # - name: K8S_SERVICE_NAME # value: "rabbitmq-headless" - name: RABBITMQ_ERLANG_COOKIE value: "mycookie" volumes: - name: config-volume configMap: name: rabbitmq-config items: - key: rabbitmq.conf path: rabbitmq.conf - key: enabled_plugins path: enabled_plugins - name: rabbitmq-data persistentVolumeClaim: claimName: rabbitmq-data-claim
至此,终于在Kubernetes上部署完成RabbitMQ Cluster 3.7.X
本文转自中文社区-在Kubernetes上搭建RabbitMQ Cluster
低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。
持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。
转载内容版权归作者及来源网站所有,本站原创内容转载请注明来源。
- 上一篇
基于drone的CI/CD,对接kubernetes实践教程
CI 概述 用一个可描述的配置定义整个工作流 程序员是很懒的动物,所以想各种办法解决重复劳动的问题,如果你的工作流中还在重复一些事,那么可能就得想想如何优化了 持续集成就是可以帮助我们解决重复的代码构建,自动化测试,发布等重复劳动,通过简单一个提交代码的动作,解决接下来要做的很多事。 容器技术使这一切变得更完美。 典型的一个场景: 我们写一个前端的工程,假设是基于vue.js的框架开发的,提交代码之后希望跑一跑测试用例,然后build压缩一个到dist目录里,再把这个目录的静态文件用nginx代理一下。 最后打成docker镜像放到镜像仓库。 甚至还可以增加一个在线上运行起来的流程。 现在告诉你,只需要一个git push动作,接下来所有的事CI工具会帮你解决!这样的系统如果你还没用上的话,那请问还在等什么。接下来会系统的向大家介绍这一切。 代码仓库管理 首先SVN这种渣渣软件就该尽早淘汰,没啥好说的,有git真的没有SVN存在的必要了我觉得。 所以我们选一个git仓库,git仓库比较多,我这里选用gogs,gitea gitlab都行,根据需求自行选择 docker run -d -...
- 下一篇
阿里云Kubernetes实战3–DevOps
前言: 在上一篇文章中,我们已经在K8S集群部署了Jenkins、Harbor和EFK。作为本系列最后一篇文章,将通过实际案例串联所有的基础软件服务,基于K8S做DevOps。 整体的业务流程如下图所示: 一、一机多Jenkins Slave 由于业务需要,我们的自动化测试需要基于windows做web功能测试,每一个测试任务独占一个windows用户桌面,所以我们首先要给Jenkins配置几个Windows的Slave Node.在我之前的post《持续集成CI实施指南三–jenkins集成测试》中详细讲解了给Jenkins添加Node的方法步骤。 本篇无需重复,但这里主要讲的是,如何在一台Windows服务器上搭建多个Jenkins Node,供多用户使用。 在目标机上建立多个用户,如下图所示: 用Administrator用户安装JDK 在Jenkins的节点管理建立三个Node,分别为WinTester01、WinTester02、WinTester03,配置如下 在目标机的Administrator,用IE打开Jenkins并进入节点管理,在WinTester01、WinTe...
相关文章
文章评论
共有0条评论来说两句吧...
文章二维码
点击排行
推荐阅读
最新文章
- Docker使用Oracle官方镜像安装(12C,18C,19C)
- CentOS7,CentOS8安装Elasticsearch6.8.6
- CentOS6,CentOS7官方镜像安装Oracle11G
- CentOS7编译安装Cmake3.16.3,解决mysql等软件编译问题
- SpringBoot2编写第一个Controller,响应你的http请求并返回结果
- CentOS8编译安装MySQL8.0.19
- SpringBoot2配置默认Tomcat设置,开启更多高级功能
- Windows10,CentOS7,CentOS8安装Nodejs环境
- Docker快速安装Oracle11G,搭建oracle11g学习环境
- CentOS8,CentOS7,CentOS6编译安装Redis5.0.7