MCollective架构篇7-多MQ下MCollective高可用部署
零基础学习Puppet自动化配置管理系列文档 存在这样一种场景,当你的puppet基于mcollective环境搭建完成之后,需要考虑MQ的高可用,否则,MQ挂掉之后就不能用mco命令进行推送了哦。 如何做MQ的高可用呢,其实有两种方法: 方法一:两台MQ做集群,通过复制队列信息进行同步,节点访问可通过浮动IP进行。 方法二:两台MQ独立,在MC Server端做failover,通过rabbtimq的plugins参数实现,可设置自动检测,切换时间等等。 一、配置Rabbitmq 安装(略),可参考http://kisspuppet.com/2013/11/10/mcollective-middleware/或http://rsyslog.org/2013/11/10/mcollective-middleware/ 1. 开启插件rabbitmq_stomp [root@linuxmaster1poc~]#rabbitmq-pluginsenablerabbitmq_stomp Thefollowingpluginshavebeenenabled: rabbitmq_stomp Pluginconfigurationhaschanged.RestartRabbitMQforchangestotakeeffect. 2. 添加tcp监听端口和范围 [root@linuxmaster1poc~]#vim/etc/rabbitmq/rabbitmq.config [ {rabbitmq_stomp,[{tcp_listeners,[61613]}]} ]. 备注:可参考http://www.rabbitmq.com/stomp.html 3. 创建账户并设置权限 如果你以前配置过,建议将配置清空 [root@linuxmaster1poc~]#rabbitmqctlstop_app Stoppingnoderabbit@linuxmaster1poc... ...done. [root@linuxmaster1poc~]#rabbitmqctlreset Resettingnoderabbit@linuxmaster1poc... ...done. [root@linuxmaster1poc~]#rabbitmqctlstart_app Startingnoderabbit@linuxmaster1poc... ...done. 删除默认用户guest,添加三个用户(web_admin-http访问用,admin--管理员,mc_rabbitmq--mcollective链接用) [root@linuxmaster1poc~]#rabbitmqctllist_users Listingusers... guest[administrator] ...done. [root@linuxmaster1poc~]#rabbitmqctldelete_userguest Deletinguser"guest"... ...done. [root@linuxmaster1poc~]#rabbitmqctladd_usermc_rabbitmq123.com Creatinguser"mc_rabbitmq"... ...done. [root@linuxmaster1poc~]#rabbitmqctladd_useradminpassword=123.com Creatinguser"admin"... ...done. [root@linuxmaster1poc~]#rabbitmqctladd_userweb_admin123.com Creatinguser"web_admin"... ...done. 设置用户的角色 [root@linuxmaster1poc~]#rabbitmqctlset_user_tagsadminadministrator Settingtagsforuser"admin"to[administrator]... ...done. [root@linuxmaster1poc~]#rabbitmqctlset_user_tagsweb_adminmonitoring Settingtagsforuser"web_admin"to[monitoring]... ...done. 创建虚拟主机组 [root@linuxmaster1poc~]#rabbitmqctladd_vhost/mcollective Creatingvhost"/mcollective"... ...done. 设置用户访问虚拟主机组的权限 [root@linuxmaster1poc~]#rabbitmqctlset_permissions-p"/mcollective"mc_rabbitmq".*"".*"".*" Settingpermissionsforuser"mc_rabbitmq"invhost"/mcollective"... ...done. [root@linuxmaster1poc~]#rabbitmqctlset_permissions-p"/mcollective"admin".*"".*"".*" Settingpermissionsforuser"admin"invhost"/mcollective"... ...done. [root@linuxmaster1poc~]#rabbitmqctlset_permissions-p"/mcollective"web_admin".*"".*"".*" Settingpermissionsforuser"web_admin"invhost"/mcollective"... ...done. 重启rabbitmq-server服务 [root@linuxmaster1poc~]#/etc/init.d/rabbitmq-serverrestart Restartingrabbitmq-server:SUCCESS rabbitmq-server. 查看用户以及角色是否创建成功 [root@linuxmaster1poc~]#rabbitmqctllist_users Listingusers... admin[administrator] mc_rabbitmq[] web_admin[monitoring] ...done. 查看虚拟主机组“/mcollective”中所有用户的权限 [root@linuxmaster1poc~]#rabbitmqctllist_permissions-p"/mcollective" Listingpermissionsinvhost"/mcollective"... admin.*.*.* mc_rabbitmq.*.*.* web_admin.*.*.* ...done. [root@linuxmaster1poc~]# 4、登录http://192.168.100.120:15672/设置虚拟主机“/mcollective”的exchanges 默认配置 [root@linuxmaster1poc~]#rabbitmqctllist_exchanges-p"/mcollective" Listingexchanges... direct amq.directdirect amq.fanoutfanout amq.headersheaders amq.matchheaders amq.rabbitmq.tracetopic amq.topictopic ...done. exchanges设置 设置后更新配置 exchanges设置 [root@linuxmaster1poc~]#rabbitmqctllist_exchanges-p"/mcollective" Listingexchanges... direct amq.directdirect amq.fanoutfanout amq.headersheaders amq.matchheaders amq.rabbitmq.tracetopic amq.topictopic mcollective_broadcasttopic mcollective_directeddirect ...done. 备注:可参考官网设置https://www.rabbitmq.com/man/rabbitmqctl.1.man.html 二、 配置MCollective: 1. 配置mcollective client端 [root@linuxmaster1poctesting]#cat/etc/mcollective/client.cfg topicprefix=/topic/ main_collective=mcollective collectives=mcollective libdir=/usr/libexec/mcollective logger_type=console #loglevel=debug loglevel=warn #Plugins securityprovider=psk plugin.psk=a36cd839414370e10fd281b8a38a4f48 direct_addressing=1 connector=rabbitmq plugin.rabbitmq.vhost=/mcollective#虚拟主机 plugin.rabbitmq.pool.size=2#设置地址池里有两个mq plugin.rabbitmq.initial_reconnect_delay=0.01 plugin.rabbitmq.max_reconnect_delay=30.0#重连时间 plugin.rabbitmq.use_exponential_back_off=true plugin.rabbitmq.back_off_multiplier=2 plugin.rabbitmq.max_reconnect_attempts=0 plugin.rabbitmq.randomize=false plugin.rabbitmq.timeout=-1 plugin.rabbitmq.pool.1.host=192.168.100.120 plugin.rabbitmq.pool.1.port=61613 plugin.rabbitmq.pool.1.user=mc_rabbitmq plugin.rabbitmq.pool.1.password=123.com plugin.rabbitmq.pool.1.ssl=false plugin.rabbitmq.pool.2.host=192.168.100.121 plugin.rabbitmq.pool.2.port=61613 plugin.rabbitmq.pool.2.user=mc_rabbitmq plugin.rabbitmq.pool.2.password=123.com plugin.rabbitmq.pool.2.ssl=false #Facts factsource=yaml plugin.yaml=/etc/mcollective/facts.yaml 2. 配置mcollective server端 [root@linux57poctmp]#cat/etc/mcollective/server.cfg #--Global-- topicprefix=/topic/ main_collective=mcollective collectives=mcollective libdir=/usr/libexec/mcollective logfile=/var/log/puppet/mcollective.log loglevel=info daemonize=1 #--rabbitmqPlugins-- securityprovider=psk plugin.psk=a36cd839414370e10fd281b8a38a4f48 direct_addressing=1 connector=rabbitmq plugin.rabbitmq.vhost=/mcollective plugin.rabbitmq.pool.size=2 plugin.rabbitmq.initial_reconnect_delay=0.01 plugin.rabbitmq.max_reconnect_delay=30.0 plugin.rabbitmq.use_exponential_back_off=true plugin.rabbitmq.back_off_multiplier=2 plugin.rabbitmq.max_reconnect_attempts=0 plugin.rabbitmq.randomize=false plugin.rabbitmq.timeout=-1 plugin.rabbitmq.pool.1.host=192.168.100.120 plugin.rabbitmq.pool.1.port=61613 plugin.rabbitmq.pool.1.user=mc_rabbitmq plugin.rabbitmq.pool.1.password=123.com plugin.rabbitmq.pool.1.ssl=false plugin.rabbitmq.pool.2.host=192.168.100.121 plugin.rabbitmq.pool.2.port=61613 plugin.rabbitmq.pool.2.user=mc_rabbitmq plugin.rabbitmq.pool.2.password=123.com plugin.rabbitmq.pool.2.ssl=false #--Puppetproviderspecificoptions-- plugin.service.provider=puppet plugin.service.puppet.hasstatus=true plugin.service.puppet.hasrestart=true plugin.puppet.command=puppetagent plugin.puppet.splay=true plugin.puppet.splaylimit=30 plugin.puppet.config=/etc/puppet/puppet.conf #--Facts-- factsource=yaml ##factsource=facter plugin.yaml=/etc/mcollective/facts.yaml 三、高可用测试 特别注意:节点mcollective的server.cfg中pool是有优先级的,默认数字小的生效,这点需要注意,也就是说当所有节点都连接在MQ2上的时候,启动MQ1,mco命令是无法使用的,因为它在运行的时候连接的是MQ1,而所有节点都连接在MQ2上。 1. 停止MQ1,查看切换状态 1.1 先看当前的节点连接状态 [root@linuxmaster1poc~]#mcoping#查看连接的节点 linux57poctime=69.46ms linux58poctime=70.05ms linux64poctime=70.59ms ----pingstatistics---- 3repliesmax:70.59min:69.46avg:70.03 [root@linuxmaster1poc~]#mcoshell"lsof-i:61613"#查看所有节点监听的端口情况,可以看到目前都连接在linuxmaster1poc上。 Doyoureallywanttosendthiscommandunfiltered?(y/n):y Discoveringhostsusingthemcmethodfor2second(s)....3 Host:linux64poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby36625root6uIPv4277710t0TCPlinux64poc:40493->linuxmaster1poc:61613(ESTABLISHED) Host:linux58poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby11060root6uIPv4340460t0TCPlinux58poc:36295->linuxmaster1poc:61613(ESTABLISHED) Host:linux57poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby18076root6uIPv41351365TCPlinux57poc:24698->linuxmaster1poc:61613(ESTABLISHED) [root@linuxmaster1poc~]#/etc/init.d/rabbitmq-serverstop Stoppingrabbitmq-server:rabbitmq-server. 1.2 再次运行mco查看切换状态 [root@linuxmaster1poc~]#mcoping linux58poctime=73.54ms linux64poctime=74.61ms linux57poctime=75.39ms ----pingstatistics---- 3repliesmax:75.39min:73.54avg:74.51 [root@linuxmaster1poc~]#mcoshell"lsof-i:61613" Doyoureallywanttosendthiscommandunfiltered?(y/n):y Discoveringhostsusingthemcmethodfor2second(s)....3 Host:linux58poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby11060root6uIPv4340460t0TCPlinux58poc:36295->linuxmaster1poc:61613(CLOSE_WAIT) ruby11060root9uIPv4341370t0TCPlinux58poc:47200->linuxmaster2poc:61613(ESTABLISHED) Host:linux64poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby36625root6uIPv4277710t0TCPlinux64poc:40493->linuxmaster1poc:61613(CLOSE_WAIT) ruby36625root8uIPv4278770t0TCPlinux64poc:37472->linuxmaster2poc:61613(ESTABLISHED) Host:linux57poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby18076root9uIPv41351484TCPlinux57poc:9309->linuxmaster2poc:61613(ESTABLISHED) 通过日志查看 [root@linuxmaster1poc~]#mcoshell"lsof-i:61613" Doyoureallywanttosendthiscommandunfiltered?(y/n):y Discoveringhostsusingthemcmethodfor2second(s)....3 Host:linux58poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby11428root6uIPv4342830t0TCPlinux58poc:36300->linuxmaster1poc:61613(CLOSE_WAIT) ruby11428root8uIPv4343380t0TCPlinux58poc:47205->linuxmaster2poc:61613(ESTABLISHED) Host:linux57poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby18447root6uIPv41351559TCPlinux57poc:59343->linuxmaster1poc:61613(CLOSE_WAIT) ruby18447root8uIPv41351622TCPlinux57poc:29757->linuxmaster2poc:61613(ESTABLISHED) Host:linux64poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby37054root4uIPv4280360t0TCPlinux64poc:37476->linuxmaster2poc:61613(ESTABLISHED) ruby37054root6uIPv4279900t0TCPlinux64poc:40497->linuxmaster1poc:61613(CLOSE_WAIT) 总结:可以看到之前的连接已经变成CLOSE_WAIT,新的连接被建立 2. 停止MQ2,启动MQ1 查看切换状态 [root@linuxmaster2pocrabbitmq]#/etc/init.d/rabbitmq-serverstop Stoppingrabbitmq-server:rabbitmq-server. [root@linux57pocservice]#lsof-i:61613 COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby18447root6uIPv41351559TCPlinux57poc:59343->linuxmaster1poc:61613(CLOSE_WAIT) ruby18447root8uIPv41351622TCPlinux57poc:29757->linuxmaster2poc:61613(CLOSE_WAIT) [root@linux58poc~]#lsof-i:61613 COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby11428root6uIPv4342830t0TCPlinux58poc:36300->linuxmaster1poc:61613(CLOSE_WAIT) ruby11428root8uIPv4343380t0TCPlinux58poc:47205->linuxmaster2poc:61613(CLOSE_WAIT) [root@linux64poc~]#lsof-i:61613 COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby37054root4uIPv4280360t0TCPlinux64poc:37476->linuxmaster2poc:61613(CLOSE_WAIT) ruby37054root6uIPv4279900t0TCPlinux64poc:40497->linuxmaster1poc:61613(CLOSE_WAIT) [root@linuxmaster1poc~]#/etc/init.d/rabbitmq-serverstart Startingrabbitmq-server:SUCCESS rabbitmq-server. 根据 plugin.rabbitmq.max_reconnect_delay = 30.0,需要过最多30秒,mcollective服务端会重新建立连接请求 [root@linuxmaster1poc~]#tailf/var/log/rabbitmq/rabbit\@linuxmaster1poc.log =INFOREPORT====24-Dec-2013::11:00:45=== acceptingSTOMPconnection<0.332.0>(192.168.100.126:36316->192.168.100.120:61613) =INFOREPORT====24-Dec-2013::11:00:45=== acceptingSTOMPconnection<0.348.0>(192.168.100.125:18945->192.168.100.120:61613) =INFOREPORT====24-Dec-2013::11:00:45=== acceptingSTOMPconnection<0.382.0>(192.168.100.127:40513->192.168.100.120:61613) [root@linuxmaster1poc~]#mcoping linux58poctime=70.60ms linux57poctime=71.32ms linux64poctime=111.56ms ----pingstatistics---- 3repliesmax:111.56min:70.60avg:84.49 [root@linuxmaster1poc~]#mcoshell"lsof-i:61613" Doyoureallywanttosendthiscommandunfiltered?(y/n):y Discoveringhostsusingthemcmethodfor2second(s)....3 Host:linux58poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby11428root6uIPv4342830t0TCPlinux58poc:36300->linuxmaster1poc:61613(CLOSE_WAIT) ruby11428root8uIPv4343380t0TCPlinux58poc:47205->linuxmaster2poc:61613(CLOSE_WAIT) ruby11428root10uIPv4344440t0TCPlinux58poc:36316->linuxmaster1poc:61613(ESTABLISHED) Host:linux57poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZENODENAME ruby18447root10uIPv41351723TCPlinux57poc:18945->linuxmaster1poc:61613(ESTABLISHED) Host:linux64poc Statuscode:0 Output: COMMANDPIDUSERFDTYPEDEVICESIZE/OFFNODENAME ruby37054root4uIPv4280360t0TCPlinux64poc:37476->linuxmaster2poc:61613(CLOSE_WAIT) ruby37054root6uIPv4279900t0TCPlinux64poc:40497->linuxmaster1poc:61613(CLOSE_WAIT) ruby37054root9uIPv4282060t0TCPlinux64poc:40513->linuxmaster1poc:61613(ESTABLISHED) 本文转自凌激冰51CTO博客,原文链接:http://blog.51cto.com/dreamfire/1586729,如需转载请自行联系原作者