搭建mcollective高可用，使puppet架构更加安全、稳定
一、配置Rabbitmq 安装（略），可参考http://kisspuppet.com/2013/11/10/mcollective-middleware/或http://rsyslog.org/2013/11/10/mcollective-middleware/ 1.开启插件rabbitmq_stomp 1 2 3 4 [root@linuxmaster1poc ~]# rabbitmq-plugins enable rabbitmq_stomp The following plugins have been enabled: rabbitmq_stomp Plugin configuration has changed. Restart RabbitMQ for changes to take effect. 2.添加tcp监听端口和范围 1 2 3 4 [root@linuxmaster1poc ~]# vim /etc/rabbitmq/rabbitmq.config [ {rabbitmq_stomp, [{tcp_listeners, [ 61613 ]}]} ]. 备注：可参考http://www.rabbitmq.com/stomp.html 3.创建账户并设置权限 如果你以前配置过，建议将配置清空 1 2 3 4 5 6 7 8 9 [root@linuxmaster1poc ~]# rabbitmqctl stop_app Stopping node rabbit@linuxmaster1poc ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl reset Resetting node rabbit@linuxmaster1poc ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl start_app Starting node rabbit@linuxmaster1poc ... ...done. 删除默认用户guest，添加三个用户（webadmin-http访问用，admin--管理员，mcrabbitmq--mcollective链接用） 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [root@linuxmaster1poc ~]# rabbitmqctl list_users Listing users ... guest [administrator] ...done. [root@linuxmaster1poc ~]# rabbitmqctl delete_user guest Deleting user "guest" ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl add_user mc_rabbitmq 123 .com Creating user "mc_rabbitmq" ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl add_user admin password= 123 .com Creating user "admin" ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl add_user web_admin 123 .com Creating user "web_admin" ... ...done. 设置用户的角色 1 2 3 4 5 6 [root@linuxmaster1poc ~]# rabbitmqctl set_user_tags admin administrator Setting tags for user "admin" to [administrator] ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl set_user_tags web_admin monitoring Setting tags for user "web_admin" to [monitoring] ... ...done. 创建虚拟主机组 1 2 3 [root@linuxmaster1poc ~]# rabbitmqctl add_vhost /mcollective Creating vhost "/mcollective" ... ...done. 设置用户访问虚拟主机组的权限 1 2 3 4 5 6 7 8 9 [root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" mc_rabbitmq ".*" ".*" ".*" Setting permissions for user "mc_rabbitmq" in vhost "/mcollective" ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" admin ".*" ".*" ".*" Setting permissions for user "admin" in vhost "/mcollective" ... ...done. [root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p "/mcollective" web_admin ".*" ".*" ".*" Setting permissions for user "web_admin" in vhost "/mcollective" ... ...done. 重启rabbitmq-server服务 1 2 3 [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server restart Restarting rabbitmq-server: SUCCESS rabbitmq-server. 查看用户以及角色是否创建成功 1 2 3 4 5 6 [root@linuxmaster1poc ~]# rabbitmqctl list_users Listing users ... admin [administrator] mc_rabbitmq [] web_admin [monitoring] ...done. 查看虚拟主机组“/mcollective”中所有用户的权限 1 2 3 4 5 6 7 [root@linuxmaster1poc ~]# rabbitmqctl list_permissions -p "/mcollective" Listing permissions in vhost "/mcollective" ... admin .* .* .* mc_rabbitmq .* .* .* web_admin .* .* .* ...done. [root@linuxmaster1poc ~]# 4、登录http://192.168.100.120:15672/设置虚拟主机“/mcollective”的exchanges 默认配置 1 2 3 4 5 6 7 8 9 10 [root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective" Listing exchanges ... direct amq.direct direct amq.fanout fanout amq.headers headers amq.match headers amq.rabbitmq. trace topic amq.topic topic ...done. 设置后更新配置 1 2 3 4 5 6 7 8 9 10 11 12 [root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p "/mcollective" Listing exchanges ... direct amq.direct direct amq.fanout fanout amq.headers headers amq.match headers amq.rabbitmq. trace topic amq.topic topic mcollective_broadcast topic mcollective_directed direct ...done. 备注：可参考官网设置https://www.rabbitmq.com/man/rabbitmqctl.1.man.html 二、配置MCollective: 1.配置mcollective client端 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 [root@linuxmaster1poc testing]# cat /etc/mcollective/client.cfg topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logger_type = console #loglevel = debug loglevel = warn # Plugins securityprovider = psk plugin.psk = a36cd839414370e10fd281b8a38a4f48 direct_addressing = 1 connector = rabbitmq plugin.rabbitmq.vhost = /mcollective #虚拟主机 plugin.rabbitmq.pool.size = 2 #设置地址池里有两个mq plugin.rabbitmq.initial_reconnect_delay = 0.01 plugin.rabbitmq.max_reconnect_delay = 30.0 #重连时间 plugin.rabbitmq.use_exponential_back_off = true plugin.rabbitmq.back_off_multiplier = 2 plugin.rabbitmq.max_reconnect_attempts = 0 plugin.rabbitmq.randomize = false plugin.rabbitmq.timeout = - 1 plugin.rabbitmq.pool. 1 .host = 192.168 . 100.120 plugin.rabbitmq.pool. 1 .port = 61613 plugin.rabbitmq.pool. 1 .user = mc_rabbitmq plugin.rabbitmq.pool. 1 .password = 123 .com plugin.rabbitmq.pool. 1 .ssl = false plugin.rabbitmq.pool. 2 .host = 192.168 . 100.121 plugin.rabbitmq.pool. 2 .port = 61613 plugin.rabbitmq.pool. 2 .user = mc_rabbitmq plugin.rabbitmq.pool. 2 .password = 123 .com plugin.rabbitmq.pool. 2 .ssl = false # Facts factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml 2.配置mcollective server端 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 [root@linux57poc tmp]# cat /etc/mcollective/server.cfg # --Global-- topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logfile = / var /log/puppet/mcollective.log loglevel = info daemonize = 1 # --rabbitmq Plugins-- securityprovider = psk plugin.psk = a36cd839414370e10fd281b8a38a4f48 direct_addressing = 1 connector = rabbitmq plugin.rabbitmq.vhost = /mcollective plugin.rabbitmq.pool.size = 2 plugin.rabbitmq.initial_reconnect_delay = 0.01 plugin.rabbitmq.max_reconnect_delay = 30.0 plugin.rabbitmq.use_exponential_back_off = true plugin.rabbitmq.back_off_multiplier = 2 plugin.rabbitmq.max_reconnect_attempts = 0 plugin.rabbitmq.randomize = false plugin.rabbitmq.timeout = - 1 plugin.rabbitmq.pool. 1 .host = 192.168 . 100.120 plugin.rabbitmq.pool. 1 .port = 61613 plugin.rabbitmq.pool. 1 .user = mc_rabbitmq plugin.rabbitmq.pool. 1 .password = 123 .com plugin.rabbitmq.pool. 1 .ssl = false plugin.rabbitmq.pool. 2 .host = 192.168 . 100.121 plugin.rabbitmq.pool. 2 .port = 61613 plugin.rabbitmq.pool. 2 .user = mc_rabbitmq plugin.rabbitmq.pool. 2 .password = 123 .com plugin.rabbitmq.pool. 2 .ssl = false # --Puppet provider specific options-- plugin.service.provider = puppet plugin.service.puppet.hasstatus = true plugin.service.puppet.hasrestart = true plugin.puppet.command = puppet agent plugin.puppet.splay = true plugin.puppet.splaylimit = 30 plugin.puppet.config = /etc/puppet/puppet.conf # --Facts-- factsource = yaml ##factsource = facter plugin.yaml = /etc/mcollective/facts.yaml 三、高可用测试 特别注意：节点mcollective的server.cfg中pool是有优先级的，默认数字小的生效，这点需要注意，也就是说当所有节点都连接在MQ2上的时候，启动MQ1，mco命令是无法使用的，因为它在运行的时候连接的是MQ1，而所有节点都连接在MQ2上。 1.停止MQ1，查看切换状态 1.1先看当前的节点连接状态 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [root@linuxmaster1poc ~]# mco ping #查看连接的节点 linux57poc time= 69.46 ms linux58poc time= 70.05 ms linux64poc time= 70.59 ms ---- ping statistics ---- 3 replies max: 70.59 min: 69.46 avg: 70.03 [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" #查看所有节点监听的端口情况，可以看到目前都连接在linuxmaster1poc上。 Do you really want to send this command unfiltered? (y/n): y Discovering hosts using the mc method for 2 second(s) .... 3 Host: linux64poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 36625 root 6u IPv4 27771 0t0 TCP linux64poc: 40493 ->linuxmaster1poc: 61613 (ESTABLISHED) Host: linux58poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 11060 root 6u IPv4 34046 0t0 TCP linux58poc: 36295 ->linuxmaster1poc: 61613 (ESTABLISHED) Host: linux57poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME ruby 18076 root 6u IPv4 1351365 TCP linux57poc: 24698 ->linuxmaster1poc: 61613 (ESTABLISHED) [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server stop Stopping rabbitmq-server: rabbitmq-server. 1.2再次运行mco查看切换状态 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [root@linuxmaster1poc ~]# mco ping linux58poc time= 73.54 ms linux64poc time= 74.61 ms linux57poc time= 75.39 ms ---- ping statistics ---- 3 replies max: 75.39 min: 73.54 avg: 74.51 [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" Do you really want to send this command unfiltered? (y/n): y Discovering hosts using the mc method for 2 second(s) .... 3 Host: linux58poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 11060 root 6u IPv4 34046 0t0 TCP linux58poc: 36295 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 11060 root 9u IPv4 34137 0t0 TCP linux58poc: 47200 ->linuxmaster2poc: 61613 (ESTABLISHED) Host: linux64poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 36625 root 6u IPv4 27771 0t0 TCP linux64poc: 40493 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 36625 root 8u IPv4 27877 0t0 TCP linux64poc: 37472 ->linuxmaster2poc: 61613 (ESTABLISHED) Host: linux57poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME ruby 18076 root 9u IPv4 1351484 TCP linux57poc: 9309 ->linuxmaster2poc: 61613 (ESTABLISHED) 通过日志查看 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" Do you really want to send this command unfiltered? (y/n): y Discovering hosts using the mc method for 2 second(s) .... 3 Host: linux58poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc: 36300 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc: 47205 ->linuxmaster2poc: 61613 (ESTABLISHED) Host: linux57poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME ruby 18447 root 6u IPv4 1351559 TCP linux57poc: 59343 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 18447 root 8u IPv4 1351622 TCP linux57poc: 29757 ->linuxmaster2poc: 61613 (ESTABLISHED) Host: linux64poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc: 37476 ->linuxmaster2poc: 61613 (ESTABLISHED) ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc: 40497 ->linuxmaster1poc: 61613 (CLOSE_WAIT) 总结：可以看到之前的连接已经变成CLOSE_WAIT，新的连接被建立 2.停止MQ2，启动MQ1查看切换状态 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 [root@linuxmaster2poc rabbitmq]# /etc/init.d/rabbitmq-server stop Stopping rabbitmq-server: rabbitmq-server. [root@linux57poc service]# lsof -i: 61613 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME ruby 18447 root 6u IPv4 1351559 TCP linux57poc: 59343 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 18447 root 8u IPv4 1351622 TCP linux57poc: 29757 ->linuxmaster2poc: 61613 (CLOSE_WAIT) [root@linux58poc ~]# lsof -i: 61613 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc: 36300 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc: 47205 ->linuxmaster2poc: 61613 (CLOSE_WAIT) [root@linux64poc ~]# lsof -i: 61613 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc: 37476 ->linuxmaster2poc: 61613 (CLOSE_WAIT) ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc: 40497 ->linuxmaster1poc: 61613 (CLOSE_WAIT) [root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server start Starting rabbitmq-server: SUCCESS rabbitmq-server. 根据plugin.rabbitmq.maxreconnectdelay =30.0，需要过最多30秒，mcollective服务端会重新建立连接请求 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 [root@linuxmaster1poc ~]# tailf / var /log/rabbitmq/rabbit\@linuxmaster1poc.log =INFO REPORT==== 24 -Dec- 2013 :: 11 : 00 : 45 === accepting STOMP connection < 0.332 . 0 > ( 192.168 . 100.126 : 36316 -> 192.168 . 100.120 : 61613 ) =INFO REPORT==== 24 -Dec- 2013 :: 11 : 00 : 45 === accepting STOMP connection < 0.348 . 0 > ( 192.168 . 100.125 : 18945 -> 192.168 . 100.120 : 61613 ) =INFO REPORT==== 24 -Dec- 2013 :: 11 : 00 : 45 === accepting STOMP connection < 0.382 . 0 > ( 192.168 . 100.127 : 40513 -> 192.168 . 100.120 : 61613 ) [root@linuxmaster1poc ~]# mco ping linux58poc time= 70.60 ms linux57poc time= 71.32 ms linux64poc time= 111.56 ms ---- ping statistics ---- 3 replies max: 111.56 min: 70.60 avg: 84.49 [root@linuxmaster1poc ~]# mco shell "lsof -i:61613" Do you really want to send this command unfiltered? (y/n): y Discovering hosts using the mc method for 2 second(s) .... 3 Host: linux58poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 11428 root 6u IPv4 34283 0t0 TCP linux58poc: 36300 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 11428 root 8u IPv4 34338 0t0 TCP linux58poc: 47205 ->linuxmaster2poc: 61613 (CLOSE_WAIT) ruby 11428 root 10u IPv4 34444 0t0 TCP linux58poc: 36316 ->linuxmaster1poc: 61613 (ESTABLISHED) Host: linux57poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME ruby 18447 root 10u IPv4 1351723 TCP linux57poc: 18945 ->linuxmaster1poc: 61613 (ESTABLISHED) Host: linux64poc Statuscode: 0 Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 37054 root 4u IPv4 28036 0t0 TCP linux64poc: 37476 ->linuxmaster2poc: 61613 (CLOSE_WAIT) ruby 37054 root 6u IPv4 27990 0t0 TCP linux64poc: 40497 ->linuxmaster1poc: 61613 (CLOSE_WAIT) ruby 37054 root 9u IPv4 28206 0t0 TCP linux64poc: 40513 ->linuxmaster1poc: 61613 (ESTABLISHED) 本文转自凌激冰51CTO博客，原文链接：http://blog.51cto.com/dreamfire/1344492，如需转载请自行联系原作者
