搭建mcollective高可用,使puppet架构更加安全、稳定
一、配置Rabbitmq
安装(略),可参考http://kisspuppet.com/2013/11/10/mcollective-middleware/或http://rsyslog.org/2013/11/10/mcollective-middleware/
1. 开启插件rabbitmq_stomp
|
1
2
3
4
|
[root@linuxmaster1poc ~]# rabbitmq-plugins enable rabbitmq_stomp
The following plugins have been enabled:
rabbitmq_stomp
Plugin configuration has changed. Restart RabbitMQ
for
changes to take effect.
|
2. 添加tcp监听端口和范围
|
1
2
3
4
|
[root@linuxmaster1poc ~]# vim /etc/rabbitmq/rabbitmq.config
[
{rabbitmq_stomp, [{tcp_listeners, [
61613
]}]}
].
|
备注:可参考http://www.rabbitmq.com/stomp.html
3. 创建账户并设置权限
如果你以前配置过,建议将配置清空
|
1
2
3
4
5
6
7
8
9
|
[root@linuxmaster1poc ~]# rabbitmqctl stop_app
Stopping node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl reset
Resetting node rabbit@linuxmaster1poc ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl start_app
Starting node rabbit@linuxmaster1poc ...
...done.
|
删除默认用户guest,添加三个用户(webadmin-http访问用,admin--管理员,mcrabbitmq--mcollective链接用)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
guest [administrator]
...done.
[root@linuxmaster1poc ~]# rabbitmqctl delete_user guest
Deleting user
"guest"
...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user mc_rabbitmq
123
.com
Creating user
"mc_rabbitmq"
...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user admin password=
123
.com
Creating user
"admin"
...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl add_user web_admin
123
.com
Creating user
"web_admin"
...
...done.
|
设置用户的角色
|
1
2
3
4
5
6
|
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags admin administrator
Setting tags
for
user
"admin"
to [administrator] ...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_user_tags web_admin monitoring
Setting tags
for
user
"web_admin"
to [monitoring] ...
...done.
|
创建虚拟主机组
|
1
2
3
|
[root@linuxmaster1poc ~]# rabbitmqctl add_vhost /mcollective
Creating vhost
"/mcollective"
...
...done.
|
设置用户访问虚拟主机组的权限
|
1
2
3
4
5
6
7
8
9
|
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p
"/mcollective"
mc_rabbitmq
".*"
".*"
".*"
Setting permissions
for
user
"mc_rabbitmq"
in
vhost
"/mcollective"
...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p
"/mcollective"
admin
".*"
".*"
".*"
Setting permissions
for
user
"admin"
in
vhost
"/mcollective"
...
...done.
[root@linuxmaster1poc ~]# rabbitmqctl set_permissions -p
"/mcollective"
web_admin
".*"
".*"
".*"
Setting permissions
for
user
"web_admin"
in
vhost
"/mcollective"
...
...done.
|
重启rabbitmq-server服务
|
1
2
3
|
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
|
查看用户以及角色是否创建成功
|
1
2
3
4
5
6
|
[root@linuxmaster1poc ~]# rabbitmqctl list_users
Listing users ...
admin [administrator]
mc_rabbitmq []
web_admin [monitoring]
...done.
|
查看虚拟主机组“/mcollective”中所有用户的权限
|
1
2
3
4
5
6
7
|
[root@linuxmaster1poc ~]# rabbitmqctl list_permissions -p
"/mcollective"
Listing permissions
in
vhost
"/mcollective"
...
admin .* .* .*
mc_rabbitmq .* .* .*
web_admin .* .* .*
...done.
[root@linuxmaster1poc ~]#
|
4、登录http://192.168.100.120:15672/设置虚拟主机“/mcollective”的exchanges
默认配置
|
1
2
3
4
5
6
7
8
9
10
|
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p
"/mcollective"
Listing exchanges ...
direct
amq.direct direct
amq.fanout fanout
amq.headers headers
amq.match headers
amq.rabbitmq.
trace
topic
amq.topic topic
...done.
|
设置后更新配置
|
1
2
3
4
5
6
7
8
9
10
11
12
|
[root@linuxmaster1poc ~]# rabbitmqctl list_exchanges -p
"/mcollective"
Listing exchanges ...
direct
amq.direct direct
amq.fanout fanout
amq.headers headers
amq.match headers
amq.rabbitmq.
trace
topic
amq.topic topic
mcollective_broadcast topic
mcollective_directed direct
...done.
|
备注:可参考官网设置https://www.rabbitmq.com/man/rabbitmqctl.1.man.html
二、配置MCollective:
1. 配置mcollective client端
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
[root@linuxmaster1poc testing]# cat /etc/mcollective/client.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logger_type = console
#loglevel = debug
loglevel = warn
# Plugins
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing =
1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective #虚拟主机
plugin.rabbitmq.pool.size =
2
#设置地址池里有两个mq
plugin.rabbitmq.initial_reconnect_delay =
0.01
plugin.rabbitmq.max_reconnect_delay =
30.0
#重连时间
plugin.rabbitmq.use_exponential_back_off =
true
plugin.rabbitmq.back_off_multiplier =
2
plugin.rabbitmq.max_reconnect_attempts =
0
plugin.rabbitmq.randomize =
false
plugin.rabbitmq.timeout = -
1
plugin.rabbitmq.pool.
1
.host =
192.168
.
100.120
plugin.rabbitmq.pool.
1
.port =
61613
plugin.rabbitmq.pool.
1
.user = mc_rabbitmq
plugin.rabbitmq.pool.
1
.password =
123
.com
plugin.rabbitmq.pool.
1
.ssl =
false
plugin.rabbitmq.pool.
2
.host =
192.168
.
100.121
plugin.rabbitmq.pool.
2
.port =
61613
plugin.rabbitmq.pool.
2
.user = mc_rabbitmq
plugin.rabbitmq.pool.
2
.password =
123
.com
plugin.rabbitmq.pool.
2
.ssl =
false
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
|
2. 配置mcollective server端
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
[root@linux57poc tmp]# cat /etc/mcollective/server.cfg
# --Global--
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /
var
/log/puppet/mcollective.log
loglevel = info
daemonize =
1
# --rabbitmq Plugins--
securityprovider = psk
plugin.psk = a36cd839414370e10fd281b8a38a4f48
direct_addressing =
1
connector = rabbitmq
plugin.rabbitmq.vhost = /mcollective
plugin.rabbitmq.pool.size =
2
plugin.rabbitmq.initial_reconnect_delay =
0.01
plugin.rabbitmq.max_reconnect_delay =
30.0
plugin.rabbitmq.use_exponential_back_off =
true
plugin.rabbitmq.back_off_multiplier =
2
plugin.rabbitmq.max_reconnect_attempts =
0
plugin.rabbitmq.randomize =
false
plugin.rabbitmq.timeout = -
1
plugin.rabbitmq.pool.
1
.host =
192.168
.
100.120
plugin.rabbitmq.pool.
1
.port =
61613
plugin.rabbitmq.pool.
1
.user = mc_rabbitmq
plugin.rabbitmq.pool.
1
.password =
123
.com
plugin.rabbitmq.pool.
1
.ssl =
false
plugin.rabbitmq.pool.
2
.host =
192.168
.
100.121
plugin.rabbitmq.pool.
2
.port =
61613
plugin.rabbitmq.pool.
2
.user = mc_rabbitmq
plugin.rabbitmq.pool.
2
.password =
123
.com
plugin.rabbitmq.pool.
2
.ssl =
false
# --Puppet provider specific options--
plugin.service.provider = puppet
plugin.service.puppet.hasstatus =
true
plugin.service.puppet.hasrestart =
true
plugin.puppet.command = puppet agent
plugin.puppet.splay =
true
plugin.puppet.splaylimit =
30
plugin.puppet.config = /etc/puppet/puppet.conf
# --Facts--
factsource = yaml
##factsource = facter
plugin.yaml = /etc/mcollective/facts.yaml
|
三、高可用测试
特别注意:节点mcollective的server.cfg中pool是有优先级的,默认数字小的生效,这点需要注意,也就是说当所有节点都连接在MQ2上的时候,启动MQ1,mco命令是无法使用的,因为它在运行的时候连接的是MQ1,而所有节点都连接在MQ2上。
1. 停止MQ1,查看切换状态
1.1 先看当前的节点连接状态
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@linuxmaster1poc ~]# mco ping #查看连接的节点
linux57poc time=
69.46
ms
linux58poc time=
70.05
ms
linux64poc time=
70.59
ms
---- ping statistics ----
3
replies max:
70.59
min:
69.46
avg:
70.03
[root@linuxmaster1poc ~]# mco shell
"lsof -i:61613"
#查看所有节点监听的端口情况,可以看到目前都连接在linuxmaster1poc上。
Do you really want to send
this
command unfiltered? (y/n): y
Discovering hosts using the mc method
for
2
second(s) ....
3
Host: linux64poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
36625
root 6u IPv4
27771
0t0 TCP linux64poc:
40493
->linuxmaster1poc:
61613
(ESTABLISHED)
Host: linux58poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
11060
root 6u IPv4
34046
0t0 TCP linux58poc:
36295
->linuxmaster1poc:
61613
(ESTABLISHED)
Host: linux57poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby
18076
root 6u IPv4
1351365
TCP linux57poc:
24698
->linuxmaster1poc:
61613
(ESTABLISHED)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.
|
1.2 再次运行mco查看切换状态
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@linuxmaster1poc ~]# mco ping
linux58poc time=
73.54
ms
linux64poc time=
74.61
ms
linux57poc time=
75.39
ms
---- ping statistics ----
3
replies max:
75.39
min:
73.54
avg:
74.51
[root@linuxmaster1poc ~]# mco shell
"lsof -i:61613"
Do you really want to send
this
command unfiltered? (y/n): y
Discovering hosts using the mc method
for
2
second(s) ....
3
Host: linux58poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
11060
root 6u IPv4
34046
0t0 TCP linux58poc:
36295
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
11060
root 9u IPv4
34137
0t0 TCP linux58poc:
47200
->linuxmaster2poc:
61613
(ESTABLISHED)
Host: linux64poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
36625
root 6u IPv4
27771
0t0 TCP linux64poc:
40493
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
36625
root 8u IPv4
27877
0t0 TCP linux64poc:
37472
->linuxmaster2poc:
61613
(ESTABLISHED)
Host: linux57poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby
18076
root 9u IPv4
1351484
TCP linux57poc:
9309
->linuxmaster2poc:
61613
(ESTABLISHED)
|
通过日志查看
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
[root@linuxmaster1poc ~]# mco shell
"lsof -i:61613"
Do you really want to send
this
command unfiltered? (y/n): y
Discovering hosts using the mc method
for
2
second(s) ....
3
Host: linux58poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
11428
root 6u IPv4
34283
0t0 TCP linux58poc:
36300
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
11428
root 8u IPv4
34338
0t0 TCP linux58poc:
47205
->linuxmaster2poc:
61613
(ESTABLISHED)
Host: linux57poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby
18447
root 6u IPv4
1351559
TCP linux57poc:
59343
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
18447
root 8u IPv4
1351622
TCP linux57poc:
29757
->linuxmaster2poc:
61613
(ESTABLISHED)
Host: linux64poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
37054
root 4u IPv4
28036
0t0 TCP linux64poc:
37476
->linuxmaster2poc:
61613
(ESTABLISHED)
ruby
37054
root 6u IPv4
27990
0t0 TCP linux64poc:
40497
->linuxmaster1poc:
61613
(CLOSE_WAIT)
|
总结:可以看到之前的连接已经变成CLOSE_WAIT,新的连接被建立
2. 停止MQ2,启动MQ1 查看切换状态
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[root@linuxmaster2poc rabbitmq]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: rabbitmq-server.
[root@linux57poc service]# lsof -i:
61613
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby
18447
root 6u IPv4
1351559
TCP linux57poc:
59343
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
18447
root 8u IPv4
1351622
TCP linux57poc:
29757
->linuxmaster2poc:
61613
(CLOSE_WAIT)
[root@linux58poc ~]# lsof -i:
61613
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
11428
root 6u IPv4
34283
0t0 TCP linux58poc:
36300
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
11428
root 8u IPv4
34338
0t0 TCP linux58poc:
47205
->linuxmaster2poc:
61613
(CLOSE_WAIT)
[root@linux64poc ~]# lsof -i:
61613
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
37054
root 4u IPv4
28036
0t0 TCP linux64poc:
37476
->linuxmaster2poc:
61613
(CLOSE_WAIT)
ruby
37054
root 6u IPv4
27990
0t0 TCP linux64poc:
40497
->linuxmaster1poc:
61613
(CLOSE_WAIT)
[root@linuxmaster1poc ~]# /etc/init.d/rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.
|
根据 plugin.rabbitmq.maxreconnectdelay =30.0,需要过最多30秒,mcollective服务端会重新建立连接请求
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
[root@linuxmaster1poc ~]# tailf /
var
/log/rabbitmq/rabbit\@linuxmaster1poc.log
=INFO REPORT====
24
-Dec-
2013
::
11
:
00
:
45
===
accepting STOMP connection <
0.332
.
0
> (
192.168
.
100.126
:
36316
->
192.168
.
100.120
:
61613
)
=INFO REPORT====
24
-Dec-
2013
::
11
:
00
:
45
===
accepting STOMP connection <
0.348
.
0
> (
192.168
.
100.125
:
18945
->
192.168
.
100.120
:
61613
)
=INFO REPORT====
24
-Dec-
2013
::
11
:
00
:
45
===
accepting STOMP connection <
0.382
.
0
> (
192.168
.
100.127
:
40513
->
192.168
.
100.120
:
61613
)
[root@linuxmaster1poc ~]# mco ping
linux58poc time=
70.60
ms
linux57poc time=
71.32
ms
linux64poc time=
111.56
ms
---- ping statistics ----
3
replies max:
111.56
min:
70.60
avg:
84.49
[root@linuxmaster1poc ~]# mco shell
"lsof -i:61613"
Do you really want to send
this
command unfiltered? (y/n): y
Discovering hosts using the mc method
for
2
second(s) ....
3
Host: linux58poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
11428
root 6u IPv4
34283
0t0 TCP linux58poc:
36300
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
11428
root 8u IPv4
34338
0t0 TCP linux58poc:
47205
->linuxmaster2poc:
61613
(CLOSE_WAIT)
ruby
11428
root 10u IPv4
34444
0t0 TCP linux58poc:
36316
->linuxmaster1poc:
61613
(ESTABLISHED)
Host: linux57poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
ruby
18447
root 10u IPv4
1351723
TCP linux57poc:
18945
->linuxmaster1poc:
61613
(ESTABLISHED)
Host: linux64poc
Statuscode:
0
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby
37054
root 4u IPv4
28036
0t0 TCP linux64poc:
37476
->linuxmaster2poc:
61613
(CLOSE_WAIT)
ruby
37054
root 6u IPv4
27990
0t0 TCP linux64poc:
40497
->linuxmaster1poc:
61613
(CLOSE_WAIT)
ruby
37054
root 9u IPv4
28206
0t0 TCP linux64poc:
40513
->linuxmaster1poc:
61613
(ESTABLISHED)
|

