Monitor 节点初始化
执行下面步骤
- 操作系统初始化
当前环境中, ceph 存储集群统一部署 centos 7.1 操作系统, 并在 /etc/hosts 中登记集群中的服务器主机名, 并完成时间同步
主机名验证
[root@hh-yun-ceph-cinder015-128055 ~]# hostname
hh-yun-ceph-cinder015-128055.vclound.com
[root@hh-yun-ceph-cinder017-128057 ~]# hostname
hh-yun-ceph-cinder017-128057.vclound.com
[root@hh-yun-ceph-cinder024-128074 ~]# hostname
hh-yun-ceph-cinder024-128074.vclound.com
cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
240.30.128.55 hh-yun-ceph-cinder015-128055.vclound.com hh-yun-ceph-cinder015-128055
240.30.128.56 hh-yun-ceph-cinder016-128056.vclound.com hh-yun-ceph-cinder016-128056
240.30.128.57 hh-yun-ceph-cinder017-128057.vclound.com hh-yun-ceph-cinder017-128057
240.30.128.73 hh-yun-ceph-cinder023-128073.vclound.com hh-yun-ceph-cinder023-128073
240.30.128.74 hh-yun-ceph-cinder024-128074.vclound.com hh-yun-ceph-cinder024-128074
240.30.128.75 hh-yun-ceph-cinder025-128075.vclound.com hh-yun-ceph-cinder025-128075
240.30.128.76 hh-yun-ceph-cinder026-128076.vclound.com hh-yun-ceph-cinder026-128076
时间同步验证
[root@hh-yun-ceph-cinder024-128074 ~]
Reference ID : 10.199.129.21 (hh-yun-129021.vclound.com)
Stratum : 12
Ref time (UTC) : Tue Jul 21 07:45:32 2015
System time : 0.000002526 seconds fast of NTP time
Last offset : 0.000005405 seconds
RMS offset : 0.007192081 seconds
Frequency : 18.428 ppm slow
Residual freq : 0.001 ppm
Skew : 0.060 ppm
Root delay : 0.000141 seconds
Root dispersion : 0.011696 seconds
Update interval : 386.4 seconds
Leap status : Normal
- ceph 软件安装
注: 假如之前安装过 ceph, 并计划清除之前的集群配置, 那么执行下面命令可以把 ceph 配置进行安装
清除方法: ceph-deploy purgedata {node-name}, ceph-deploy purge {node-name}
yum 源
yum install -y epel-release.noarch
echo "sslverify=0" >> /etc/yum.conf
ceph 软件安装
yum install -y ceph* bash-completion
- 创建配置文件, 默认是 /etc/ceph/ceph.conf
vim /etc/ceph/ceph.conf
[global]
fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9
mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074
mon host = 240.30.128.55, 240.30.128.57, 240.30.128.74
public network = 240.30.128.0/21
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1
- 创建集群密钥对创建监控密钥
[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
creating /etc/ceph/ceph.mon.keyring
[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring
[mon.]
key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==
caps mon = "allow *"
注意: ceph-0.87 版本需要把文件 /etc/ceph/ceph.mon.keyring 重命名为 /etc/ceph/ceph.keyring 或 /etc/ceph/keyring 或 /etc/ceph/ceph.mon.(hostname).keyring 否则无法执行注册
- 生成管理员密钥, 生成 client.admin 用户并增加用户密钥对
[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
creating /etc/ceph/ceph.client.admin.keyring
[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 137 7月 23 16:20 /etc/ceph/ceph.client.admin.keyring
[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
auid = 0
caps mds = "allow"
caps mon = "allow *"
caps osd = "allow *"
- 增加 client.admin 密钥对到 ceph.mon.keyring 中
[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring
[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring
-rw------- 1 root root 137 7月 23 16:20 /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 214 7月 23 16:20 /etc/ceph/ceph.mon.keyring
注意, 这里 /etc/ceph/ceph.mon.keyring 从之前的 77 字节变成 214 字节, 并自动添加如下的内容
[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring
[mon.]
key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==
caps mon = "allow *"
[client.admin]
key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
auid = 0
caps mds = "allow"
caps mon = "allow *"
caps osd = "allow *"
- 使用主机名, IP 地址及 FSID 生成 monitor map 保存到 /etc/ceph/monmap 注意, 命令创建时候需要一次过把所有的 mon 节点都直接进行定义
[root@hh-yun-ceph-cinder015-128055 tmp]# monmaptool --create --add hh-yun-ceph-cinder015-128055 240.30.128.55 --add hh-yun-ceph-cinder017-128057 240.30.128.57 --add hh-yun-ceph-cinder024-128074 240.30.128.74 --fsid dc4f91c1-8792-4948-b68f-2fcea75f53b9 /etc/ceph/monmap
monmaptool: monmap file /etc/ceph/monmap
monmaptool: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
monmaptool: writing epoch 0 to /etc/ceph/monmap (3 monitors)
- 在对应的监控主机上创建默认数据目录
[root@hh-yun-ceph-cinder015-128055 /]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055
[root@hh-yun-ceph-cinder017-128057 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057
[root@hh-yun-ceph-cinder024-128074 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074
- 定义监控进程与监控 MAP 及密钥对
[root@hh-yun-ceph-cinder015-128055 ~]# ceph-mon
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 for mon.hh-yun-ceph-cinder015-128055
把 /etc/ceph 目录下的文件: ceph.conf ceph.client.admin.keyring ceph.mon.keyring monmap 同步到另外两台 mon 节点中
[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder017-128057:/etc/ceph/.
ceph.client.admin.keyring 100% 137 0.1KB/s 00:00
ceph.conf 100% 544 0.5KB/s 00:00
ceph.mon.keyring 100% 214 0.2KB/s 00:00
monmap 100% 550 0.5KB/s 00:00
[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder024-128074:/etc/ceph/.
ceph.client.admin.keyring 100% 137 0.1KB/s 00:00
ceph.conf 100% 544 0.5KB/s 00:00
ceph.mon.keyring 100% 214 0.2KB/s 00:00
monmap 100% 550 0.5KB/s 00:00
为另外两台 mon 节点进行监控数据初始化
[root@hh-yun-ceph-cinder017-128057 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder017-128057 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 for mon.hh-yun-ceph-cinder017-128057
[root@hh-yun-ceph-cinder024-128074 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder024-128074 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074 for mon.hh-yun-ceph-cinder024-128074
- 分别启动三台 mon 节点
[root@hh-yun-ceph-cinder015-128055 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/sysvinit
[root@hh-yun-ceph-cinder017-128057 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/sysvinit
[root@hh-yun-ceph-cinder024-128074 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/sysvinit
[root@hh-yun-ceph-cinder015-128055 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder015-128055
Starting Ceph mon.hh-yun-ceph-cinder015-128055 on hh-yun-ceph-cinder015-128055...
Running as unit run-53061.service.
Starting ceph-create-keys on hh-yun-ceph-cinder015-128055...
[root@hh-yun-ceph-cinder017-128057 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder017-128057
Starting Ceph mon.hh-yun-ceph-cinder017-128057 on hh-yun-ceph-cinder017-128057...
Running as unit run-52736.service.
Starting ceph-create-keys on hh-yun-ceph-cinder017-128057...
[root@hh-yun-ceph-cinder024-128074 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder024-128074
Starting Ceph mon.hh-yun-ceph-cinder024-128074 on hh-yun-ceph-cinder024-128074...
Running as unit run-51440.service.
Starting ceph-create-keys on hh-yun-ceph-cinder024-128074...
