ceph 安全迁移

    ceph迁移有很多种方案。最简单的就是上一台新的，拆一台旧的。但是这样的方法不安全，有可能在数据迁移到一半的时候出现异常而老的数据在迁移回去费时费力而且又可能出现新的问题。所以今天分享一个比较安全的迁移方案。

1    设计方案

    1.1 准备硬件

            硬件准备包括安装操作系统，防火墙关闭，插入磁盘，分配网络，安装ceph软件。

    1.2 迁移规划

迁移前：

主机	IP	组件
ceph-admin	172.18.0.131	mon,osd
ceph-node1	172.18.0.132	mon,osd
ceph-node2	172.18.0.133	mon,osd

迁移后：

主机	IP	组件
ceph-admin	172.18.0.131	mon
ceph-node1	172.18.0.132	mon
ceph-node2	172.18.0.133	mon
transfer01	172.18.0.135	osd
transfer02	172.18.0.34	osd
transfer03	172.18.0.51	osd

2 迁移原理

    迁移原理基于ceph 的crush 伪随机原理。简单的说就是当副本数增加的时候原来bucket中的数据不迁移，只是拷贝一份副本到新的bucket中。当副本数减少的时候 也是把指定的bucket中的数据删除，其他bucket中的数据不会迁移。

   2.1  搭建独立的bucket（new_root）

  2.2  在老的 bucket (default) 中选择3个副本

  2.3  修改pool的副本数增加到6副本

  2.3  在新的bucket（new_root）中再选择3个副本 等待从老的bucket中选出的3个副本中的数据 拷贝到另外新bucket中的3副本上，达到6副本的效果

  2.4  在新的bucket（new_root） 中选择3个副本（等于是放弃老bucket中的3个副本的数据）并修改pool的副本数从6副本降低至3副本（自动删除老bucket的3副本中的数据保留新bucket中的3副本数据从而达到无缝安全迁移的目的）

   优点：

    1  安全。在最后一刻修改副本数自动删除老bucket数据之前老的bucket中的数据从来都没有迁移过，不用担心因为迁移过程中的异常而丢损坏数据。

    2  省时。每一步都是可逆的，如果再操作过程中出现任何问题只用简单的修改crushmap和副本数 就达到了回退的目的，避免在迁移过程中回退花费大量的时间和精力。

操作步骤	操作命令	回退命令	集群数据变化	是否有风险
设置规则1	ceph osd pool set [pool-name]  crush_rule replicated_rule_1	ceph osd pool set [pool-name] set crush_rule replicated_rule	无变化	无
修改pool 副本数	ceph osd pool set [pool-name]  size 6	ceph osd pool set [pool-name]  size 3	无变化	无
设置规则2	ceph osd pool set [pool-name]  crush_rule replicated_rule_2	ceph osd pool set [pool-name]  crush_rule replicated_rule_1	数据从老bucket树拷贝到新bucket树	极低
设置规则3	ceph osd pool set [pool-name]  crush_rule replicated_rule_3	ceph osd pool set [pool-name]  crush_rule replicated_rule_2	老bucket树中删除数据	极低
修改pool副本数	ceph osd pool set [pool-name]  size 3	无	无变化	无

在设置规则3前老的数据都未挪动半步，不影响业务使用也不担心数据丢失和损坏。当执行步骤3的的时候就表示步骤2已经完全拷贝了一份数据导新bucket中，全程安全无缝对接。

 

3 迁移实施

    3.1 准备基础环境

# 准备磁盘 [root@transfer01 ~]# lsblk|grep vda vda 252:0 0 100G 0 disk # 更新yum [root@transfer01 yum.repos.d]# ll|grep ceph.repo -rw-r--r--. 1 root root 614 Apr 1 14:34 ceph.repo [root@transfer01 yum.repos.d]# ll|grep epel.repo -rw-r--r--. 1 root root 921 Apr 1 14:34 epel.repo # 安装ceph 软件 [root@transfer01 yum.repos.d]# yum install ceph net-tools vim -y # 防火墙关闭 [root@transfer01 yum.repos.d]# systemctl stop firewalld && systemctl disable firewalld 

   3.2 osd 初始化

# 修改集群参数osd_crush_update_on_start 这一点最关键一定不可忘记，否则迁移失败 在ceph.conf 中添加参数osd_crush_update_on_start = false，阻止因为osd的变化而引起crush的变化 添加参数后重启mon服务使得参数生效。（可以使用ceph --show-config | grep osd_crush_update_on_start 验证参数是否改变） # 初始化osd ceph-deploy osd create transfer01 --data /dev/vda ceph-deploy osd create transfer02 --data /dev/vda ceph-deploy osd create transfer03 --data /dev/vda # 初始化后的ceph tree 是这样的 [root@ceph-admin ceph]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0 root new_root -1 0.14699 root default -3 0.04900 host ceph-admin 0 hdd 0.04900 osd.0 up 1.00000 1.00000 -5 0.04900 host ceph-node1 1 hdd 0.04900 osd.1 up 1.00000 1.00000 -7 0.04900 host ceph-node2 2 hdd 0.04900 osd.2 up 1.00000 1.00000 3 hdd 0 osd.3 up 1.00000 1.00000 4 hdd 0 osd.4 up 1.00000 1.00000 5 hdd 0 osd.5 up 1.00000 1.00000

   3.3 搭建新的bucket 树

# 添加 新bucket 树中的叶子节点 [root@ceph-admin ceph]# ceph osd crush add-bucket new_root root [root@ceph-admin ceph]# ceph osd crush add-bucket transfer01 host added bucket transfer01 type host to crush map [root@ceph-admin ceph]# ceph osd crush add-bucket transfer02 host added bucket transfer02 type host to crush map [root@ceph-admin ceph]# ceph osd crush add-bucket transfer03 host added bucket transfer03 type host to crush map [root@ceph-admin ceph]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -13 0 host transfer03 -12 0 host transfer02 -11 0 host transfer01 -9 0 root new_root -1 0.14699 root default -3 0.04900 host ceph-admin 0 hdd 0.04900 osd.0 up 1.00000 1.00000 -5 0.04900 host ceph-node1 1 hdd 0.04900 osd.1 up 1.00000 1.00000 -7 0.04900 host ceph-node2 2 hdd 0.04900 osd.2 up 1.00000 1.00000 3 hdd 0 osd.3 up 1.00000 1.00000 4 hdd 0 osd.4 up 1.00000 1.00000 5 hdd 0 osd.5 up 1.00000 1.00000 # host级别bucket 添加到root 级别bucket下 [root@ceph-admin ceph]# ceph osd crush move transfer01 root=new_root moved item id -11 name 'transfer01' to location {root=new_root} in crush map [root@ceph-admin ceph]# ceph osd crush move transfer02 root=new_root moved item id -12 name 'transfer02' to location {root=new_root} in crush map [root@ceph-admin ceph]# ceph osd crush move transfer03 root=new_root moved item id -13 name 'transfer03' to location {root=new_root} in crush map [root@ceph-admin ceph]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0 root new_root -11 0 host transfer01 -12 0 host transfer02 -13 0 host transfer03 -1 0.14699 root default -3 0.04900 host ceph-admin 0 hdd 0.04900 osd.0 up 1.00000 1.00000 -5 0.04900 host ceph-node1 1 hdd 0.04900 osd.1 up 1.00000 1.00000 -7 0.04900 host ceph-node2 2 hdd 0.04900 osd.2 up 1.00000 1.00000 3 hdd 0 osd.3 up 1.00000 1.00000 4 hdd 0 osd.4 up 1.00000 1.00000 5 hdd 0 osd.5 up 1.00000 1.00000 # osd 级别bucket 添加到host级别bucket 下，完成搭建一个新的bucket树 [root@ceph-admin ceph]# ceph osd crush add osd.3 0.049 host=transfer01 add item id 3 name 'osd.3' weight 0.049 at location {host=transfer01} to crush map [root@ceph-admin ceph]# ceph osd crush add osd.4 0.049 host=transfer02 add item id 4 name 'osd.4' weight 0.049 at location {host=transfer02} to crush map [root@ceph-admin ceph]# ceph osd crush add osd.5 0.049 host=transfer03 add item id 5 name 'osd.5' weight 0.049 at location {host=transfer03} to crush map [root@ceph-admin ceph]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -9 0.14699 root new_root -11 0.04900 host transfer01 3 hdd 0.04900 osd.3 up 1.00000 1.00000 -12 0.04900 host transfer02 4 hdd 0.04900 osd.4 up 1.00000 1.00000 -13 0.04900 host transfer03 5 hdd 0.04900 osd.5 up 1.00000 1.00000 -1 0.14699 root default -3 0.04900 host ceph-admin 0 hdd 0.04900 osd.0 up 1.00000 1.00000 -5 0.04900 host ceph-node1 1 hdd 0.04900 osd.1 up 1.00000 1.00000 -7 0.04900 host ceph-node2 2 hdd 0.04900 osd.2 up 1.00000 1.00000

  3.4 编辑crushmap 并更新

# 获取crushmap [root@ceph-admin opt]# ceph osd getcrushmap -o /opt/map2 59 # 使用crushtools 镜像反编译成可读文档 [root@ceph-admin opt]# crushtool -d /opt/map2 -o /opt/map2.txt 

# 编辑/opt/map2.txt vim /opt/map2.txt # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable chooseleaf_stable 1 tunable straw_calc_version 1 tunable allowed_bucket_algs 54 # devices device 0 osd.0 class hdd device 1 osd.1 class hdd device 2 osd.2 class hdd #### 加上初始化好的osd bucket device 3 osd.3 class hdd device 4 osd.4 class hdd device 5 osd.5 class hdd ### 加上host bucket …… host transfer01 { id -11 # do not change unnecessarily id -16 class hdd # do not change unnecessarily # weight 0.049 alg straw2 hash 0 # rjenkins1 item osd.3 weight 0.049 } host transfer02 { id -12 # do not change unnecessarily id -15 class hdd # do not change unnecessarily # weight 0.049 alg straw2 hash 0 # rjenkins1 item osd.4 weight 0.049 } host transfer03 { id -13 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.049 alg straw2 hash 0 # rjenkins1 item osd.5 weight 0.049 } ### 添加新搭建独立的bucket结构new_root root new_root { id -9 # do not change unnecessarily id -10 class hdd # do not change unnecessarily # weight 0.147 alg straw2 hash 0 # rjenkins1 item transfer01 weight 0.049 item transfer02 weight 0.049 item transfer03 weight 0.049 } ### 新增3条规则 # 规则1 在老的bucket结构中选出3副本（默认就是3副本，所以这里系统没有变化） rule replicated_rule_1 { id 1 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 3 type host step emit } # 规则2 在老的bucket结构中选出3副本并且在新bucket结构中选出3副本 rule replicated_rule_2 { id 2 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 3 type host step emit step take new_root step chooseleaf firstn 3 type host step emit } # 规则3 在新的bucket结构中选出3副本（等于是废弃老的bucket结构） rule replicated_rule_3 { id 3 type replicated min_size 1 max_size 10 step take new_root step chooseleaf firstn 3 type host step emit } 

#编译更新后的crushmap可编辑文件/opt/map2.txt 为map2.bin [root@ceph-admin opt]# crushtool -c /opt/map2.txt -o /opt/map2.bin # 注入新的crushmap [root@ceph-admin opt]# ceph osd setcrushmap -i /opt/map2.bin 60

  3.5 开始迁移

# 修改pool的crush_rule规则为我们新加的规则1，我们用rbd pool做演示，真实环境下每一个pool都要操作 # 当前rbd crush_rule 是默认的0 [root@ceph-admin ~]# ceph osd dump|grep rbd|grep crush_rule pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 398 flags hashpspool stripe_width 0 # 设置crush_rule 为修改的crushmap 文件中的规则1（replicated_rule_1） [root@ceph-admin ~]# ceph osd pool set rbd crush_rule replicated_rule_1 set pool 1 crush_rule to replicated_rule_1 # 由于默认的规则和规则1相同，数据结构不变化 [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_OK services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.32GiB used, 444GiB / 450GiB avail pgs: 144 active+clean # 设置pool中的副本数为6副本 [root@ceph-admin ~]# ceph osd pool set rbd size 6 set pool 1 size to 6 # 由于规则1指定的bucket是从老的bucket树中选择3个副本，但是当前修改pool的副本数为6副本，不能满足，所以ceph状态为active+undersized [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_WARN Degraded data redundancy: 32 pgs undersized services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.33GiB used, 444GiB / 450GiB avail pgs: 112 active+clean 32 active+undersized # 设置crush_rule 为修改的规则2，规则2开始数据拷贝 [root@ceph-admin ~]# ceph osd pool set rbd crush_rule replicated_rule_2 set pool 1 crush_rule to replicated_rule_2 # 数据集开始拷贝（迁移） [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_WARN Reduced data availability: 13 pgs peering services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.33GiB used, 444GiB / 450GiB avail pgs: 9.028% pgs not active 131 active+clean 13 peering # 数据拷贝完成后 新老bucket树中都各有3个完整的副本，达到6副本。ceph集群状态又恢复到active+clean [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_OK services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.33GiB used, 444GiB / 450GiB avail pgs: 144 active+clean # 修改规则为3 [root@ceph-admin ~]# ceph osd pool set rbd crush_rule replicated_rule_3 set pool 1 crush_rule to replicated_rule_3 # 由于规则3是 只保留在新bucket树中的数据，系统会自动删除老bucket中的3副本数据，但是这个时候pool 中的副本数还是6副本所以集群状态active+clean+remapped [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_OK services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in; 32 remapped pgs rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.33GiB used, 444GiB / 450GiB avail pgs: 112 active+clean 32 active+clean+remapped io: client: 4.00KiB/s rd, 0B/s wr, 3op/s rd, 2op/s wr # 修改pool的副本为3副本， [root@ceph-admin ~]# ceph osd pool set rbd size 3 set pool 1 size to 3 # 上面pool的副本数 大于 实际的副本数 导致状态为active+clean+remapped ，修改pool副本数为3后，状态恢复健康 [root@ceph-admin ~]# ceph -s cluster: id: e3a671b9-9bf2-4f25-9c04-af79b5cffc7a health: HEALTH_OK services: mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 mgr: ceph-admin(active), standbys: ceph-node1, ceph-node2 mds: cephfs-1/1/1 up {0=ceph-admin=up:active}, 2 up:standby osd: 6 osds: 6 up, 6 in rgw: 3 daemons active data: pools: 9 pools, 144 pgs objects: 347 objects, 5.79MiB usage: 6.33GiB used, 444GiB / 450GiB avail pgs: 144 active+clean io: client: 4.00KiB/s rd, 0B/s wr, 3op/s rd, 2op/s wr 

  3.6 检查验证

  

rbd 池中的pg 已经迁移到了新的osd3,4,5中，符合迁移要求。