您现在的位置是：首页 > 文章详情

Ceph学习笔记2-在Kolla-Ansible中使用Ceph后端存储

日期：2018-10-23点击：446收藏

环境说明

使用Kolla-Ansible请参考《使用Kolla-Ansible在CentOS 7单节点上部署OpenStack Pike》；
部署Ceph服务请参考《Ceph学习笔记1-Mimic版本多节点部署》。

配置Ceph

以osdev用户登录：

$ ssh osdev@osdev01 $ cd /opt/ceph/deploy/

创建Pool

创建镜像Pool

用于保存Glance镜像：

$ ceph osd pool create images 32 32 pool 'images' created

创建卷Pool

用于保存Cinder的卷：

$ ceph osd pool create volumes 32 32 pool 'volumes' created

用于保存Cinder的卷备份：

$ ceph osd pool create backups 32 32 pool 'backups' created

创建虚拟机Pool

用于保存虚拟机系统卷：

$ ceph osd pool create vms 32 32 pool 'vms' created

查看Pool

$ ceph osd lspools 1 .rgw.root 2 default.rgw.control 3 default.rgw.meta 4 default.rgw.log 6 rbd 8 images 9 volumes 10 backups 11 vms

创建用户

查看用户

查看所有用户：

$ ceph auth list installed auth entries: mds.osdev01 key: AQCabn5b18tHExAAkZ6Aq3IQ4/aqYEBBey5O3Q== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx mds.osdev02 key: AQCbbn5bcq4yJRAAUfhoqPNfyp2m/ORu/7vHBA== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx mds.osdev03 key: AQCcbn5bTAIdORAApGu9NJvC3AmS+L3EWXLMdw== caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow rwx osd.0 key: AQCyJH5bG2ZBHRAAsDaLHcoOxv/mLCHwITA7JQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQDTJH5bjvQ8HxAA4cyLttvZwiqFq1srFoSXWg== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQD9JH5bbPi6IRAA7DbwaCh5JBaa6RfWPoe9VQ== caps: [mgr] allow profile osd caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQA1In5bZkxwGBAA9bBLE5/NKstK1CRMzfGgKQ== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQA1In5boIRwGBAAgj5OccvTGYkuB+btlgL0BQ== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQA1In5bS6pwGBAA379v3LXJrdURLmA1gnTaLQ== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQA1In5bnMpwGBAAXohUfa4rGS0Rd2weMl4dPg== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQA1In5buelwGBAANQSalrSzH3yslSc4rYPu1g== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rgw key: AQA1In5b0ghxGBAAIGK3WmBSkKZMnSEfvnEQow== caps: [mon] allow profile bootstrap-rgw client.rgw.osdev01 key: AQDZbn5b6aChEBAAzRuX4UWlxyws+aX1i+D26Q== caps: [mon] allow rw caps: [osd] allow rwx client.rgw.osdev02 key: AQDabn5bypCDJBAAt18L5ppG5lEg6NkGQLYs5w== caps: [mon] allow rw caps: [osd] allow rwx client.rgw.osdev03 key: AQDbbn5bbEVNNBAArX+/AKQu9q3hCRn/05Ya3A== caps: [mon] allow rw caps: [osd] allow rwx mgr.osdev01 key: AQDPIn5beqPTORAAEzcX3fMCCclLR2RiPyvugw== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.osdev02 key: AQDRIn5bLRVqDxAA/yWXO8pX6fQynJNyCcoNww== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.osdev03 key: AQDSIn5bGyrhHxAAvtAEOveovRxmdDlF45i2Cg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *

查看指定用户：

$ ceph auth get client.admin exported keyring for client.admin [client.admin] key = AQA1In5bZkxwGBAA9bBLE5/NKstK1CRMzfGgKQ== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *"

创建Glance用户

创建glance用户，并授予images存储池访问权限：

$ ceph auth get-or-create client.glance [client.glance] key = AQBQq4NboVHdGxAAlfK2WJkiZMolluATpvOviQ== $ ceph auth caps client.glance mon 'allow r' osd 'allow rwx pool=images' updated caps for client.glance

查看并保存glance用户的KeyRing文件：

$ ceph auth get client.glance exported keyring for client.glance [client.glance] key = AQBQq4NboVHdGxAAlfK2WJkiZMolluATpvOviQ== caps mon = "allow r" caps osd = "allow rwx pool=images" $ ceph auth get client.glance -o /opt/ceph/deploy/ceph.client.glance.keyring exported keyring for client.glance

创建Cinder用户

创建cinder-volume用户，并授予volumes存储池访问权限：

$ ceph auth get-or-create client.cinder-volume [client.cinder-volume] key = AQBKt4NbqROVIxAACnH+pVv141+wOpgWj14RjA== $ ceph auth caps client.cinder-volume mon 'allow r' osd 'allow rwx pool=volumes' updated caps for client.cinder-volume

查看并保存cinder-volume用户的KeyRing文件：

$ ceph auth get client.cinder-volume exported keyring for client.cinder-volume [client.cinder-volume] key = AQBKt4NbqROVIxAACnH+pVv141+wOpgWj14RjA== caps mon = "allow r" caps osd = "allow rwx pool=volumes" $ ceph auth get client.cinder-volume -o /opt/ceph/deploy/ceph.client.cinder-volume.keyring exported keyring for client.cinder-volume

创建cinder-backup用户，并授予volumes和backups存储池访问权限：

$ ceph auth get-or-create client.cinder-backup [client.cinder-backup] key = AQBit4NbN0rvLRAAYoa4SBM0qvwY8kPo5Md0og== $ ceph auth caps client.cinder-backup mon 'allow r' osd 'allow rwx pool=volumes, allow rwx pool=backups' updated caps for client.cinder-backup

查看并保存cinder-backup用户的KeyRing文件：

$ ceph auth get client.cinder-backup exported keyring for client.cinder-backup [client.cinder-backup] key = AQBit4NbN0rvLRAAYoa4SBM0qvwY8kPo5Md0og== caps mon = "allow r" caps osd = "allow rwx pool=volumes, allow rwx pool=backups" $ ceph auth get client.cinder-backup -o /opt/ceph/deploy/ceph.client.cinder-backup.keyring exported keyring for client.cinder-backup

创建Nova用户

创建nova用户，并授予vms存储池的访问权限：

$ ceph auth get-or-create client.nova [client.nova] key = AQD7tINb4A58GRAA7CsAM9EAwFwtIpTdQFGO7A== $ ceph auth caps client.nova mon 'allow r' osd 'allow rwx pool=vms' updated caps for client.nova

查看并保存nova用户的KeyRing文件：

$ ceph auth get client.nova exported keyring for client.nova [client.nova] key = AQD7tINb4A58GRAA7CsAM9EAwFwtIpTdQFGO7A== caps mon = "allow r" caps osd = "allow rwx pool=vms" $ ceph auth get client.nova -o /opt/ceph/deploy/ceph.client.nova.keyring exported keyring for client.nova

配置Kolla-Ansible

以root用户身份登录osdev01部署节点，并设置好环境变量：

$ ssh root@osdev01 $ export KOLLA_ROOT=/opt/kolla $ cd ${KOLLA_ROOT}/myconfig

全局配置

编辑globals.yml，禁止部署Ceph：

enable_ceph: "no"

开启Cinder服务，并开启Glance、Cinder和Nova的后端Ceph功能：

enable_cinder: "yes" glance_backend_ceph: "yes" cinder_backend_ceph: "yes" nova_backend_ceph: "yes"

配置Glance

配置Glance使用glance用户使用Ceph的images存储池：

$ mkdir -pv config/glance mkdir: 已创建目录 "config/glance" $ vi config/glance/glance-api.conf [glance_store] stores = rbd default_store = rbd rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf

新增Glance的Ceph客户端配置和glance用户的KeyRing文件：

$ vi config/glance/ceph.conf [global] fsid = 383237bd-becf-49d5-9bd6-deb0bc35ab2a mon_initial_members = osdev01, osdev02, osdev03 mon_host = 172.29.101.166,172.29.101.167,172.29.101.168 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx $ cp -v /opt/ceph/deploy/ceph.client.glance.keyring config/glance/ceph.client.glance.keyring "/opt/ceph/deploy/ceph.client.glance.keyring" -> "config/glance/ceph.client.glance.keyring"

配置Cinder

配置Cinder卷服务使用Ceph的cinder-volume用户使用volumes存储池，Cinder卷备份服务使用Ceph的cinder-backup用户使用backups存储池：

$ mkdir -pv config/cinder/ mkdir: 已创建目录 "config/cinder/" $ vi config/cinder/cinder-volume.conf [DEFAULT] enabled_backends=rbd-1 [rbd-1] rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=cinder-volume backend_host=rbd:volumes rbd_pool=volumes volume_backend_name=rbd-1 volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_secret_uuid = {{ cinder_rbd_secret_uuid }} $ vi config/cinder/cinder-backup.conf [DEFAULT] backup_ceph_conf=/etc/ceph/ceph.conf backup_ceph_user=cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool=backups backup_driver = cinder.backup.drivers.ceph backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true

新增Cinder的卷服务和卷备份服务的Ceph客户端配置和KeyRing文件：

$ cp config/glance/ceph.conf config/cinder/ceph.conf $ mkdir -pv config/cinder/cinder-backup/ config/cinder/cinder-volume/ mkdir: 已创建目录 "config/cinder/cinder-backup/" mkdir: 已创建目录 "config/cinder/cinder-volume/" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/cinder/cinder-backup/ceph.client.cinder-volume.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/cinder/cinder-backup/ceph.client.cinder-volume.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-backup.keyring config/cinder/cinder-backup/ceph.client.cinder-backup.keyring "/opt/ceph/deploy/ceph.client.cinder-backup.keyring" -> "config/cinder/cinder-backup/ceph.client.cinder-backup.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/cinder/cinder-volume/ceph.client.cinder-volume.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/cinder/cinder-volume/ceph.client.cinder.keyring"

配置Nova

配置Nova使用Ceph的nova用户使用vms存储池：

$ vi config/nova/nova-compute.conf [libvirt] images_rbd_pool=vms images_type=rbd images_rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=nova

新增Nova的Ceph客户端配置和nova用户的KeyRing文件：

$ cp -v config/glance/ceph.conf config/nova/ceph.conf "config/glance/ceph.conf" -> "config/nova/ceph.conf" $ cp -v /opt/ceph/deploy/ceph.client.nova.keyring config/nova/ceph.client.nova.keyring "/opt/ceph/deploy/ceph.client.nova.keyring" -> "config/nova/ceph.client.nova.keyring" $ cp -v /opt/ceph/deploy/ceph.client.cinder-volume.keyring config/nova/ceph.client.cinder.keyring "/opt/ceph/deploy/ceph.client.cinder-volume.keyring" -> "config/nova/ceph.client.cinder.keyring"

部署测试

开始部署

编辑部署脚本osdev.sh：

#!/bin/bash set -uexv usage() { echo -e "usage : \n$0 <action>" echo -e " \$1 action" } if [ $# -lt 1 ]; then usage exit 1 fi ${KOLLA_ROOT}/kolla-ansible/tools/kolla-ansible --configdir ${KOLLA_ROOT}/myconfig --passwords ${KOLLA_ROOT}/myconfig/passwords.yml --inventory ${KOLLA_ROOT}/myconfig/mynodes.conf $1

增加可执行权限：

$ chmod a+x osdev.sh

部署OpenStack集群：

$ ./osdev.sh bootstrap-servers $ ./osdev.sh prechecks $ ./osdev.sh pull $ ./osdev.sh deploy $ ./osdev.sh post-deploy # ./osdev.sh "destroy --yes-i-really-really-mean-it"

查看部署的服务概况：

$ openstack service list +----------------------------------+-------------+----------------+ | ID | Name | Type | +----------------------------------+-------------+----------------+ | 304c9c5073f14f4a97ca1c3cf5e1b49e | neutron | network | | 46de4440a5cf4a5697fa94b2d0424ba9 | heat | orchestration | | 60b46b491ce7403aaec0c064384dde49 | heat-cfn | cloudformation | | 7726ab5d41c5450d954f073f1a9aff28 | cinderv2 | volumev2 | | 7a4bd5fc12904cc7b5c3810412f98c57 | gnocchi | metric | | 7ae6f98018fb4d509e862e45ebf10145 | glance | image | | a0ec333149284c09ac0e157753205fd6 | nova | compute | | b15e90c382864723945b15c37d3317a6 | placement | placement | | b5eaa49c50d64316b583eb1c0c4f9ce2 | cinderv3 | volumev3 | | c6474640f5d9424da0ec51c70c1e6e01 | nova_legacy | compute_legacy | | db27eb8524be4db3be12b9dd0dab16b8 | keystone | identity | | edf5c8b894a74a69b65bb49d8e014fff | cinder | volume | +----------------------------------+-------------+----------------+ $ openstack volume service list +------------------+-------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-------------------+------+---------+-------+----------------------------+ | cinder-scheduler | osdev02 | nova | enabled | up | 2018-08-27T11:33:27.000000 | | cinder-volume | rbd:volumes@rbd-1 | nova | enabled | up | 2018-08-27T11:33:18.000000 | | cinder-backup | osdev02 | nova | enabled | up | 2018-08-27T11:33:17.000000 | +------------------+-------------------+------+---------+-------+----------------------------+

初始化环境

查看初始的RBD存储池情况，全部是空的：

$ rbd -p images ls $ rbd -p volumes ls $ rbd -p vms ls

设置环境变量，并初始化OpenStack环境：

$ . ${KOLLA_ROOT}/myconfig/admin-openrc.sh $ ${KOLLA_ROOT}/myconfig/init-runonce

查看新增的镜像信息：

$ openstack image list +--------------------------------------+--------+--------+ | ID | Name | Status | +--------------------------------------+--------+--------+ | 293b25bb-30be-4839-b4e2-1dba3c43a56a | cirros | active | +--------------------------------------+--------+--------+ $ openstack image show 293b25bb-30be-4839-b4e2-1dba3c43a56a +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | 443b7623e27ecf03dc9e01ee93f67afe | | container_format | bare | | created_at | 2018-08-27T11:25:29Z | | disk_format | qcow2 | | file | /v2/images/293b25bb-30be-4839-b4e2-1dba3c43a56a/file | | id | 293b25bb-30be-4839-b4e2-1dba3c43a56a | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | 68ada1726a864e2081a56be0a2dca3a0 | | properties | locations='[{u'url': u'rbd://383237bd-becf-49d5-9bd6-deb0bc35ab2a/images/293b25bb-30be-4839-b4e2-1dba3c43a56a/snap', u'metadata': {}}]', os_type='linux' | | protected | False | | schema | /v2/schemas/image | | size | 12716032 | | status | active | | tags | | | updated_at | 2018-08-27T11:25:30Z | | virtual_size | None | | visibility | public | +------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

查看RBD存储池的变化，可见镜像被存储在images存储池中，并且有一个快照：

$ rbd -p images ls 293b25bb-30be-4839-b4e2-1dba3c43a56a $ rbd -p volumes ls $ rbd -p vms ls $ rbd -p images info 293b25bb-30be-4839-b4e2-1dba3c43a56a rbd image '293b25bb-30be-4839-b4e2-1dba3c43a56a': size 12 MiB in 2 objects order 23 (8 MiB objects) id: 178f4008d95 block_name_prefix: rbd_data.178f4008d95 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Aug 27 19:25:29 2018 $ rbd -p images snap list 293b25bb-30be-4839-b4e2-1dba3c43a56a SNAPID NAME SIZE TIMESTAMP 6 snap 12 MiB Mon Aug 27 19:25:30 2018

创建虚拟机

创建一个虚拟机：

$ openstack server create --image cirros --flavor m1.tiny --key-name mykey --nic net-id=9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 demo1 +-------------------------------------+-----------------------------------------------+ | Field | Value | +-------------------------------------+-----------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | 65cVBJ7S6yaD | | config_drive | | | created | 2018-08-27T11:29:03Z | | flavor | m1.tiny (1) | | hostId | | | id | 309f1364-4d58-413d-a865-dfc37ff04308 | | image | cirros (293b25bb-30be-4839-b4e2-1dba3c43a56a) | | key_name | mykey | | name | demo1 | | progress | 0 | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2018-08-27T11:29:03Z | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | | volumes_attached | | +-------------------------------------+-----------------------------------------------+ $ openstack server show 309f1364-4d58-413d-a865-dfc37ff04308 +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | osdev03 | | OS-EXT-SRV-ATTR:hypervisor_hostname | osdev03 | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2018-08-27T11:29:16.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | demo-net=10.0.0.11 | | config_drive | | | created | 2018-08-27T11:29:03Z | | flavor | m1.tiny (1) | | hostId | 4e345dd9f770f63f80d3eafe97c20d97746e890b2971a8398e26db86 | | id | 309f1364-4d58-413d-a865-dfc37ff04308 | | image | cirros (293b25bb-30be-4839-b4e2-1dba3c43a56a) | | key_name | mykey | | name | demo1 | | progress | 0 | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2018-08-27T11:29:16Z | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+

可见虚拟机在vms存储池中创建了一个卷：

$ rbd -p images ls 293b25bb-30be-4839-b4e2-1dba3c43a56a $ rbd -p volumes ls $ rbd -p backups ls $ rbd -p vms ls 309f1364-4d58-413d-a865-dfc37ff04308_disk

登录虚拟机所在节点，可以看到虚拟机的系统卷使用的是在vms中创建的这个卷，从进程参数可以看出qemu直接使用的是Ceph的librbd库访问的RBD块设备：

$ ssh osdev@osdev03 $ sudo docker exec -it nova_libvirt virsh list Id Name State ---------------------------------------------------- 1 instance-00000001 running $ sudo docker exec -it nova_libvirt virsh dumpxml 1 ... <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='nova'> <secret type='ceph' uuid='2ea5db42-c8f1-4601-927c-3c64426907aa'/> </auth> <source protocol='rbd' name='vms/309f1364-4d58-413d-a865-dfc37ff04308_disk'> <host name='172.29.101.166' port='6789'/> <host name='172.29.101.167' port='6789'/> <host name='172.29.101.168' port='6789'/> </source> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> ... $ ps -aux | grep qemu 42436 2678909 4.6 0.0 1341144 171404 ? Sl 19:29 0:08 /usr/libexec/qemu-kvm -name guest=instance-00000001,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,avx512f=on,avx512dq=on,clflushopt=on,clwb=on,avx512cd=on,avx512bw=on,avx512vl=on,pku=on,stibp=on,pdpe1gb=on -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 309f1364-4d58-413d-a865-dfc37ff04308 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=17.0.2,serial=74bf926c-70b7-03df-b211-d21d6016081a,uuid=309f1364-4d58-413d-a865-dfc37ff04308,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-instance-00000001/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -object secret,id=virtio-disk0-secret0,data=zNy84nlNYigA4vjbuOxcGQa1/hh8w28i/WoJbO1Xsl4=,keyid=masterKey0,iv=OhX+FApyFyq2XLWq0ff/Ew==,format=base64 -drive file=rbd:vms/309f1364-4d58-413d-a865-dfc37ff04308_disk:id=nova:auth_supported=cephx\;none:mon_host=172.29.101.166\:6789\;172.29.101.167\:6789\;172.29.101.168\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=79,id=hostnet0,vhost=on,vhostfd=80 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:04:e8:e9,bus=pci.0,addr=0x3 -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/309f1364-4d58-413d-a865-dfc37ff04308/console.log,logappend=off -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.29.101.168:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on $ ldd /usr/libexec/qemu-kvm | grep -e ceph -e rbd librbd.so.1 => /lib64/librbd.so.1 (0x00007fde38815000) libceph-common.so.0 => /usr/lib64/ceph/libceph-common.so.0 (0x00007fde28247000)

创建卷

创建一个卷：

$ openstack volume create --size 1 volume1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2018-08-27T11:33:52.000000 | | description | None | | encrypted | False | | id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 | | migration_status | None | | multiattach | False | | name | volume1 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | None | | updated_at | None | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | +---------------------+--------------------------------------+

查看存储池状态，可以看到新建的卷被放在volumes存储池：

$ rbd -p images ls 293b25bb-30be-4839-b4e2-1dba3c43a56a $ rbd -p volumes ls volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 $ rbd -p backups ls $ rbd -p vms ls 309f1364-4d58-413d-a865-dfc37ff04308_disk

创建备份

创建一个卷备份，可以看到是创建在backups存储池中：

$ openstack volume backup create 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | f2321578-88d5-4337-b93c-798855b817ce | | name | None | +-------+--------------------------------------+ $ openstack volume backup list +--------------------------------------+------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------+-------------+-----------+------+ | f2321578-88d5-4337-b93c-798855b817ce | None | None | available | 1 | +--------------------------------------+------+-------------+-----------+------+ $ openstack volume backup show f2321578-88d5-4337-b93c-798855b817ce +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | availability_zone | nova | | container | backups | | created_at | 2018-08-27T11:39:40.000000 | | data_timestamp | 2018-08-27T11:39:40.000000 | | description | None | | fail_reason | None | | has_dependent_backups | False | | id | f2321578-88d5-4337-b93c-798855b817ce | | is_incremental | False | | name | None | | object_count | 0 | | size | 1 | | snapshot_id | None | | status | available | | updated_at | 2018-08-27T11:39:46.000000 | | volume_id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 | +-----------------------+--------------------------------------+ $ rbd -p backups ls volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base

在此创建一个备份，发现backups存储池并无变化，仅仅是在原有的备份卷中增加一个快照：

$ volume backup create 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 07132063-9bdb-4391-addd-a791dae2cfea | | name | None | +-------+--------------------------------------+ $ rbd -p backups ls volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base $ rbd -p backups snap list volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9.backup.base SNAPID NAME SIZE TIMESTAMP 4 backup.f2321578-88d5-4337-b93c-798855b817ce.snap.1535369984.08 1 GiB Mon Aug 27 19:39:46 2018 5 backup.07132063-9bdb-4391-addd-a791dae2cfea.snap.1535370126.76 1 GiB Mon Aug 27 19:42:08 2018

连接卷

把新增的卷链接到之前创建的虚拟机中：

$ openstack server add volume demo1 volume1 $ openstack volume show volume1 +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{u'server_id': u'309f1364-4d58-413d-a865-dfc37ff04308', u'attachment_id': u'fb4d9ec0-8a33-4ed0-8845-09e6f17aac81', u'attached_at': u'2018-08-27T11:44:51.000000', u'host_name': u'osdev03', u'volume_id': u'3ccca300-bee3-4b5a-b89b-32e6b8b806d9', u'device': u'/dev/vdb', u'id': u'3ccca300-bee3-4b5a-b89b-32e6b8b806d9'}] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2018-08-27T11:33:52.000000 | | description | None | | encrypted | False | | id | 3ccca300-bee3-4b5a-b89b-32e6b8b806d9 | | migration_status | None | | multiattach | False | | name | volume1 | | os-vol-host-attr:host | rbd:volumes@rbd-1#rbd-1 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 68ada1726a864e2081a56be0a2dca3a0 | | properties | attached_mode='rw' | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | in-use | | type | None | | updated_at | 2018-08-27T11:44:52.000000 | | user_id | c7111728fbbd4fd79bdd2b60e7d7cb42 | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

到虚拟机所在节点查看其libvirt上参数的变化，发现新增了一个RBD磁盘：

$ sudo docker exec -it nova_libvirt virsh dumpxml 1 ... <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='nova'> <secret type='ceph' uuid='2ea5db42-c8f1-4601-927c-3c64426907aa'/> </auth> <source protocol='rbd' name='vms/309f1364-4d58-413d-a865-dfc37ff04308_disk'> <host name='172.29.101.166' port='6789'/> <host name='172.29.101.167' port='6789'/> <host name='172.29.101.168' port='6789'/> </source> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none' discard='unmap'/> <auth username='cinder-volume'> <secret type='ceph' uuid='3fa55f7c-b556-4095-9253-b908d5408ec8'/> </auth> <source protocol='rbd' name='volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9'> <host name='172.29.101.166' port='6789'/> <host name='172.29.101.167' port='6789'/> <host name='172.29.101.168' port='6789'/> </source> <target dev='vdb' bus='virtio'/> <serial>3ccca300-bee3-4b5a-b89b-32e6b8b806d9</serial> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> ...

为虚拟机创建一个浮动IP，使用SSH登陆进去：

$ openstack console url show demo1 +-------+-------------------------------------------------------------------------------------+ | Field | Value | +-------+-------------------------------------------------------------------------------------+ | type | novnc | | url | http://172.29.101.167:6080/vnc_auto.html?token=9f835216-1c53-41ae-849a-44a85429a334 | +-------+-------------------------------------------------------------------------------------+ $ openstack floating ip create public1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2018-08-27T11:49:02Z | | description | | | fixed_ip_address | None | | floating_ip_address | 192.168.162.52 | | floating_network_id | ff69b3ff-c2c4-4474-a7ba-952fa99df919 | | id | 2aa86075-9c62-49f5-84ac-e7b6353c9591 | | name | 192.168.162.52 | | port_id | None | | project_id | 68ada1726a864e2081a56be0a2dca3a0 | | qos_policy_id | None | | revision_number | 0 | | router_id | None | | status | DOWN | | subnet_id | None | | tags | [] | | updated_at | 2018-08-27T11:49:02Z | +---------------------+--------------------------------------+ $ openstack server add floating ip demo1 192.168.162.52 $ openstack server list +--------------------------------------+-------+--------+------------------------------------+--------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------+--------+------------------------------------+--------+---------+ | 309f1364-4d58-413d-a865-dfc37ff04308 | demo1 | ACTIVE | demo-net=10.0.0.11, 192.168.162.52 | cirros | m1.tiny | +--------------------------------------+-------+--------+------------------------------------+--------+---------+ $ ssh root@osdev02 $ ip netns qrouter-65759e60-6e20-41cc-a79c-fc492232b127 (id: 1) qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 (id: 0) $ ip netns exec qrouter-65759e60-6e20-41cc-a79c-fc492232b127 ping 192.168.162.50 $ ip netns exec qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 ping 10.0.0.9 （用户名"cirros"，密码"gocubsgo"） $ ip netns exec qrouter-65759e60-6e20-41cc-a79c-fc492232b127 ssh cirros@192.168.162.52 $ ip netns exec qdhcp-9aa15b3e-7084-450f-b0a4-7c905e6bb7c0 ssh cirros@10.0.0.11 $ sudo passwd root Changing password for root New password: Bad password: too weak Retype password: Password for root changed by root $ su - Password:

创建分区并写入测试文件，最后卸载分区：

# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 1G 0 disk |-vda1 253:1 0 1015M 0 part / `-vda15 253:15 0 8M 0 part vdb 253:16 0 1G 0 disk # mkfs.ext4 /dev/vdb mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 262144 4k blocks and 65536 inodes Filesystem UUID: ede8d366-bfbc-4b9a-9d3f-306104f410d7 Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done # mount /dev/vdb /mnt # df -h Filesystem Size Used Available Use% Mounted on /dev 240.1M 0 240.1M 0% /dev /dev/vda1 978.9M 23.9M 914.1M 3% / tmpfs 244.2M 0 244.2M 0% /dev/shm tmpfs 244.2M 92.0K 244.1M 0% /run /dev/vdb 975.9M 1.3M 907.4M 0% /mnt # echo "hello openstack, volume test." > /mnt/ceph_rbd_test # umount /mnt # df -h Filesystem Size Used Available Use% Mounted on /dev 240.1M 0 240.1M 0% /dev /dev/vda1 978.9M 23.9M 914.1M 3% / tmpfs 244.2M 0 244.2M 0% /dev/shm tmpfs 244.2M 92.0K 244.1M 0% /run

断开卷

断开卷，同时查看虚拟机内部变化：

$ openstack server remove volume demo1 volume1 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 1G 0 disk |-vda1 253:1 0 1015M 0 part / `-vda15 253:15 0 8M 0 part

在宿主机映射和挂载RBD卷，并查看之前虚拟机内部创建的文件，完全相同：

$ rbd showmapped id pool image snap device 0 rbd rbd_test - /dev/rbd0 $ rbd feature disable volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 object-map fast-diff deep-flatten $ rbd map volumes/volume-3ccca300-bee3-4b5a-b89b-32e6b8b806d9 /dev/rbd1 $ mkdir /mnt/volume1 $ mount /dev/rbd1 /mnt/volume1/ $ cat /mnt/volume1/ ceph_rbd_test lost+found/ $ cat /mnt/volume1/ceph_rbd_test hello openstack, volume test.