自建ES通过OSS快照迁移至阿里云ES

在同一个VPC内使用3台ECS部署自建ES集群，可以通过给自建Elasticsearch打快照存储到OSS，在利用OSS中的快照进行数据库恢复，将自建Elasticsearch迁移至阿里云Elasticsearch。

安装OSS快照仓库插件，下载插件并解压到ES集群各节点Elasticsearch根目录下的plugins目录

# wget https://github.com/zhichen/elasticsearch-repository-oss/releases/download/v5.5.3/elasticsearch-repository-oss-5.5.3.zip # unzip elasticsearch-repository-oss-5.5.3.zip -d /srv/elasticsearch-5.4.3/plugins/

修改配置文件中的版本号为当前ES集群的版本，完成后需要重启ES各节点。

# cat /srv/elasticsearch-5.4.3/plugins/elasticsearch/plugin-descriptor.properties |grep version |grep -v "^#" version=5.4.3 java.version=1.8 elasticsearch.version=5.4.3

新购与自建ES集群在同一个region的标准存储类型OSS，（阿里云暂不支持归档存储类型OSS）。

在自建ES上创建仓库，OSS所属阿里云主账号AK信息，若使用子账号需要有管理OSS的权限。

PUT _snapshot/robin_backup { "type": "oss", "settings": { "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com", "access_key_id": "Lxxxxxxxxxa", "secret_access_key": "Nxxxxxxxxxxxxxxxxxxpt", "bucket": "shasengtest", "compress": true } }

可以根据实际情况在setting中增加设置

 "chunk_size": "500mb", #限制snapshot过程中分块的大小，超过这个大小，数据将会被分块上传到OSS中 "base_path": "snapshot/" #设置仓库的起始位置，默认为根目录

获取全部仓库信息，设置了robin_backup及robin_backup2两个仓库

# curl -XGET es1:9001/_snapshot/?pretty { "robin_backup" : { "type" : "oss", "settings" : { "bucket" : "shasengtest", "compress" : "true", "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com" } }, "robin_backup2" : { "type" : "oss", "settings" : { "bucket" : "robin7", "compress" : "true", "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com" } } }

删除仓库

DELETE _snapshot/robin_backup1

获取指定仓库信息

# curl -XGET es1:9001/_snapshot/robin_backup?pretty { "robin_backup" : { "type" : "oss", "settings" : { "bucket" : "shasengtest", "compress" : "true", "endpoint" : "http://oss-cn-hangzhou-internal.aliyuncs.com" } } }

备份当前自建ES集群所有打开的索引到指定仓库，创建快照并命名为test

PUT _snapshot/robin_backup/test

获得test快照信息，可以看到备份了当前集群全部的索引test，test1，.kibana

# curl -XGET es1:9001/_snapshot/robin_backup/test?pretty { "snapshots" : [ { "snapshot" : "test", "uuid" : "oXtFY174RyeKRgYFi_dXKA", "version_id" : 5040399, "version" : "5.4.3", "indices" : [ "test", "test1", ".kibana" ], "state" : "SUCCESS", "start_time" : "2018-04-10T12:44:26.027Z", "start_time_in_millis" : 1523364266027, "end_time" : "2018-04-10T12:44:27.204Z", "end_time_in_millis" : 1523364267204, "duration_in_millis" : 1177, "failures" : [ ], "shards" : { "total" : 11, "failed" : 0, "successful" : 11 } } ] }

备份指定test2索引到robin_backup下并命名快照为test2

PUT _snapshot/robin_backup/test2 { "indices": "test2" }

获得单个快照test2信息，指包含了test2这一个索引

# curl -XGET es1:9001/_snapshot/robin_backup/test2?pretty { "snapshots" : [ { "snapshot" : "test2", "uuid" : "L6MknW92RGe55fZS1HIVUw", "version_id" : 5040399, "version" : "5.4.3", "indices" : [ "test2" ], "state" : "SUCCESS", "start_time" : "2018-04-10T12:47:19.430Z", "start_time_in_millis" : 1523364439430, "end_time" : "2018-04-10T12:47:20.099Z", "end_time_in_millis" : 1523364440099, "duration_in_millis" : 669, "failures" : [ ], "shards" : { "total" : 5, "failed" : 0, "successful" : 5 } } ] }

删除索引test及test1，并关闭索引.kibana

# curl -XDELETE es2:9001/test {"acknowledged":true} # curl -XDELETE es2:9001/test1 {"acknowledged":true} # curl -XPOST es2:9001/.kibana/_close {"acknowledged":true} # curl -XGET es2:9001/_cat/indices green open test3 YcU56zKqTNaTNH8Vcr6jpw 5 1 168 0 395.8kb 217.5kb green open test2 M-kOMZVkRcWwOwQlFRKRWw 5 1 139 0 343.1kb 171.5kb close .kibana Lcv2WmUPTxqPjoNJV2C7sQ 

elasticsearch的索引快照与ECS快照不同，并不是打快照那一时刻的整个集群状态。这里可以把索引理解成ECS下挂载的数据盘之一，只需要对重要的数据盘打快照，并不是所有的磁盘都需要打快照，恢复时可以选择只恢复某一块盘的数据，而不是对所有的磁盘进行回滚。

从test快照恢复索引

# curl -XPOST es1:9001/_snapshot/robin_backup/test/_restore {"accepted":true}

查看恢复效果

# curl -XGET es2:9001/_cat/indices green open test fEukT7eWQ-OjD3uzxKmXcg 5 1 211 0 507.6kb 253.8kb green open test1 mw6MvvuZSlys0YLjo-UU-Q 5 1 277 0 611.3kb 305.6kb green open .kibana Lcv2WmUPTxqPjoNJV2C7sQ 1 1 2 0 15.4kb 7.7kb green open test3 YcU56zKqTNaTNH8Vcr6jpw 5 1 168 0 395.8kb 217.5kb green open test2 M-kOMZVkRcWwOwQlFRKRWw 5 1 139 0 343.1kb 171.5kb

也可以安装cerebro插件查看已存在的索引快照，单击restore即可恢复索引。

也可以选择只恢复部分索引

# curl -XPOST es1:9001/_snapshot/robin_backup/test/_restore { "indices": "test1", #指定需要恢复的索引名称 } 

在阿里云ES上创建相同的快照仓库，与自建的ES集群共享同一个oss仓库，完成后即可查看到快照信息

查看当前阿里云ES集群的索引

在阿里云ES上恢复快照，可以看到已经恢复的test和test1

由于阿里云ES存在.kibana索引，恢复快照时需要先关闭.kibana索引，否则会报错

{ "error": { "root_cause": [ { "type": "snapshot_restore_exception", "reason": "[robin_backup:test/oXtFY174RyeKRgYFi_dXKA] cannot restore index [.kibana] because it's open" } ], "type": "snapshot_restore_exception", "reason": "[robin_backup:test/oXtFY174RyeKRgYFi_dXKA] cannot restore index [.kibana] because it's open" }, "status": 500 }