故障模拟：
模拟 2 个 master 节点 down，需要重装，直接关闭故障节点，事先已经备份过 etcd。
需要事先备份了 etcd数据，离线环境需要 image ：registry.redhat.io/rhel8/support-tools，否则不能启动 oc debug。
具体方法如下：
[root@bastion~]# oc debug node/master-1.offline.nielasaran.comsh-4.4# chroot /hostsh-4.4# /usr/local/bin/cluster-backup.sh/home/core/assets/backupsh-4.4# scp -r /home/core/assets/backup/ root@bastion:
1. 模拟故障关机后，可以看到 oc 超时
[root@bastion~]# oc get nodes Error from server (Timeout): the server was unable to return aresponse in the time allotted, but may still be processing the request (getnodes)
2. 把备份的 etcd 数据和 kubeconfig 证书 scp 过去，后面 master-2 要用 oc 命令并且需要管理员权限
[root@bastion~]# scp -r backup/ core@master-2:/home/core/ [root@bastion ~]# scp .kube/configcore@master-2:
3. 注意：如果只是丢了1个master 节点，意味着此时有 2 个 master 节点可用，这些内容需要在非恢复主机上做，也就是说，如果我计划把 master-2 主机作为 恢复主机，以下操作请不要再 master-2 上做，文档原文，如果只剩1个恢复节点，做不做都行，反正后面恢复脚本会帮你把这个做了。
3.1 进入额外剩余的 controll plane node
[root@bastion ~]# ssh core@master-1
3.2 移除 etcd static pod，然后等 1 - 2 分钟
[core@master-1~]$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
3.3 确定 etcd Pod 是否还在，如果还在继续等，等到没了为止
[core@master-1~]$ sudo crictl ps | grep etcd | grep -v operator
3.4 移除 kube-apiserver static pod，然后等 1 - 2 分钟
[core@master-1~]$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp
3.5 确保 api-server 停了
[core@master-1~]$ sudo crictl ps | grep kube-apiserver | grep -v operator
3.6 清理 etcd 目录
[core@master-1~]$ sudo mv /var/lib/etcd/ /tmp
4. 进入恢复用 master 节点，执行恢复脚本
[core@master-2~]$ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
...stoppingkube-apiserver-pod.yaml...stopping kube-controller-manager-pod.yaml...stoppingkube-scheduler-pod.yaml...stopping etcd-pod.yamlWaiting for container etcd tostopcompleteWaiting for container etcdctl to stopcompleteWaiting for containeretcd-metrics to stopcompleteWaiting for container kube-controller-manager tostop.completeWaiting for container kube-apiserver to stopcompleteWaiting forcontainer kube-scheduler to stopcompletestarting restore-etcd staticpodstartingkube-apiserver-pod.yamlstatic-pod-resources/kube-apiserver-pod-4/kube-apiserver-pod.yamlstartingkube-controller-manager-pod.yamlstatic-pod-resources/kube-controller-manager-pod-6/kube-controller-manager-pod.yamlstartingkube-scheduler-pod.yamlstatic-pod-resources/kube-scheduler-pod-5/kube-scheduler-pod.yaml
5. 重启 kubelet 服务，如果只坏1 个master 节点，需要 剩余2个 master 都重启
[core@master-1~]$ sudo systemctl restart kubelet.service
Warning: The unit file, source configurationfile or drop-ins of kubelet.service changed on disk. Run 'systemctldaemon-reload' to reload units.
6. 验证单 etcd 是否恢复
[core@master-2~]$ sudo crictl ps | grep etcd | grep -v operator
00f72c57e27b9 621d5c808fe6847daf29bf02a1c47aef440f5ce4a08749cb8c7014a712565b6d 3 minutes ago Running etcd 0 0db089710e5a4
7. 此时发现 oc 已经可以恢复使用
[core@master-2~]$ oc get nodes NAME
![图片]()
8. 查看 etcd 集群状况，目前只有一个 member，无需手动移除
$ oc get pods ‐n openshift‐etcd | grep etcd 查看[core@master‐2 ~]$ oc rsh etcd‐master‐2.offline.nielasaran.comsh‐4.4# etcdctl endpoint status ‐w table
![图片]()
9. 移除废弃的 master 节点
![图片]()
10. 重装 master node，装好之后，等 csr
![图片]()
#oc get csr ‐o name | xargs oc adm certificate approve
11. 检查集群状况，如果遇到不正常大概多等一会，测试环境10分钟恢复正常，根据集群规模可能时间会更长
![图片]()
![图片]()
![图片]()
检查节点状态，看是否都是一致的
![图片]()
上图所用cli：
[root@bastion ~]# oc get etcd ‐o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
[root@bastion ~]# oc get kubeapiserver ‐o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
[root@bastion ~]# oc get kubecontrollermanager ‐o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}'
[root@bastion ~]# oc get kubescheduler ‐o=jsonpath='{range .items[0].status.conditions[?(@.type=="NodeInstallerProgressing")]}{.reason}{"\n"}{.message}{"\n"}
