记录一次oracle rac的OCR_VOTE磁盘一块磁盘坏掉的情况测试。-低调大师

记录一次oracle rac的OCR_VOTE磁盘一块磁盘坏掉的情况测试。

2021-05-27 632

现状

root@testdb1[+ASM1]/root$ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       3260
     Available space (kbytes) :     258860
     ID                       :  545177980
     Device/File Name         :  +OCR_VOTE
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

     Cluster registry integrity check succeeded

     Logical corruption check succeeded

root@testdb1[+ASM1]/root$crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   531089329c6e4fd7bf55527321648aeb (/dev/asm-diskm) [OCR_VOTE]
 2. ONLINE   228bc2cc1a624f2bbff2ed7ed24d3a36 (/dev/asm-diski) [OCR_VOTE]
 3. ONLINE   2ddb39eaa1a54ff0bf38c48b12a7315c (/dev/asm-diskg) [OCR_VOTE]
 4. ONLINE   735db22d36d14f31bf446493c25cf1bc (/dev/asm-diskf) [OCR_VOTE]
 5. ONLINE   5cc9d45f6ade4f98bf16c78be12fa786 (/dev/asm-diskd) [OCR_VOTE]
Located 5 voting disk(s).
root@testdb1[+ASM1]/root$

SQL> COLUMN PATH FORMAT A20;
COLUMN name FORMAT A20;
SET LINESIZE 500
select dg.NAME as disk_group,d.NAME,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dg
where d.GROUP_NUMBER=dg.GROUP_NUMBER
order by dg.NAME;
SQL> SQL> SQL>   2    3  
DISK_GROUP                           NAME         MOUNT_STATUS   HEADER_STATUS        MODE_STATUS    PATH
------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------
ACFS01                               ACFS01_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskj
DGARCH                               DGARCH_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskl
DGDATA                               DGDATA_0000      CACHED     MEMBER           ONLINE     /dev/asm-diske
DGSYS                                DGSYS_0000       CACHED     MEMBER           ONLINE     /dev/asm-diskk
OCR_VOTE                             OCR_VOTE_0001    CACHED     MEMBER           ONLINE     /dev/asm-diski
OCR_VOTE                             OCR_VOTE_0004    CACHED     MEMBER           ONLINE     /dev/asm-diskd
OCR_VOTE                             OCR_VOTE_0002    CACHED     MEMBER           ONLINE     /dev/asm-diskg
OCR_VOTE                             OCR_VOTE_0000    CACHED     MEMBER           ONLINE     /dev/asm-diskm
OCR_VOTE                             OCR_VOTE_0003    CACHED     MEMBER           ONLINE     /dev/asm-diskf
OCR_VOTE                             OCR_VOTE_0005    CACHED     MEMBER           ONLINE     /dev/asm-diskc
OCR_VOTE2                            OCR_VOTE2_0000   CACHED     MEMBER           ONLINE     /dev/asm-diskh

模拟/dev/asm-diski被dd后的情况
dd if=/dev/zero of=/dev/asm-diski  bs=1024k count=20

dd if=/dev/zero of=/dev/asm-diski  bs=1024k count=20

root@testdb1[+ASM1]/root$dd if=/dev/zero of=/dev/asm-diski  bs=1024k count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 0.016327 s, 1.3 GB/s
root@testdb1[+ASM1]/root$

SQL> /

DISK_GROUP                           NAME         MOUNT_STATUS   HEADER_STATUS        MODE_STATUS    PATH
------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------
ACFS01                               ACFS01_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskj
DGARCH                               DGARCH_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskl
DGDATA                               DGDATA_0000      CACHED     MEMBER           ONLINE     /dev/asm-diske
DGSYS                                DGSYS_0000       CACHED     MEMBER           ONLINE     /dev/asm-diskk
OCR_VOTE                             OCR_VOTE_0001    CACHED     CANDIDATE        ONLINE     /dev/asm-diski
OCR_VOTE                             OCR_VOTE_0004    CACHED     MEMBER           ONLINE     /dev/asm-diskd
OCR_VOTE                             OCR_VOTE_0002    CACHED     MEMBER           ONLINE     /dev/asm-diskg
OCR_VOTE                             OCR_VOTE_0000    CACHED     MEMBER           ONLINE     /dev/asm-diskm
OCR_VOTE                             OCR_VOTE_0003    CACHED     MEMBER           ONLINE     /dev/asm-diskf
OCR_VOTE                             OCR_VOTE_0005    CACHED     MEMBER           ONLINE     /dev/asm-diskc
OCR_VOTE2                            OCR_VOTE2_0000   CACHED     MEMBER           ONLINE     /dev/asm-diskh

11 rows selected.

可以看到这块磁盘对应的磁盘头已经成为CANDIDATE状态

强制关闭节点，再重启，发现已经启动不起来了

crsctl stop crs -f
查看后台日志

root@testdb1[+ASM1]/grid/grid_home/log/testdb1$tail -100 alerttestdb1.log 

2021-05-26 20:23:37.862: 
[ohasd(3005)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2021-05-26 20:23:37.862: 
[ohasd(3005)]CRS-2769:Unable to failover resource 'ora.diskmon'.
[client(3820)]CRS-10001:26-May-21 20:23 ACFS-9391: Checking for existing ADVM/ACFS installation.
[client(3825)]CRS-10001:26-May-21 20:23 ACFS-9392: Validating ADVM/ACFS installation files for operating system.
[client(3827)]CRS-10001:26-May-21 20:23 ACFS-9393: Verifying ASM Administrator setup.
[client(3830)]CRS-10001:26-May-21 20:23 ACFS-9308: Loading installed ADVM/ACFS drivers.
[client(3833)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleoks.ko' driver.
[client(3842)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleadvm.ko' driver.
[client(3867)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleacfs.ko' driver.
[client(3926)]CRS-10001:26-May-21 20:23 ACFS-9327: Verifying ADVM/ACFS devices.
[client(3931)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
[client(3935)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device '/dev/ofsctl'.
[client(3940)]CRS-10001:26-May-21 20:23 ACFS-9322: completed
2021-05-26 20:23:44.167: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2021-05-26 20:23:49.508: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2021-05-26 20:24:10.920: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:24:11.099: 
[ohasd(3005)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2021-05-26 20:24:12.096: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:24:42.115: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:25:12.146: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:25:42.186: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:26:12.216: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:26:42.241: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:27:12.253: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2021-05-26 20:27:42.263: 
[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log".

强制启动

先crsctl start crs，再进行crsctl check crs
发现并不能完全启动，但是能够通过grid进入sqlplus / as sysasm

由于我们采用了HIGH这种方式，只坏了一块盘，可以强制mount
alter diskgroup OCR_VOTE mount force;

再正常启动

crsctl start crs

查询集群状态，等的时间可能很长
root@testdb1[+ASM1]/grid/grid_home/log/testdb1$crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

查看集群的磁盘状态


grid@testdb1[+ASM1]/home/grid$sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed May 26 20:52:39 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> COLUMN PATH FORMAT A20;
COLUMN name FORMAT A20;
SET LINESIZE 500
select dg.NAME as disk_group,d.NAME,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dg
where d.GROUP_NUMBER=dg.GROUP_NUMBER
order by dg.NAME;SQL> SQL> SQL>   2    3  

DISK_GROUP                           NAME         MOUNT_STATUS   HEADER_STATUS        MODE_STATUS    PATH
------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------
ACFS01                               ACFS01_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskj
DGARCH                               DGARCH_0000      CACHED     MEMBER           ONLINE     /dev/asm-diskl
DGDATA                               DGDATA_0000      CACHED     MEMBER           ONLINE     /dev/asm-diske
DGSYS                                DGSYS_0000       CACHED     MEMBER           ONLINE     /dev/asm-diskk
OCR_VOTE                             OCR_VOTE_0002    CACHED     MEMBER           ONLINE     /dev/asm-diskg
OCR_VOTE                             OCR_VOTE_0001    MISSING    UNKNOWN          OFFLINE
OCR_VOTE                             OCR_VOTE_0000    CACHED     MEMBER           ONLINE     /dev/asm-diskm
OCR_VOTE                             OCR_VOTE_0005    CACHED     MEMBER           ONLINE     /dev/asm-diskc
OCR_VOTE                             OCR_VOTE_0003    CACHED     MEMBER           ONLINE     /dev/asm-diskf
OCR_VOTE                             OCR_VOTE_0004    CACHED     MEMBER           ONLINE     /dev/asm-diskd
OCR_VOTE2                            OCR_VOTE2_0000   CACHED     MEMBER           ONLINE     /dev/asm-diskh

11 rows selected.

发现磁盘已经成为    MISSING     UNKNOWN 状态。

微信关注我们

原文链接：https://blog.51cto.com/lishiyan/2820440

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

表决盘与仲裁盘操作

查看表决盘 [root@testdb1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 4a4d2ebc4ab64fa5bfb9c2ad51df2f06 (/dev/asm-diskm) [OCR_VOTE] 2. ONLINE 4098f6aa168a4f93bf1f6364273fe911 (/dev/asm-diski) [OCR_VOTE] 3. ONLINE 2a2c91ae30374fa0bfc8cb65feae7cf8 (/dev/asm-diskg) [OCR_VOTE] 4. ONLINE a808cd9fc5664f73bf1873c948f35fa6 (/dev/asm-diskf) [OCR_VOTE] 5. ONLINE 445d6133f4714fadbfd653f09ddca99c (/dev/asm-diskd) [OCR_VOTE] ...

2021-05-27

588

今日课程推荐：https://edu.51cto.com/course/27045.html 镜像的定制实际上就是定制镜像的每一层所添加的配置、文件等信息，实际上当我们在一个容器中添加或者修改了一些文件后，我们可以通过docker commit命令来生成一个新的镜像，但是这个方法不够直观，没办法追溯我们镜像里面到底有哪些内容，所以实际定制镜像的过程我们很少采用这种方式。而是使用一个名为Dockerfile的文本文件来进行镜像定制，我们可以把镜像的每一层修改、安装、构建、操作的命令都写入到这个文件中，一行就对应镜像的一层，这种方法显然要更高级，因为我们有一个文件来直观反映我们的镜像内容，还可以作为版本记录进行跟踪。定制镜像以我们之前的 nginx 镜像为例，我们现在来定制一个 nginx 镜像，要求默认的应用页面访问的内容是Hello Docker。首先我们新建一个目录 testnginx，然后在该目录下面新建一个名为Dockerfile的空白文本文件： $mkdir-ptestnginx&&cdtestnginx&&touchDockerfile ...

2021-05-27

586

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。