记录一次oracle rac的OCR_VOTE磁盘一块磁盘坏掉的情况测试。
现状 root@testdb1[+ASM1]/root$ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3260 Available space (kbytes) : 258860 ID : 545177980 Device/File Name : +OCR_VOTE Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded root@testdb1[+ASM1]/root$crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 531089329c6e4fd7bf55527321648aeb (/dev/asm-diskm) [OCR_VOTE] 2. ONLINE 228bc2cc1a624f2bbff2ed7ed24d3a36 (/dev/asm-diski) [OCR_VOTE] 3. ONLINE 2ddb39eaa1a54ff0bf38c48b12a7315c (/dev/asm-diskg) [OCR_VOTE] 4. ONLINE 735db22d36d14f31bf446493c25cf1bc (/dev/asm-diskf) [OCR_VOTE] 5. ONLINE 5cc9d45f6ade4f98bf16c78be12fa786 (/dev/asm-diskd) [OCR_VOTE] Located 5 voting disk(s). root@testdb1[+ASM1]/root$ SQL> COLUMN PATH FORMAT A20; COLUMN name FORMAT A20; SET LINESIZE 500 select dg.NAME as disk_group,d.NAME,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dg where d.GROUP_NUMBER=dg.GROUP_NUMBER order by dg.NAME; SQL> SQL> SQL> 2 3 DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH ------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- -------------------- ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskj DGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-diskl DGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diske DGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskk OCR_VOTE OCR_VOTE_0001 CACHED MEMBER ONLINE /dev/asm-diski OCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskd OCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskg OCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskm OCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskf OCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskc OCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh 模拟/dev/asm-diski被dd后的情况 dd if=/dev/zero of=/dev/asm-diski bs=1024k count=20 dd if=/dev/zero of=/dev/asm-diski bs=1024k count=20 root@testdb1[+ASM1]/root$dd if=/dev/zero of=/dev/asm-diski bs=1024k count=20 20+0 records in 20+0 records out 20971520 bytes (21 MB) copied, 0.016327 s, 1.3 GB/s root@testdb1[+ASM1]/root$ SQL> / DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH ------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- -------------------- ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskj DGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-diskl DGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diske DGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskk OCR_VOTE OCR_VOTE_0001 CACHED CANDIDATE ONLINE /dev/asm-diski OCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskd OCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskg OCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskm OCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskf OCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskc OCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh 11 rows selected. 可以看到这块磁盘对应的磁盘头已经成为CANDIDATE状态 强制关闭节点，再重启，发现已经启动不起来了 crsctl stop crs -f 查看后台日志 root@testdb1[+ASM1]/grid/grid_home/log/testdb1$tail -100 alerttestdb1.log 2021-05-26 20:23:37.862: [ohasd(3005)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2021-05-26 20:23:37.862: [ohasd(3005)]CRS-2769:Unable to failover resource 'ora.diskmon'. [client(3820)]CRS-10001:26-May-21 20:23 ACFS-9391: Checking for existing ADVM/ACFS installation. [client(3825)]CRS-10001:26-May-21 20:23 ACFS-9392: Validating ADVM/ACFS installation files for operating system. [client(3827)]CRS-10001:26-May-21 20:23 ACFS-9393: Verifying ASM Administrator setup. [client(3830)]CRS-10001:26-May-21 20:23 ACFS-9308: Loading installed ADVM/ACFS drivers. [client(3833)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleoks.ko' driver. [client(3842)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleadvm.ko' driver. [client(3867)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading 'oracleacfs.ko' driver. [client(3926)]CRS-10001:26-May-21 20:23 ACFS-9327: Verifying ADVM/ACFS devices. [client(3931)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. [client(3935)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device '/dev/ofsctl'. [client(3940)]CRS-10001:26-May-21 20:23 ACFS-9322: completed 2021-05-26 20:23:44.167: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log" 2021-05-26 20:23:49.508: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log" 2021-05-26 20:24:10.920: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:24:11.099: [ohasd(3005)]CRS-2807:Resource 'ora.crsd' failed to start automatically. 2021-05-26 20:24:12.096: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:24:42.115: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:25:12.146: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:25:42.186: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:26:12.216: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:26:42.241: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:27:12.253: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2021-05-26 20:27:42.263: [/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log". 强制启动 先crsctl start crs，再进行crsctl check crs 发现并不能完全启动，但是能够通过grid进入sqlplus / as sysasm 由于我们采用了HIGH这种方式，只坏了一块盘，可以强制mount alter diskgroup OCR_VOTE mount force; 再正常启动 crsctl start crs 查询集群状态，等的时间可能很长 root@testdb1[+ASM1]/grid/grid_home/log/testdb1$crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online 查看集群的磁盘状态 grid@testdb1[+ASM1]/home/grid$sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Wed May 26 20:52:39 2021 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> COLUMN PATH FORMAT A20; COLUMN name FORMAT A20; SET LINESIZE 500 select dg.NAME as disk_group,d.NAME,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dg where d.GROUP_NUMBER=dg.GROUP_NUMBER order by dg.NAME;SQL> SQL> SQL> 2 3 DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH ------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- -------------------- ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskj DGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-diskl DGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diske DGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskk OCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskg OCR_VOTE OCR_VOTE_0001 MISSING UNKNOWN OFFLINE OCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskm OCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskc OCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskf OCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskd OCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh 11 rows selected. 发现磁盘已经成为 MISSING UNKNOWN 状态。
