记录一次oracle rac的OCR_VOTE磁盘一块磁盘坏掉的情况测试

现状roottestdb1[ASM1]/root$ocrcheckStatus of Oracle Cluster Registry is as follows :Version : 3Total space (kbytes) : 262120Used space (kbytes) : 3260Available space (kbytes) : 258860ID : 545177980Device/File Name : OCR_VOTEDevice/File integrity check succeededDevice/File not configuredDevice/File not configuredDevice/File not configuredDevice/File not configuredCluster registry integrity check succeededLogical corruption check succeededroottestdb1[ASM1]/root$crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 531089329c6e4fd7bf55527321648aeb (/dev/asm-diskm) [OCR_VOTE]2. ONLINE 228bc2cc1a624f2bbff2ed7ed24d3a36 (/dev/asm-diski) [OCR_VOTE]3. ONLINE 2ddb39eaa1a54ff0bf38c48b12a7315c (/dev/asm-diskg) [OCR_VOTE]4. ONLINE 735db22d36d14f31bf446493c25cf1bc (/dev/asm-diskf) [OCR_VOTE]5. ONLINE 5cc9d45f6ade4f98bf16c78be12fa786 (/dev/asm-diskd) [OCR_VOTE]Located 5 voting disk(s).roottestdb1[ASM1]/root$SQL COLUMN PATH FORMAT A20;COLUMN name FORMAT A20;SET LINESIZE 500select as disk_group,,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dgwhere d.GROUP_NUMBERdg.GROUP_NUMBERorder by ;SQL SQL SQL 2 3DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskjDGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-disklDGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diskeDGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskkOCR_VOTE OCR_VOTE_0001 CACHED MEMBER ONLINE /dev/asm-diskiOCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskdOCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskgOCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskmOCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskfOCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskcOCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh模拟/dev/asm-diski被dd后的情况dd if/dev/zero of/dev/asm-diski bs1024k count20dd if/dev/zero of/dev/asm-diski bs1024k count20roottestdb1[ASM1]/root$dd if/dev/zero of/dev/asm-diski bs1024k count20200 records in200 records out20971520 bytes (21 MB) copied, 0.016327 s, 1.3 GB/sroottestdb1[ASM1]/root$SQL /DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskjDGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-disklDGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diskeDGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskkOCR_VOTE OCR_VOTE_0001 CACHED CANDIDATE ONLINE /dev/asm-diskiOCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskdOCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskgOCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskmOCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskfOCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskcOCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh11 rows selected.可以看到这块磁盘对应的磁盘头已经成为CANDIDATE状态强制关闭节点再重启发现已经启动不起来了crsctl stop crs -f查看后台日志roottestdb1[ASM1]/grid/grid_home/log/testdb1$tail -100 alerttestdb1.log2021-05-26 20:23:37.862:[ohasd(3005)]CRS-2767:Resource state recovery not attempted for ora.diskmon as its target state is OFFLINE2021-05-26 20:23:37.862:[ohasd(3005)]CRS-2769:Unable to failover resource ora.diskmon.[client(3820)]CRS-10001:26-May-21 20:23 ACFS-9391: Checking for existing ADVM/ACFS installation.[client(3825)]CRS-10001:26-May-21 20:23 ACFS-9392: Validating ADVM/ACFS installation files for operating system.[client(3827)]CRS-10001:26-May-21 20:23 ACFS-9393: Verifying ASM Administrator setup.[client(3830)]CRS-10001:26-May-21 20:23 ACFS-9308: Loading installed ADVM/ACFS drivers.[client(3833)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading oracleoks.ko driver.[client(3842)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading oracleadvm.ko driver.[client(3867)]CRS-10001:26-May-21 20:23 ACFS-9154: Loading oracleacfs.ko driver.[client(3926)]CRS-10001:26-May-21 20:23 ACFS-9327: Verifying ADVM/ACFS devices.[client(3931)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device /dev/asm/.asm_ctl_spec.[client(3935)]CRS-10001:26-May-21 20:23 ACFS-9156: Detecting control device /dev/ofsctl.[client(3940)]CRS-10001:26-May-21 20:23 ACFS-9322: completed2021-05-26 20:23:44.167:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource ASM failed: details at (:CLSN00006:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log2021-05-26 20:23:49.508:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5011:Check of resource ASM failed: details at (:CLSN00006:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log2021-05-26 20:24:10.920:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:24:11.099:[ohasd(3005)]CRS-2807:Resource ora.crsd failed to start automatically.2021-05-26 20:24:12.096:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:24:42.115:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:25:12.146:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:25:42.186:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:26:12.216:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:26:42.241:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:27:12.253:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.2021-05-26 20:27:42.263:[/grid/grid_home/bin/oraagent.bin(3376)]CRS-5019:All OCR locations are on ASM disk groups [OCR_VOTE], and none of these disk groups are mounted. Details are at (:CLSN00100:) in /grid/grid_home/log/testdb1/agent/ohasd/oraagent_grid/oraagent_grid.log.强制启动先crsctl start crs再进行crsctl check crs发现并不能完全启动但是能够通过grid进入sqlplus / as sysasm由于我们采用了HIGH这种方式只坏了一块盘可以强制mountalter diskgroup OCR_VOTE mount force;再正常启动crsctl start crs查询集群状态等的时间可能很长roottestdb1[ASM1]/grid/grid_home/log/testdb1$crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online查看集群的磁盘状态复制gridtestdb1[ASM1]/home/grid$sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Wed May 26 20:52:39 2021Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL COLUMN PATH FORMAT A20;COLUMN name FORMAT A20;SET LINESIZE 500select as disk_group,,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,PATH from V$ASM_DISK d ,V$ASM_DISKGROUP dgwhere d.GROUP_NUMBERdg.GROUP_NUMBERorder by ;SQL SQL SQL 2 3DISK_GROUP NAME MOUNT_STATUS HEADER_STATUS MODE_STATUS PATH------------------------------------------------------------ -------------------- -------------- ------------------------ -------------- --------------------ACFS01 ACFS01_0000 CACHED MEMBER ONLINE /dev/asm-diskjDGARCH DGARCH_0000 CACHED MEMBER ONLINE /dev/asm-disklDGDATA DGDATA_0000 CACHED MEMBER ONLINE /dev/asm-diskeDGSYS DGSYS_0000 CACHED MEMBER ONLINE /dev/asm-diskkOCR_VOTE OCR_VOTE_0002 CACHED MEMBER ONLINE /dev/asm-diskgOCR_VOTE OCR_VOTE_0001 MISSING UNKNOWN OFFLINEOCR_VOTE OCR_VOTE_0000 CACHED MEMBER ONLINE /dev/asm-diskmOCR_VOTE OCR_VOTE_0005 CACHED MEMBER ONLINE /dev/asm-diskcOCR_VOTE OCR_VOTE_0003 CACHED MEMBER ONLINE /dev/asm-diskfOCR_VOTE OCR_VOTE_0004 CACHED MEMBER ONLINE /dev/asm-diskdOCR_VOTE2 OCR_VOTE2_0000 CACHED MEMBER ONLINE /dev/asm-diskh11 rows selected.发现磁盘已经成为 MISSING UNKNOWN 状态。