Oracle 11g RAC ASM VOTEDISK所在磁盘全部丢失恢复

一、环境描述
系统：AIX 7100-02-03-1334
数据库：Oracle 11.2.0.3.10 RAC ASM

二、故障描述
存储故障导致ASM VOTEDISK所在磁盘全部丢失。
因为OCR和VOTEDISK的丢失，集群服务终止，OHAS进程还在。

三、操作步骤
思路：使用CRS自动备份的文件恢复OCR，VOTEDISK。
1)所有节点停掉CRS服务
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.crf' on 'testdb3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testdb3'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'testdb3'
CRS-2677: Stop of 'ora.cssdmonitor' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.crf' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb3'
CRS-2677: Stop of 'ora.gipcd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb3'
CRS-2677: Stop of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
root@testdb3:/#
root@testdb3:/#ps -ef |grep grid
root 2884826 3146642 0 15:56:44 pts/0 0:00 grep grid

2)在一个节点上以NOCRS方式启动CRS，此操作会启动ASM实例。
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'testdb3'
CRS-2676: Start of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'testdb3'
CRS-2676: Start of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testdb3'
CRS-2672: Attempting to start 'ora.gipcd' on 'testdb3'
CRS-2676: Start of 'ora.cssdmonitor' on 'testdb3' succeeded
CRS-2676: Start of 'ora.gipcd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'testdb3'
CRS-2672: Attempting to start 'ora.diskmon' on 'testdb3'
CRS-2676: Start of 'ora.diskmon' on 'testdb3' succeeded
CRS-2676: Start of 'ora.cssd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'testdb3'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testdb3'
CRS-2672: Attempting to start 'ora.ctssd' on 'testdb3'
CRS-2676: Start of 'ora.ctssd' on 'testdb3' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testdb3' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'testdb3'
CRS-2681: Clean of 'ora.asm' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'testdb3'
CRS-2676: Start of 'ora.asm' on 'testdb3' succeeded

3)新增一块盘，修改权限并在ASM中查看磁盘状态，创建一个新的磁盘组
root@testdb3:/#chown grid:asmadmin /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb3:/#chmod 660 /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb4:/#chown grid:asmadmin /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb4:/#chmod 660 /dev/vx/rdmp/hitachi_vsp0_11d4

root@testdb3:/#su - grid
$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 16:14:51 2014

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> set pagesize 10000
SQL> set linesize 300
SQL> alter system set asm_diskstring = '/dev/vx/rdmp/*';

System altered.

SQL> set pagesize 10000 linesize 300
SQL> select group_number, disk_number, state, path from v$asm_disk order by 1,2;

GROUP_NUMBER DISK_NUMBER STATE PATH
------------ ----------- -------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 0 NORMAL /dev/rhitachi_v0_11cc
0 1 NORMAL /dev/rhitachi_v0_11cd
0 2 NORMAL /dev/rhitachi_v0_11ce
0 3 NORMAL /dev/rhitachi_v0_11cf
0 4 NORMAL /dev/rhitachi_v0_11d0
0 5 NORMAL /dev/rhitachi_v0_11d1
0 6 NORMAL /dev/rhitachi_v0_11d2
0 7 NORMAL /dev/rhitachi_v0_11d3
0 8 NORMAL /voting_disk/vote_disk_nfs
0 9 NORMAL /dev/vx/rdmp/hitachi_vsp0_11cc
0 10 NORMAL /dev/vx/rdmp/hitachi_vsp0_11cd
0 11 NORMAL /dev/vx/rdmp/hitachi_vsp0_11ce
0 12 NORMAL /dev/vx/rdmp/hitachi_vsp0_11cf
0 13 NORMAL /dev/vx/rdmp/hitachi_vsp0_11d0
0 14 NORMAL /dev/vx/rdmp/hitachi_vsp0_11d1
0 15 NORMAL /dev/vx/rdmp/hitachi_vsp0_11d2
0 16 NORMAL /dev/vx/rdmp/hitachi_vsp0_11d3
0 17 NORMAL /dev/vx/rdmp/hitachi_vsp0_11d4

18 rows selected.

SQL> create diskgroup OCRVOTE1 external redundancy
2 disk '/dev/vx/rdmp/hitachi_vsp0_11d4'
3 ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';

Diskgroup created.

4)修改/etc/oracle/ocr.loc文件，将ocrconfig_loc修改成新增的磁盘组名称，以便将OCR恢复到新的磁盘组中。
root@testdb3:/#cat /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE
local_only=FALSE

root@testdb3:/#vi /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE1
local_only=FALSE

5)从以前OCR备份中恢复OCR
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

testdb4 2014/07/03 15:55:26 /oracle/app/11.2.0/grid/cdata/testdb/backup00.ocr

testdb4 2014/07/03 11:55:25 /oracle/app/11.2.0/grid/cdata/testdb/backup01.ocr

testdb4 2014/07/03 07:55:25 /oracle/app/11.2.0/grid/cdata/testdb/backup02.ocr

testdb4 2014/07/01 23:55:22 /oracle/app/11.2.0/grid/cdata/testdb/day.ocr

testdb4 2014/06/20 07:54:56 /oracle/app/11.2.0/grid/cdata/testdb/week.ocr

testdb2 2014/05/27 09:37:16 /oracle/app/11.2.0/grid/cdata/testdb/backup_20140527_093716.ocr

testdb2 2014/05/27 09:37:10 /oracle/app/11.2.0/grid/cdata/testdb/backup_20140527_093710.ocr

testdb4 2014/05/26 16:06:13 /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160613.ocr

testdb4 2014/05/26 16:05:54 /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160554.ocr

testdb4 2014/05/26 16:04:48 /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160448.ocr
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrconfig -restore /oracle/app/11.2.0/grid/cdata/testdb/backup00.ocr
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3244
Available space (kbytes) : 258876
ID : 1187520997
Device/File Name : +OCRVOTE1
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

6)创建VOTEDISK
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl replace votedisk +OCRVOTE1
Successful addition of voting disk 833551381fb94fd0bf09eefda7f1b380.
Successful deletion of voting disk 89210622f0864ff0bf9517205691e679.
Successful deletion of voting disk 55c4ee685a824ff3bf6ce510bf09468e.
Successful deletion of voting disk 159234e88fe64f55bf0d4571362c3b07.
Successfully replaced voting disk group with +OCRVOTE1.
CRS-4266: Voting file(s) successfully replaced
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 833551381fb94fd0bf09eefda7f1b380 (/dev/vx/rdmp/hitachi_vsp0_11d4) [OCRVOTE1]
Located 1 voting disk(s).

7)创建SPFILE并按照以前标准修改ASM参数，重启ASM
$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 16:28:37 2014

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create spfile='+OCRVOTE1' from memory;

File created.

SQL> startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started

Total System Global Area 342388736 bytes
Fixed Size 2221360 bytes
Variable Size 315001552 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled

8)重启CRS
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.ctssd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.asm' on 'testdb3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testdb3'
CRS-2677: Stop of 'ora.ctssd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.asm' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'testdb3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'testdb3'
CRS-2677: Stop of 'ora.cssd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb3'
CRS-2677: Stop of 'ora.gipcd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb3'
CRS-2677: Stop of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

9)启动节点2的CRS
修改节点2的ocr.loc
root@testdb4:/#vi /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE1
local_only=FALSE

root@testdb4:/#/oracle/app/11.2.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

10)查看CRS资源状态，已恢复完成
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
ONLINE OFFLINE testdb3
ONLINE OFFLINE testdb4
ora.DATADG.dg
ONLINE OFFLINE testdb3
ONLINE OFFLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE OFFLINE testdb3
ONLINE OFFLINE testdb4
ora.SYSDG.dg
ONLINE OFFLINE testdb3
ONLINE OFFLINE testdb4
ora.asm
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb3
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb3
ora.scan1.vip
1 ONLINE ONLINE testdb3
ora.testdb.db
1 ONLINE OFFLINE Instance Shutdown
3 ONLINE OFFLINE Instance Shutdown
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
$

ITPub博客

Oracle 11g RAC ASM VOTEDISK所在磁盘全部丢失重大故障恢复

最新文章