ORA-15003 DISKGROUP "DATA" ALREADY MOUNTED IN ANOTHER LOCK NAME SPACE +ERROR: PST FOUND ANOTHER HEARTBEAT (GRP 1)

经过测试发现:重启故障节点也可以解决该问题;官方解决办法如下:

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.

SYMPTOMS

1. ASM diskgroups in single instance environment will not be mounted with the following similar error in sqlplus screen.

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ORA-15017 diskgroup DATA cannot be mounted
ORA-15003 diskgroup DATA already mounted in another lock name space

2. Heartbeat detection error is seen In ASM alert.log,

Mon Nov 03 15:11:11 2012
SQL> alter diskgroup DATA mount
NOTE: cache registered group DATA number=1 incarn=0x3866aca2
NOTE: cache began mount (first) of group DATA number=1 incarn=0x3866aca2
Mon Nov 03 15:11:31 2012
NOTE: Assigning number (1,0) to disk (/dev/asm/data1)
NOTE: Assigning number (1,1) to disk (/dev/asm/data2)
Mon Nov 03 15:11:38 2012
ERROR: PST found another heartbeat (grp 1) <<<<<<--- Heartbeat detected.
NOTE: cache dismounting (clean) group 11/0x3866ACA2 (DATA)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x3866ACA2 (DATA)
NOTE: cache ending mount (fail) of group DATA number=1 incarn=0x3866aca2
kfdp_dismount(): 87
kfdp_dismountBg(): 87
NOTE: De-assigning number (1,0) from disk (/dev/asm/data1)
NOTE: De-assigning number (1,1) from disk (/dev/asm/data2)
ERROR: diskgroup DATA was not mounted

CAUSE

ASM devices are shared between 2 database servers with single instance ASM environment for migration work.

The same diskgroup ( DATA here ) is tried to be mounted on the 2nd node while it is currently mounted on the first node.

SOLUTION

Database servers have to be clustered with RAC configuration to mount the same ASM diskgroups pointing to the same storage on multiple nodes. Oracle ASM uses heartbeat to prevent the same diskgroup from being mounted on multiple nodes from non-clustered servers at the same time.

Check whether heartbeat is detected for the failed diskgroup.

1. Use AMDU tool before mounting the diskgroup in question.

	$ amdu -diskstring '/dev/asm/data*' -dump DATA  -nomap -nodir   | grep HEARTBEAT

AMDU-00204: Disk N0001 is in currently mounted diskgroup DATA

AMDU-00201: Disk N0001: '/dev/asm/data1'

** HEARTBEAT DETECTED **

2. Use kfed . Heartbeat block is located at the last 4k block on aun=1.

If AU size for the affected diskgroup is 1M, heartbeat can be checked in the following way.

	$ kfed read  aus=1048576 aun=1 blkn=255 | grep  kfbh.check   

And also note the following command with various size of AU.

	$ kfed read  aus=4194304 aun=1 blkn=1023 | grep  kfbh.check   

$ kfed read  aus=8388608 aun=1 blkn=2047 | grep  kfbh.check   

$ kfed read  aus=16777216 aun=1 blkn=4095 | grep  kfbh.check 

$ kfed read  aus=33554432  aun=1 blkn=8191 | grep  kfbh.check

$ kfed read  aus=67108864  aun=1 blkn=16383 | grep  kfbh.check

<<< If the command above show a different checksum value over time ( say 10 sec interval ), it indicates that the heatbeat block is being updated from other node beyond the server.

3. Use OS command dd in the following way.

	$ dd if= of=/tmp/1.dd bs=4096 skip=511

10 secs later, take another dd output with a different file name

	$ dd if= of=/tmp/2.dd bs=4096 skip=511  

<<< 1M AU, skip=511 / 4M AU, skip=2047 / 8M AU, skip=4095 / 16M AU, skip=8191 / 32M AU, skip=16383 / 64M AU, skip=32767

And then check if they shows different content.

	   $ diff /tmp/1.dd /tmp/2.dd

   Binary files /tmp/1.dd and /tmp/2.dd differ             

<<< If these 2 files show different content, it also indicates that the heatbeat block is being updated from other node beyond the server.

4. If heartbeat is detected while the affected diskgroup is not mounted, it indicates that the diskgroup in question is currently mounted by other node. But it does not tell which node is currently mounting it. Find the node with help of Storage Admin team which is currently mounting the diskgorup in question and dismount and try to mount it on the 2nd node.

ITPub博客

ORA-15003 DISKGROUP "DATA" ALREADY MOUNTED IN ANOTHER LOCK NAME SPACE