ITPub博客

首页 > 数据库 > Oracle > Quorum FailGroup

Quorum FailGroup

原创 Oracle 作者:westzq1984 时间:2014-02-13 20:04:03 0 删除 编辑
参考:
How to Manually Add NFS voting disk to an Extended Cluster using ASM in 11.2 (Doc ID 1421588.1)
Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
RAC: Frequently Asked Questions [ID 220970.1]

***********************************************************
什么是Quorum FailGroup
***********************************************************
Quorum FailGroup中,只保存Voting Disk,用于RAC on Extended Distance Clusters,做仲裁盘用
要放在其他存储节点上或者通过NFS共享一个zero-padded文件作为voting disk
对于使用2个阵列提供保护的环境中,也可以使用Quorum FG
假设使用high冗余,一共5个投票盘。那么5个投票盘在2个阵列上的数目肯定不同
假设一个阵列上有3个,另一个阵列上有2个,如果存在3个投票盘的阵列宕机,那么将只有2个投票盘可用
这种情况下,集群的所有节点将宕机
这种情况下,必须在第三个地点放置一个投票盘,在2个阵列上各有2个投票盘。这样任意一个阵列宕机,还可以保证有3个投票盘是可用的

***********************************************************
如何添加Quorum FailGroup
***********************************************************

一般有3个阵列的情况不多,下面测试通过NFS来配置

1.当前情况:
HIGH冗余,5个投票盘
[oracle@database2 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7fc99b8d4dc54f27bf967f23524a19e3 (/dev/asm-crs1) [CRSDG]
 2. ONLINE   2ffd39d609bc4f71bf2b19de2e71c7a8 (/dev/asm-crs2) [CRSDG]
 3. ONLINE   002eb188e9b14ffbbf4d5a607ade51c2 (/dev/asm-crs3) [CRSDG]
 4. ONLINE   2319348cf3cc4f6abf116f973d8fd922 (/dev/asm-crs4) [CRSDG]
 5. ONLINE   6c3cb875ba7e4fe2bffe97189e2bae25 (/dev/asm-crs5) [CRSDG]
Located 5 voting disk(s).

SQL>  select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;

GROUP_NUMBER DISK_NUMBER      OS_MB NAME                           PATH            FAILGROUP                      STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
           1           0       2048 CRSDG_0000                     /dev/asm-crs01  CRSDG_0000                     NORMAL
           1           1       2048 CRSDG_0001                     /dev/asm-crs02  CRSDG_0001                     NORMAL
           1           2       2048 CRSDG_0002                     /dev/asm-crs03  CRSDG_0002                     NORMAL
           1           3       2048 CRSDG_0003                     /dev/asm-crs04  CRSDG_0003                     NORMAL
           1           4       2048 CRSDG_0004                     /dev/asm-crs05  CRSDG_0004                     NORMAL
           2           0      20480 DATADG_0000                    /dev/asm-data1  DATADG_0000                    NORMAL
           2           1      20480 DATADG_0001                    /dev/asm-data2  DATADG_0001                    NORMAL

2.配置NFS
2.1 NFS服务器端
[root@dm01db01 /]# cat /etc/exports
/oracle/votedisk 192.168.123.31(rw,sync,no_root_squash)
/oracle/votedisk 192.168.123.32(rw,sync,no_root_squash)
[root@dm01db01 /]#  /etc/rc.d/init.d/portmap start
Starting portmap:                                          [  OK  ]
[root@dm01db01 /]#  /etc/rc.d/init.d/nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Stopping RPC idmapd:                                       [  OK  ]
Starting RPC idmapd:                                       [  OK  ]

2.2 数据库服务器端MOUNT文件系统
# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048
# chown -R grid.oinstall /u01/app/oracle/votedisk

2.3 添加 quorum fg disk
SQL> alter system set asm_diskstring="/dev/asm*","/u01/app/oracle/votedisk/vote*" sid='*' scope=both;
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01';

SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;

GROUP_NUMBER DISK_NUMBER      OS_MB NAME                           PATH            FAILGROUP                      STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
           1           0       2048 CRSDG_0000                     /dev/asm-crs01  CRSDG_0000                     NORMAL
           1           1       2048 CRSDG_0001                     /dev/asm-crs02  CRSDG_0001                     NORMAL
           1           2       2048 CRSDG_0002                     /dev/asm-crs03  CRSDG_0002                     NORMAL
           1           3       2048 CRSDG_0003                     /dev/asm-crs04  CRSDG_0003                     NORMAL
           1           4       2048 CRSDG_0004                     /dev/asm-crs05  CRSDG_0004                     NORMAL
           1           5       2048 CRSDG_0005                     /u01/app/oracle FGQ                            NORMAL      <==
                                                                   /votedisk/voted
                                                                   isk01

           2           0      20480 DATADG_0000                    /dev/asm-data1  DATADG_0000                    NORMAL
           2           1      20480 DATADG_0001                    /dev/asm-data2  DATADG_0001                    NORMAL
 
2.4 删除多余的磁盘    
在quorum fg添加后,可以看到还没有votedisk切换到其上
这时,我们将删除有3个投票盘阵列上的一个投票盘

[grid@database1 votedisk]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
 2. ONLINE   10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
 3. ONLINE   0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
 4. ONLINE   83fa917a0e844f23bf27238aff51b57a (/dev/asm-crs05) [CRSDG]
 5. ONLINE   18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]

SQL> alter diskgroup crsdg drop disk CRSDG_0004;                                                 <==

[grid@database1 votedisk]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
 2. ONLINE   10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
 3. ONLINE   0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
 4. ONLINE   18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
 5. ONLINE   6fc616a6923a4fb2bffca18e44a58533 (/u01/app/oracle/votedisk/votedisk01) [CRSDG]      <==

***********************************************************
相关配置
***********************************************************
对于数据库DiskGroup,创建时应该为不同阵列的盘,指定好FAILGROUP,以确保2份数据(NORMAL冗余)放在不同的阵列中
修改参数asm_preferred_read_failure_groups,不同的节点,从不同的FG组中读取数据
 
***********************************************************
测试
***********************************************************
1.NFS服务器宕机

2014-02-13 16:56:38.476:
[cssd(3577)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 99910 milliseconds
2014-02-13 16:57:28.541:
[cssd(3577)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 49850 milliseconds
2014-02-13 16:57:58.565:
[cssd(3577)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 19830 milliseconds
2014-02-13 16:58:18.573:
[cssd(3577)]CRS-1604:CSSD voting file is offline: /u01/app/oracle/votedisk/votedisk01; details at (:CSSNM00058:) in /u01/app/11.2.0/grid/log/database1/cssd/ocssd.log.

[root@database1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
 2. ONLINE   10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
 3. ONLINE   0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
 4. ONLINE   18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).

集群件运行正常此时

恢复
[root@database1 ~]# umount -f /u01/app/oracle/votedisk
[root@database1 ~]# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk

GROUP_NUMBER DISK_NUMBER      OS_MB NAME                           PATH                           FAILGROUP                      STATE
------------ ----------- ---------- ------------------------------ ------------------------------ ------------------------------ --------
           0           5       2048                                /u01/app/oracle/votedisk/voted                                NORMAL  <== 文件此时并不属于CRSDG
                                                                   isk01
           0           9       2048                                /dev/asm-crs05                                                NORMAL
           1           0       2048 CRSDG_0000                     /dev/asm-crs01                 CRSDG_0000                     NORMAL
           1           1       2048 CRSDG_0001                     /dev/asm-crs02                 CRSDG_0001                     NORMAL
           1           2       2048 CRSDG_0002                     /dev/asm-crs03                 CRSDG_0002                     NORMAL
           1           3       2048 CRSDG_0003                     /dev/asm-crs04                 CRSDG_0003                     NORMAL
           1           4          0 CRSDG_0004                                                    FGQ                            NORMAL  <==
           2           0      20480 DATADG_0000                    /dev/asm-data1                 DATADG_0000                    NORMAL
           2           1      20480 DATADG_0001                    /dev/asm-data2                 DATADG_0001                    NORMAL


[root@database1 votedisk]# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048      <== 重新格式化Voting disk

SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01';         <== 重新添加

GROUP_NUMBER DISK_NUMBER      OS_MB NAME                           PATH            FAILGROUP                      STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
           0           4       2048                                /dev/asm-crs05                                 NORMAL
           1           0       2048 CRSDG_0000                     /dev/asm-crs01  CRSDG_0000                     NORMAL
           1           1       2048 CRSDG_0001                     /dev/asm-crs02  CRSDG_0001                     NORMAL
           1           2       2048 CRSDG_0002                     /dev/asm-crs03  CRSDG_0002                     NORMAL
           1           3       2048 CRSDG_0003                     /dev/asm-crs04  CRSDG_0003                     NORMAL
           1           4       2048 CRSDG_0004                     /u01/app/oracle FGQ                            NORMAL        <==
                                                                   /votedisk/voted
                                                                   isk01

           1           5          0 CRSDG_0005                                     FGQ                            NORMAL        <==
           2           0      20480 DATADG_0000                    /dev/asm-data1  DATADG_0000                    NORMAL
           2           1      20480 DATADG_0001                    /dev/asm-data2  DATADG_0001                    NORMAL

SQL> alter diskgroup crsdg drop quorum disk CRSDG_0005 force;                                           <== 将旧的删除

GROUP_NUMBER DISK_NUMBER      OS_MB NAME                           PATH            FAILGROUP                      STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
           0           4       2048                                /dev/asm-crs05                                 NORMAL        <==
           1           0       2048 CRSDG_0000                     /dev/asm-crs01  CRSDG_0000                     NORMAL
           1           1       2048 CRSDG_0001                     /dev/asm-crs02  CRSDG_0001                     NORMAL
           1           2       2048 CRSDG_0002                     /dev/asm-crs03  CRSDG_0002                     NORMAL
           1           3       2048 CRSDG_0003                     /dev/asm-crs04  CRSDG_0003                     NORMAL
           1           4       2048 CRSDG_0004                     /u01/app/oracle FGQ                            NORMAL        <==
                                                                   /votedisk/voted
                                                                   isk01

           2           0      20480 DATADG_0000                    /dev/asm-data1  DATADG_0000                    NORMAL
           2           1      20480 DATADG_0001                    /dev/asm-data2  DATADG_0001                    NORMAL

2.先于NFS服务器启动GRID
[root@database1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
 2. ONLINE   10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
 3. ONLINE   0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
 4. ONLINE   18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
 5. OFFLINE  ad0268dcec654f14bfe60497d9490780 () []
Located 5 voting disk(s).

[root@database1 ~]# crsctl stat res -t                                             <== 无法启动
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[root@database2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  INTERMEDIATE database2                OCR not started       <== ASM启动,但是CRSDG无法MOUNT
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       database2                                    
ora.crf
      1        ONLINE  ONLINE       database2                                    
ora.crsd
      1        ONLINE  OFFLINE                                                   
ora.cssd
      1        ONLINE  ONLINE       database2                                    
ora.cssdmonitor
      1        ONLINE  ONLINE       database2                                    
ora.ctssd
      1        ONLINE  ONLINE       database2                ACTIVE:0            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.drivers.acfs
      1        ONLINE  ONLINE       database2                                    
ora.evmd
      1        ONLINE  INTERMEDIATE database2                                    
ora.gipcd
      1        ONLINE  ONLINE       database2                                    
ora.gpnpd
      1        ONLINE  ONLINE       database2                                    
ora.mdnsd
      1        ONLINE  ONLINE       database2    
      
SQL> alter diskgroup crsdg mount;
alter diskgroup crsdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "4" is missing from group number "1"

SQL> alter diskgroup crsdg mount force;                     <== 强制MOUNT

Diskgroup altered.

[root@database1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.DATADG.dg
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.LISTENER_1522.lsnr
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.LISTENER_1523.lsnr
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.asm
               ONLINE  ONLINE       database1                Started             
               ONLINE  ONLINE       database2                Started             
ora.gsd
               OFFLINE OFFLINE      database1                                    
               OFFLINE OFFLINE      database2                                    
ora.net1.network
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.ons
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
ora.registry.acfs
               ONLINE  ONLINE       database1                                    
               ONLINE  ONLINE       database2                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       database1                                    
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.database1.vip
      1        ONLINE  ONLINE       database1                                    
ora.database2.vip
      1        ONLINE  ONLINE       database2                                    
ora.oc4j
      1        ONLINE  ONLINE       database1                                    
ora.orcl.billa3.svc
      1        ONLINE  ONLINE       database1                                    
ora.orcl.db
      1        ONLINE  ONLINE       database1                Open                
      2        ONLINE  ONLINE       database2                Open                
ora.scan1.vip
      1        ONLINE  ONLINE       database1   
      
[root@database1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
 2. ONLINE   10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
 3. ONLINE   0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
 4. ONLINE   18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).      

恢复和测试1相同

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/8242091/viewspace-1081078/,如需转载,请注明出处,否则将追究法律责任。

上一篇: PCM锁与BLOCK资源
请登录后发表评论 登录
全部评论

注册时间:2009-04-06

  • 博文量
    251
  • 访问量
    953201