ITPub博客

首页 > Linux操作系统 > Linux操作系统 > 对10gR2 RAC环境中在线添加、删除Votedisk和在线替换OCR的探讨

对10gR2 RAC环境中在线添加、删除Votedisk和在线替换OCR的探讨

原创 Linux操作系统 作者:尛样儿 时间:2012-06-13 20:56:06 0 删除 编辑
         这里我们不讨论11gR2 RAC在线添加、删除Votedisk和OCR,在11gR2平台可以非常容易的实现。这里也不讨论Votedisk和OCR的还原,还原操作必须在停止Clusterware的情况下完成。下面以实际的例子讨论10gR2 RAC for Linux环境下,Votedisk和OCR的在线添加和删除。

一.Votedisk的在线添加和删除。
1).查看集群资源的状态:
[root@rhel1 bin]# ./crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....L1.lsnr application    ONLINE    ONLINE    rhel1
ora.rhel1.gsd  application    ONLINE    ONLINE    rhel1
ora.rhel1.ons  application    ONLINE    ONLINE    rhel1
ora.rhel1.vip  application    ONLINE    ONLINE    rhel1
ora....L2.lsnr application    ONLINE    ONLINE    rhel2
ora.rhel2.gsd  application    ONLINE    ONLINE    rhel2
ora.rhel2.ons  application    ONLINE    ONLINE    rhel2
ora.rhel2.vip  application    ONLINE    ONLINE    rhel2

2).查看当前votedisk位置及文件名:
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u02/vdisk/vdisk1
 1.     0    /u02/vdisk/vdisk2
 2.     0    /u02/vdisk/vdisk3

located 3 votedisk(s).

3).查看ocssd进程号:
[root@rhel1 bin]# ps -e | grep -i ocssd
 5942 ?        00:00:21 ocssd.bin

4).查看ocssd进程定位的votedisk文件:
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
        加粗的文件说明当前ocssd定位的votedisk文件。分别是vdisk1、vdisk2和vdisk3。 

5).在线添加Votedisk文件:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force
Now formatting voting disk: /u03/vdisk/vdisk4
successful addition of votedisk /u03/vdisk/vdisk4.
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force
Now formatting voting disk: /u03/vdisk/vdisk5
successful addition of votedisk /u03/vdisk/vdisk5.
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force
Now formatting voting disk: /u03/vdisk/vdisk6
successful addition of votedisk /u03/vdisk/vdisk6.
        执行上面的命令在线添加了3个votedisk文件到不同的ocfs2文件系统上,这样做的目的是为了实现存储设备的在线迁移。

6).查看当前的votedisk路径及文件名:
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u02/vdisk/vdisk1
 1.     0    /u02/vdisk/vdisk2
 2.     0    /u02/vdisk/vdisk3
 3.     0    /u03/vdisk/vdisk4
 4.     0    /u03/vdisk/vdisk5
 5.     0    /u03/vdisk/vdisk6

located 6 votedisk(s).
从上面的输出结果可以看出,3个votedisk文件添加是成功的。

7).查看ocssd进程定位的votedisk文件:
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
        从上面的输出可以看出,ocssd进程定位的votedisk文件并未发生变化,可以猜测,ocssd对votedisk的更新也不会同步到新 添加的3个votedisk文件中,那么在长时间的运行之后,重启服务器时新添加的votedisk过旧。

8).在线删除votedisk文件:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk1 -force
successful deletion of votedisk /u02/vdisk/vdisk1.
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk2 -force
successful deletion of votedisk /u02/vdisk/vdisk2.
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk3 -force
successful deletion of votedisk /u02/vdisk/vdisk3.

9).查看删除votedisk后的votedisk路径及文件名:
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk4
 1.     0    /u03/vdisk/vdisk5
 2.     0    /u03/vdisk/vdisk6

located 3 votedisk(s).

10).查看在线删除votedisk之后ocssd进程定位的votedisk文件:
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
定位未发生任何变化。

下面我们来尝试重启Clusterware,看会发生什么情况。

11).停止Clustereware:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
在所有RAC节点停止Clusterware。

12).启动Clusterware:
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
只在一个节点启动Clusterware。

13).查看CRS组件的状态及votedisk文件:
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    0
 1.     0    0
 2.     0    /u03/vdisk/vdisk6

located 3 votedisk(s).
         可以发现,votedisk只有vdisk6显示正常,至少可以说明这种情况是不正常的,可以大胆猜想是由于在线增加votedisk引起的问题。

14).查看ocssd进程定位的votedisk文件:
[root@rhel1 bin]# ps -e | grep -i ocssd
 9386 ?        00:00:04 ocssd.bin
[root@rhel1 bin]# cd /proc/9386/fd/
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 10 -> socket:[44767]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 11 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 12 -> /u03/vdisk/vdisk6
l-wx------ 1 oracle oinstall 64 Jun 12 18:30 13 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 14 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 15 -> socket:[44950]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 16 -> socket:[44953]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 17 -> socket:[45584]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 18 -> socket:[44980]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 19 -> socket:[44983]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 20 -> socket:[44986]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 21 -> socket:[44988]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 22 -> socket:[44992]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 23 -> socket:[44993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 24 -> socket:[45585]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 25 -> socket:[45586]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 26 -> socket:[45587]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 27 -> socket:[45588]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:30 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 9 -> socket:[44766]
从上面的结果可以看出,ocssd进程只定位了vdisk6这一个文件。

15).停止所有节点的Clusterware:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:01 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:23 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:23 pts/3    00:00:00 -bash
oracle   10433  8198  0 18:29 pts/3    00:00:00 tail -f alertrhel1.log
root     13299  3340  0 18:34 pts/0    00:00:00 grep ora
root     30740  9263  0 17:59 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:59 pts/2    00:00:00 -bash

16).手动删除没有使用的vdisk4和vdisk5:
[root@rhel1 bin]# cd /u03/vdisk/
[root@rhel1 vdisk]# rm -rf vdisk4 vdisk5

17).在Clusterware脱机的情况下添加vdisk4和vdisk5:
[root@rhel1 vdisk]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force
Now formatting voting disk: /u03/vdisk/vdisk4
successful addition of votedisk /u03/vdisk/vdisk4.
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force
Now formatting voting disk: /u03/vdisk/vdisk5
successful addition of votedisk /u03/vdisk/vdisk5.

17).修改votedisk正确的权限:
[root@rhel1 bin]# cd -
/u03/vdisk
[root@rhel1 vdisk]# ll
total 30000
-rw-r--r-- 1 root   root     10240000 Jun 12 18:37 vdisk4
-rw-r--r-- 1 root   root     10240000 Jun 12 18:37 vdisk5
-rw-r--r-- 1 oracle oinstall 10240000 Jun 12 18:19 vdisk6
[root@rhel1 vdisk]# chown oracle:oinstall *

18).启动Clusterware:
[root@rhel1 vdisk]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check css
CSS appears healthy
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

19).查看votedisk路径及文件名:
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk4
 1.     0    /u03/vdisk/vdisk5
 2.     0    0
 3.     0    0
 4.     0    /u03/vdisk/vdisk4

located 5 votedisk(s).
        可以看出vdisk4和vdisk5被正确添加,vdisk6不见了踪迹,说明10g votedisk在Clusterware的情况下是可以添加的,并且在脱机的情况下添加之后启动ocssd进程会立即定位到这些votedisk文件,所以这并不会导致前面新添加的votedisk未被定位的情况发生。

20).添加vdisk6和清理有问题的votedisk:
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:02 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:22 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:22 pts/3    00:00:00 -bash
oracle   20770  8198  0 18:50 pts/3    00:00:00 tail -f alertrhel1.log
root     21487  3340  0 18:51 pts/0    00:00:00 grep ora
root     30740  9263  0 17:59 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:59 pts/2    00:00:00 -bash
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force
Now formatting voting disk: /u03/vdisk/vdisk6
successful addition of votedisk /u03/vdisk/vdisk6.
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk4
 1.     0    /u03/vdisk/vdisk5
 2.     0    /u03/vdisk/vdisk6
 3.     0    0
 4.     0    0
 5.     0    /u03/vdisk/vdisk4

located 6 votedisk(s).
[root@rhel1 bin]# ./crsctl delete css votedisk 0 -force
successful deletion of votedisk 0.
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk4
 1.     0    /u03/vdisk/vdisk5
 2.     0    /u03/vdisk/vdisk6
 3.     0    0
 4.     0    /u03/vdisk/vdisk4

located 5 votedisk(s).
       从上面返回的结果可以看出,虽然votedisk的路径和文件都是0,在删除0的时候也不会被全部删除,只会删除其中的一条。

[root@rhel1 bin]# ./crsctl delete css votedisk 0 -force
successful deletion of votedisk 0.
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk4
 1.     0    /u03/vdisk/vdisk5
 2.     0    /u03/vdisk/vdisk6
 3.     0    /u03/vdisk/vdisk4

located 4 votedisk(s).
[root@rhel1 bin]# ./crsctl delete css votedisk /u03/vdisk/vdisk4 -force
successful deletion of votedisk /u03/vdisk/vdisk4.
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk5
 1.     0    /u03/vdisk/vdisk6
 2.     0    /u03/vdisk/vdisk4

located 3 votedisk(s).
删除 /u03/vdisk/vdisk4的votedisk文件也只删除了其中一条记录。

21).启动Clusterware:
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@rhel1 bin]# ./crsctl query css votedisk
 0.     0    /u03/vdisk/vdisk5
 1.     0    /u03/vdisk/vdisk6
 2.     0    /u03/vdisk/vdisk4

located 3 votedisk(s).

        从上面的例子可以说明一点,在10gR2 RAC 的环境中最好不要在Clusterware在线的情况下添加votedisk,虽然可以成功添加,但在重启Clusterware之后可能出现意想不到的问题。

二.OCR的在线替换(添加和删除)。 
1).检查OCR的完整性:
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2512
         Available space (kbytes) :     259608
         ID                       : 2111044529
         Device/File Name         : /u02/ocr/ocr1
                                    Device/File integrity check succeeded
         Device/File Name         : /u02/ocr/ocr2
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded

2).提前将需要创建的ocr文件touch:
[root@rhel1 bin]# su - oracle
[oracle@rhel1 ~]$
[oracle@rhel1 ~]$ cd /u03/ocr/
[oracle@rhel1 ocr]$ ls
[oracle@rhel1 ocr]$ touch ocr3
[oracle@rhel1 ocr]$ touch ocr4
        上面添加votedisk的时候并未提前touch votedisk文件,但某些情况下votedisk文件也需要提前touch。

3).替换ocrmirror文件,实际是一个迁移ocr的过程:
[root@rhel1 bin]# ./ocrconfig -replace ocrmirror /u03/ocr/ocr4
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2512
         Available space (kbytes) :     259608
         ID                       : 2111044529
         Device/File Name         : /u02/ocr/ocr1
                                    Device/File integrity check succeeded
         Device/File Name         : /u03/ocr/ocr4
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded
        迁移后检查ocr的完整性,没问题。

4).查看ocssd进程定位的ocr文件:
ocr文件也是由ocssd进程来维护。
[root@rhel1 bin]# ps -e | grep -i ocssd
23379 ?        00:00:06 ocssd.bin
[root@rhel1 bin]# cd /proc/23379/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 10 -> socket:[62386]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 14 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 15 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 16 -> /u03/vdisk/vdisk4
l-wx------ 1 oracle oinstall 64 Jun 12 19:00 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 19 -> socket:[62607]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 20 -> socket:[62610]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 21 -> socket:[62611]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 22 -> socket:[62627]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 23 -> socket:[62632]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 24 -> socket:[62637]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 25 -> socket:[62640]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 26 -> socket:[62645]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 27 -> socket:[62646]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 28 -> socket:[62650]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 29 -> socket:[62651]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 30 -> socket:[62652]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 31 -> socket:[62653]
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 19:00 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 9 -> socket:[62385]
       从上面的结果可以看出,ocssd进程定位的ocr文件还是oc1和ocr2,并未随着ocr的迁移发生改变,和votedisk的效果相同。

5).停止Clusterware:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
停止所有节点的Clusterware。

[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:02 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:22 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:22 pts/3    00:00:00 -bash
root     26683  3340  0 19:01 pts/0    00:00:00 grep ora
root     30740  9263  0 17:58 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:58 pts/2    00:00:00 -bash

6).在Clusterware脱机的情况下迁移ocr文件:
[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3
PROT-1: Failed to initialize ocrconfig
由此可以说明OCR文件不能在脱机的情况下迁移。

7).启动Clusterware:
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2512
         Available space (kbytes) :     259608
         ID                       : 2111044529
         Device/File Name         : /u02/ocr/ocr1
                                    Device/File integrity check succeeded
         Device/File Name         : /u03/ocr/ocr4
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded

8).在线替换ocr文件:
[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2512
         Available space (kbytes) :     259608
         ID                       : 2111044529
         Device/File Name         : /u03/ocr/ocr3
                                    Device/File integrity check succeeded
         Device/File Name         : /u03/ocr/ocr4
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded
        替换成功,OCR文件一致性检查成功。

9).检查ocssd进程定位的ocr文件:
[root@rhel1 bin]# ps -e | grep ocssd
28226 ?        00:00:03 ocssd.bin
[root@rhel1 bin]# cd /proc/28226/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
l-wx------ 1 oracle oinstall 64 Jun 12 19:04 10 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 14 -> /u03/vdisk/vdisk6
l-wx------ 1 oracle oinstall 64 Jun 12 19:04 15 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 16 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 17 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 18 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 19 -> socket:[69253]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 20 -> socket:[69256]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 21 -> socket:[69257]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 22 -> socket:[69285]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 23 -> socket:[69303]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 24 -> socket:[69346]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 25 -> socket:[69353]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 26 -> socket:[69370]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 27 -> socket:[69371]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 28 -> socket:[69379]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 29 -> socket:[69380]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 30 -> socket:[69381]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 31 -> socket:[69382]
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 6 -> /u03/ocr/ocr4
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 8 -> socket:[68925]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 9 -> socket:[68926]
定位文件未发生变化,需要重启才能定位新替换的OCR文件。

10).重启Clusterware:
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

11).查看ocssd进程定位的ocr文件:
[root@rhel1 bin]# ps -e | grep ocssd
30773 ?        00:00:02 ocssd.bin
[root@rhel1 bin]# cd /proc/30773/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 10 -> socket:[73581]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 14 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 15 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 16 -> /u03/vdisk/vdisk4
l-wx------ 1 oracle oinstall 64 Jun 12 19:05 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 19 -> socket:[73764]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 20 -> socket:[73767]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 22 -> socket:[73769]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 23 -> socket:[73772]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 24 -> socket:[73774]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 25 -> socket:[73778]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 26 -> socket:[73780]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 27 -> socket:[73781]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 28 -> socket:[73787]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 29 -> socket:[73788]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 30 -> socket:[73789]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 31 -> socket:[73790]
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 5 -> /u03/ocr/ocr3
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 6 -> /u03/ocr/ocr4
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 19:05 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 9 -> socket:[73580]
重启后ocssd进程定位ocr文件成功。

        从上面的OCR替换的例子可以看出,OCR的替换(添加和删除)只能在Clusterware在线的情况下才能执行,最好在替换成功,检查OCR一致性正确之后立即重启Clusterware,使得替换的OCR文件生效。

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/23135684/viewspace-732734/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
Oracle数据库管理员,Oracle数据库系统构架员;2012年7月出版《构建最高可用Oracle数据库系统:Oracle 11gR2 RAC管理、维护与性能优化》一书;Oracle 10g OCM。

注册时间:2010-01-05

  • 博文量
    483
  • 访问量
    5327571