参考文档http://space.itpub.net/7607759/viewspace-620131

Oralce文档：270512.1

现有环境rac1 rac2,增加节点rac3

前期工作：

1 复制rac1节点到rac3节点

2 建立相同ORACLE_HOME CRS_HOME

3 验证oracle 登陆其他机器等效性（无需密码登陆）（在各个机器上都要互相测试，真实ip和priv ip都要测试）----非常重要

4 验证ASM磁盘有效性

[root@rac3 raw]# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@rac3 raw]# /etc/init.d/oracleasm listdisks

VOL1

VOL2

5  If the CRS Home is owned by root and you are on a version < 10.1.0.4, change 

the ownership of the CRS Home directories on all nodes to the Oracle user

so that OUI can read and write to these directories

开始：

首先是检查安装环境，仍然是使用runcluvfy.sh脚本来进行验证，该脚本可以在现有配置中的任意节点上执行，这里在节点1执行(一定要以oracle用户运行，否则无法通过)，如下：

l $cd /stage/cluster/clusterware_64/cluvfy

[oracle@rac1 cluvfy]$ export LANG=C

[oracle@rac1 cluvfy]$ ./runcluvfy.sh stage -pre crsinst -n rac3 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rac1"

Destination Node Reachable?

------------------------------------ ------------------------

rac3 yes

Result: Node reachability check passed from node "rac1".

Checking user equivalence...

Check: User equivalence for user "oracle"

Node Name Comment

------------------------------------ ------------------------

rac3 passed

Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle"

Node Name User Exists Comment

------------ ------------------------ ------------------------

rac3 yes passed

Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"

Node Name Status Group ID

------------ ------------------------ ------------------------

rac3 exists 500

Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]

Node Name User Exists Group Exists User in Group Primary Comment

---------------- ------------ ------------ ------------ ------------ ------------

rac3 yes yes yes yes passed

Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.

Administrative privileges check passed.

Checking node connectivity...

Interface information for node "rac3"

Interface Name IP Address Subnet

------------------------------ ------------------------------ ----------------

eth0 192.168.0.253 192.168.0.0

eth1 10.10.10.3 10.10.10.0

Check: Node connectivity of subnet "192.168.0.0"

Result: Node connectivity check passed for subnet "192.168.0.0" with node(s) rac3.

Check: Node connectivity of subnet "10.10.10.0"

Result: Node connectivity check passed for subnet "10.10.10.0" with node(s) rac3.

Suitable interfaces for the private interconnect on subnet "192.168.0.0":

rac3 eth0:192.168.0.253

Suitable interfaces for the private interconnect on subnet "10.10.10.0":

rac3 eth1:10.10.10.3

ERROR:

Could not find a suitable set of interfaces for VIPs.

Result: Node connectivity check failed.

Checking system requirements for 'crs'...

No checks registered for this product.

Pre-check for cluster services setup was unsuccessful on all the nodes.

[oracle@rac1 cluvfy]$

如果返回信息中提示“Could not find a suitable set of interfaces for VIPs.”，可以忽略该错误信息，这是一个bug，Metalink中有详细说明，doc.id:338924.1。

　　没有其它错误的话，安装可以正常进行，下面准备正式进入安装阶段。

安装clusterware到新节点

　　新节点中clusterware的安装也是从现有的RAC环境中开始的，在当前RAC环境中任意节点的$ORA_CRS_HOME，执行oui/bin/addNode.sh，操作如下：

错误提示：OUI-10009:There are no new nodes to add to this installation

参考http://studycow.itpub.net/ linux下添加10g rac节点OUI-10009错误的解决办法

一linux下的10g RAC环境，使用$ORACLE_HOME/oui/bin/addNode.sh在新节点上安装数据库软件时，出现OUI-10009错误：
OUI-10009 --> There are no new nodes to add
提示没有新添加的节点，可以使用如下命令添加节点
./addNode.sh -silent "CLUSTER_NEW_NODES={rac3}" -logLevel trace -debug

如果节点名称是已经删除过的节点，安装时会提示已经安装了节点，这时候可以使用下列命令先更新节点信息,再安装新节点：
$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=Oracle_home "CLUSTER_NODES={remaining_nodes_list}"

注意：debug可以不开启，无非是日志少了些，可是rac3前后的括号是一定要有的，否则还会提示10009错误，如下图

下面重新安装

$./addNode.sh "CLUSTER_NEW_NODES={rac3}"

根据提示在rac3运行root.sh

[root@rac3 db_1]# sh root.sh

Running Oracle10 root.sh script...

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /opt/ora10g/product/10.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying dbhome to /usr/local/bin ...

The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying oraenv to /usr/local/bin ...

The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)

[n]: y

Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

/bin/cat: /opt/ora10g/product/10.2.0/dbhome/evm/admin/conf/evm.auth: 没有那个文件或目录

/bin/cat: /opt/ora10g/product/10.2.0/dbhome/evm/admin/conf/evm.auth: 没有那个文件或目录

root.sh: line 459: /opt/ora10g/product/10.2.0/dbhome/evm/admin/conf/evm.auth.11174: 没有那个文件或目录

/bin/cp: 无法 stat “/opt/ora10g/product/10.2.0/dbhome/evm/admin/conf/evm.auth.11174”: 没有那个文件或目录

root.sh: line 470: /opt/ora10g/product/10.2.0/dbhome/bin/evmreload: 没有那个文件或目录

[root@rac3 db_1]#

报错了，检查crs状态

[oracle@rac1 bin]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....B1.inst application ONLINE ONLINE rac1

ora....B2.inst application ONLINE ONLINE rac2

ora.RACDB.db application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

怎么没有rac3节点呢？

看来有问题啊，而且上面提示

/opt/ora10g/product/10.2.0/dbhome/evm/admin/conf/evm.auth.11174也说明找错了crs_home,看来这种安装无法选择HOME目录有些问题啊

于是想办法先删除节点，然后重新添加

参考 Note:239998.1

Subject:  10g RAC: How to Clean Up After a Failed CRS Install
  Doc ID:  Note:239998.1 Type:  BULLETIN
  Last Revision Date:  25-OCT-2005 Status:  PUBLISHED
10g RAC: How to Clean Up After a Failed CRS Install
---------------------------------------------------
Not cleaning up a failed CRS install can cause problems like node reboots.
Follow these steps to clean up a failed CRS install:
1. Run the rootdelete.sh script. then the rootdeinstall.sh script. from the
$ORA_CRS_HOME/install directory.  Running these scripts should be sufficent
to clean up your CRS install.  If you have any problems with these scripts
please open a service request. 
If for some reason you have to manually remove the install due to problems
with the scripts, continue to step 2:
2. Stop the Nodeapps on all nodes:（需要）
        srvctl stop nodeapps -n
3. Prevent CRS from starting when the node boots.  To do this issue the following
as root:
……………………
Linux:（这步需要做）
        rm -f /etc/init.d/init.cssd
        rm -f /etc/init.d/init.crs
        rm -f /etc/init.d/init.crsd
        rm -f /etc/init.d/init.evmd
        rm -f /etc/rc2.d/K96init.crs
        rm -f /etc/rc2.d/S96init.crs
        rm -f /etc/rc3.d/K96init.crs
        rm -f /etc/rc3.d/S96init.crs
        rm -f /etc/rc5.d/K96init.crs
        rm -f /etc/rc5.d/S96init.crs
        rm -Rf /etc/oracle/scls_scr
        rm -f /etc/inittab.crs
        cp /etc/inittab.orig /etc/inittab

4. If they are not already down, kill off EVM, CRS, and CSS processes or reboot
   the node:（需要做）
        ps -ef | grep crs       
        kill
        ps -ef | grep evm
        kill
        ps -ef | grep css       
        kill
   Do not kill any OS processes, for example icssvr_daemon process !
5. If there is no other Oracle software running (like listeners, DB's, etc...),
   you can remove the files in /var/tmp/.oracle or /tmp/.oracle.  Example:
        rm -f /var/tmp/.oracle（需要）
        or
        rm -f /tmp/.oracle
6. Remove the ocr.loc （需要）
   Usually the ocr.loc can be found at /etc/oracle
7. Remove the CRS install location:（需要）
          rm -Rf /*
重新删掉新节点所有相关文件，重新添加

这次不加参数居然可以了，奇怪（最后才知道前面运行root.sh的时候报错是因为rac3节点的ORACLE_HOME和ORA_CRS_HOME定义颠倒了，导致他吧clusterware软件装到了ORACLE_HOME下而不是CRS_HOME下，三个节点CRS_HOME不一致能正常才怪呢）

这是时间不同步的问题，先不理会他，继续

#./rootaddnode.sh

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Attempting to add 1 new nodes to the configuration

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node :

node 3: rac3 rac3-priv rac3

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

/opt/ora10g/product/10.2.0/dbhome/jdk/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

awk: (FILENAME=- FNR=1) fatal: attempt to access field -1

/opt/ora10g/product/10.2.0/dbhome/bin/srvctl add nodeapps -n rac3 -A rac3-vip// -o /opt/ora10g/product/10.2.0/dbhome

/opt/ora10g/product/10.2.0/dbhome/jdk/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

解决方法：

在rac1节点，修改$CRS_HOME/bin/vipca and srvctl，添加如下行

if [ "$arch" = "i686" -o "$arch" = "ia64" -o "$arch" = "x86_64" ]
then
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
fi
unset LD_ASSUME_KERNEL <<<== Line to be added

然后重新运行./rootaddnode.sh

# ./rootaddnode.sh

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Node rac3 is already assigned nodenum 3.

Aborting: No configuration data has been changed.

clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...

[-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]

-nn specifies nodenames in the same fashion as -nn in -install mode

-pn specifies private interconnect names as -pn in -install mode

-hn specifies hostnames in the same fashion as -hn in -install mode

-t specifies port numbers to be used by CRS daemons on the new node(s)

default ports: 49895,49896,49897,49898

WARNING: Using this tool may corrupt your cluster configuration. Do not

use unless you positively know what you are doing.

/opt/ora10g/product/10.2.0/dbhome/bin/srvctl add nodeapps -n rac3 -A rac3-vip/255.255.255.0/eth0 -o /opt/ora10g/product/10.2.0/dbhome

[root@rac1 install]#

接着在rac3运行root.sh

解决方法就是先注释掉那个参数，然后用root登陆图形界面运行vipca

方法和安装rac的时候一样

自此，新节点添加clusterware完成

验证

[oracle@rac2 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....B1.inst application ONLINE ONLINE rac1

ora....B2.inst application ONLINE ONLINE rac2

ora.RACDB.db application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

ora.rac3.gsd application ONLINE ONLINE rac3

ora.rac3.ons application ONLINE ONLINE rac3

ora.rac3.vip application ONLINE ONLINE rac3

[oracle@rac2 ~]$