RHEL5上的11gr2 RAC在线添加一个节点
前提：已经在rhel5上部署了一套rac系统（本文中存储部分使用的是iscsi服务，单独用一台服务器做了存储）；
之前的部署环境
（RHEL5.8）：
iscsi服务器端 : 192.168.8.214
RAC端：
192.168.8.216 slot7
192.168.8.215 slot6
10.0.0.20 slot7_pri
10.0.0.10 slot6_pri
192.168.8.219 slot7_vip
192.168.8.218 slot6_vip
192.168.8.220 scanip
添加新节点后的部署规划
iscsi服务器端 : 192.168.8.214
RAC端：
192.168.8.216 slot7
192.168.8.215 slot6
192.168.8.217 slot8
10.0.0.20 slot7_pri
10.0.0.10 slot6_pri
10.0.0.30 slot8_pri
192.168.8.219 slot7_vip
192.168.8.218 slot6_vip
192.168.8.221 slot8_vip
192.168.8.220 scanip
1\ iscsi客户端（在rhel5.8中已经默认安装了iscsi服务）
[root@slot6 ~]# chkconfig --list|grep iscsi
iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
chkconfig iscsi on
/etc/init.d/iscsi start
[root@slot6 ~]# iscsiadm -m discovery -p 192.168.8.214 -t sendtargets
192.168.8.214:3260,1 iqn.2001-04.com.example:storage.disk2.sys1.xyz
[root@slot6 ~]# iscsiadm --mode node --targetname iqn.2001-04.com.example:storage.disk2.sys1.xyz --portal 192.168.8.214:3260 --login
3\ 安装grid--创建用户并设置密码
Grid Infrastructure 操作系统用户grid , 主组为oinstall， 辅助组为asmadmin, asmdba, asmoper
Oracle RAC 操作系统用户 oracle , 主组为oinstall , 辅助组为dba, oper , asmdba
groupadd -g 501 oinstall
groupadd -g 502 asmadmin
groupadd -g 503 asmdba
groupadd -g 504 asmoper
groupadd -g 505 dba
groupadd -g 506 oper
useradd -u 500 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
useradd -u 501 -g oinstall -G asmdba,dba,oper oracle
passwd grid
passwd oracle
4\ 安装grid--网络设置
eth0 public
eth1 private
具体ip规划见开始环境部署说明。
5\ 安装grid--各个节点时间同步
iscsi服务端作为ntp服务器；
ntp服务端：
[root@slot5 iscsi]# chkconfig --list |grep ntp
ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
chkconfig ntpd on
/etc/init.d/ntpd start
[root@slot5 iscsi]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(0) .LOCL. 10 l 38 64 377 0.000 0.000 0.001
ntp客户端（rac节点）：
chkconfig ntpd on
vi /etc/ntp.conf 将server 127.0.0.1 修改为：
server 192.168.8.214 # local clock
/etc/init.d/ntpd start
[root@slot6 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*192.168.8.214 LOCAL(0) 11 u 17 64 377 0.105 -0.322 0.004
过段时间节点间时间就会同步了。
另外，为防止在检测时报ntp时间同步错误：
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
– Cause: NTP daemon on one or more nodes lacked slewing option.
– Action: Shut down and restart the NTP daemon after setting the slewing option as follows:
For Linux, edit /etc/sysconfig/ntpd and add -x to the command line option.
For SUSE Linux, edit /etc/sysconfig/ntp and add -x to the OPTIONS variable.
For AIX, edit /etc/rc.tcpip and add -x to the command line option.
For HP-UX, edit /etc/rc.config.d/netdaemons and add -x to the command line option.
For Solaris, edit /etc/inet/ntp.conf and add ‘slewalways yes’ and ‘disable pll’ in ntp.conf.
需要 vi /etc/sysconfig/ntpd 将OPTIONS后添加 -x ，如下：
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
然后重启ntp服务；
/etc/init.d/ntpd restart
6\ 安装grid--配置内核参数
kernel.shmall=2097152
#kernel.shmmax=1073741824
kernel.shmmni=4096
kernel.sem=250 32000 100 128
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048576
fs.file-max = 6815744
fs.aio-max-nr = 1048576
然后执行 sysctl -p
7\ 安装grid--修改limit
vi /etc/security/limits.conf 添加
grid soft nproc 2047
grid hard nproc 32768
grid soft nofile 1024
grid hard nofile 250000
oracle soft nproc 2047
oracle hard nproc 32768
oracle soft nofile 1024
oracle hard nofile 250000
8\ 安装grid--修改/etc/pam.d/login
vi /etc/pam.d/login 添加
session required pam_limits.so
9\ 安装grid--创建目录
规划(grid的base目录和home目录不能是父子关系)：
ORACLE_BASE=/u01/product/grid/crs
ORACLE_HOME=/u01/product/grid/11.2.0
mkdir -p /u01/product/oraInventory
chown -R grid:oinstall /u01/product/oraInventory
chmod -R 775 /u01/product/oraInventory/
mkdir -p /u01/product/grid/crs
mkdir -p /u01/product/grid/11.2.0
chown -R grid.oinstall /u01/product/grid/crs
chown -R grid.oinstall /u01/product/grid/11.2.0
chmod -R 775 /u01/product/grid/crs
chmod -R 775 /u01/product/grid/11.2.0
10\ 安装grid--检查os rpm包
rpm -q binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel gcc gcc-c++ libaio libaio-devel libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel
11\ 安装grid--设置环境变量
vi ~/.bash_profile
ORACLE_SID=+ASM3; export ORACLE_SID
ORACLE_BASE=/u01/product/grid/crs; export ORACLE_BASE
ORACLE_HOME=/u01/product/grid/11.2.0; export ORACLE_HOME
#export PATH=$ORACLE_HOME/bin:$PATH
JAVA_HOME=/usr/local/java; export JAVA_HOME
PATH=${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/sbin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/product/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
在另一个节点，注意修改ORACLE_SID;
12\ 安装grid--安装cvuqdisk RPM
在grid安装包目录下，有该rpm包，路径为 grid/rpm/cvuqdisk-1.0.9-1.rpm
rpm -ivh grid/rpm/cvuqdisk-1.0.9-1.rpm
13\ 安装grid--节点间用户等效性
新节点slot8：
su - grid
mkdir .ssh
chmod 700 .ssh
ssh-keygen -t dsa
ssh-keygen -t rsa
slot8节点：
cd ~/.ssh
ssh slot6 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot6 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
ssh slot7 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot7 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
ssh slot8 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot8 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
chmod 600 authorized_keys
scp authorized_keys slot7:/home/grid/.ssh/
scp authorized_keys slot6:/home/grid/.ssh/
原rac节点：
cd /home/grid/.ssh
chmod 600 authorized_keys
测试等效性：
ssh slot6 date
ssh slot7 date
ssh slot8 date
ssh slot6_pri date
ssh slot7_pri date
ssh slot8_pri date
14\ 安装grid--配置raw，使用udev管理【此步骤可选】
vi /etc/udev/rules.d/60-raw.rules 添加如下：
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="17", RUN+="/bin/raw /dev/raw/raw1 %M %m"
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="33", RUN+="/bin/raw /dev/raw/raw2 %M %m"
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="49", RUN+="/bin/raw /dev/raw/raw3 %M %m"
KERNEL=="raw[1-3]", WNER="grid", GROUP="asmadmin", MODE="660"
start_udev
[root@slot6 ~]# ll /dev/raw/raw*
crw-rw---- 1 grid asmadmin 162, 1 Dec 3 13:59 /dev/raw/raw1
crw-rw---- 1 grid asmadmin 162, 2 Dec 3 13:59 /dev/raw/raw2
crw-rw---- 1 grid asmadmin 162, 3 Dec 3 16:59 /dev/raw/raw3
15\ 安装grid--在rac的其中一个节点执行
slot7上操作以下步骤：
--验证新节点
cd /u01/product/grid/11.2.0/oui/bin/
./addNode.sh -pre "CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip}"
也可以在现有rac其中一个节点上执行
./runcluvfy.sh stage -pre crsinst -n slot6,slot7,slot8 -fixup -verbose
16\ 安装grid--在rac的其中一个节点执行
slot7上操作以下步骤：
./addNode.sh -- 在10g中，这样会调用ui安装界面，从而将新节点安装上软件，并加入rac中；
./addNode.sh -- 在11g中，这样会调用check，从而验证新节点与之前的rac是否符合安装条件，并不会出现安装界面，下面是11g中addNode.sh脚本的内容：
#!/bin/sh
OHOME=/u01/product/grid/11.2.0
INVPTRLOC=$OHOME/oraInst.loc
ADDNODE="$OHOME/oui/bin/runInstaller -addNode -invPtrLoc $INVPTRLOC ORACLE_HOME=$OHOME $*"
if [ "$IGNORE_PREADDNODE_CHECKS" = "Y" -o ! -f "$OHOME/cv/cvutl/check_nodeadd.pl" ]
then
$ADDNODE
else
CHECK_NODEADD="$OHOME/perl/bin/perl $OHOME/cv/cvutl/check_nodeadd.pl -pre $*"
$CHECK_NODEADD
if [ $? -eq 0 ]
then
$ADDNODE
fi
fi
从该脚本入手，可以分析出为什么执行addNode.sh脚本时总是出现check的界面了，在check全部通过后，oracle才会进行安装步骤，而安装11grac时，需要配置dns，验证才会全部通过，所以如果这样操作，永远也到不了安装步骤，于是可以手动跳过该步骤，直接执行相应的 runInstaller脚本；
./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
正常情况下，此时应该能出现正常安装了，如下log（如果执行此步骤之前，已经在slot8上安装了grid软件，该脚本会覆盖掉之前的安装）：
[grid@slot7 bin]$ ./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/product/grid/11.2.0
New Nodes
Space Requirements
New Nodes
slot8
/: Required 4.82GB : Available 34.19GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.3
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Server) 11.2.0.2.0
Installation Plugin Files 11.2.0.2.0
Universal Storage Manager Files 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Automatic Storage Management Assistant 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.2.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Oracle Net Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Enterprise Manager plugin Common Files 11.2.0.2.0
Platform. Required Support Files 11.2.0.2.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.2.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Cluster Verification Utility Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle LDAP administration 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
PL/SQL 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Cluster Ready Services Files 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Tuesday, December 11, 2012 11:19:42 AM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, December 11, 2012 11:19:45 AM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, December 11, 2012 11:21:27 AM CST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/product/grid/11.2.0/root.sh #On nodes slot8
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/product/grid/11.2.0 was successful.
Please check '/tmp/silentInstall.log' for more details.
可以看到，已经将rac上一个节点的grid软件copy到slot8这个新节点上，请别忘记，需要再slot8上执行root.sh脚本：
/u01/product/grid/11.2.0/root.sh #On nodes slot8
[root@slot8 ~]# /u01/product/grid/11.2.0/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/product/grid/11.2.0
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/product/grid/11.2.0/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node slot7, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
脚本执行成功，查看crs情况：
[grid@slot6 bin]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE slot6
ora....ER.lsnr ora....er.type ONLINE ONLINE slot6
ora....N1.lsnr ora....er.type ONLINE ONLINE slot7
ora.OCR.dg ora....up.type ONLINE ONLINE slot6
ora.asm ora.asm.type ONLINE ONLINE slot6
ora.cvu ora.cvu.type ONLINE ONLINE slot7
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE slot6
ora.oc4j ora.oc4j.type ONLINE ONLINE slot7
ora.ons ora.ons.type ONLINE ONLINE slot6
ora.racdb.db ora....se.type ONLINE ONLINE slot7
ora....ry.acfs ora....fs.type ONLINE ONLINE slot6
ora.scan1.vip ora....ip.type ONLINE ONLINE slot7
ora....SM2.asm application ONLINE ONLINE slot6
ora....T6.lsnr application ONLINE ONLINE slot6
ora.slot6.gsd application OFFLINE OFFLINE
ora.slot6.ons application ONLINE ONLINE slot6
ora.slot6.vip ora....t1.type ONLINE ONLINE slot6
ora....SM1.asm application ONLINE ONLINE slot7
ora....T7.lsnr application ONLINE ONLINE slot7
ora.slot7.gsd application OFFLINE OFFLINE
ora.slot7.ons application ONLINE ONLINE slot7
ora.slot7.vip ora....t1.type ONLINE ONLINE slot7
ora....SM3.asm application ONLINE ONLINE slot8
ora....T8.lsnr application ONLINE ONLINE slot8
ora.slot8.gsd application OFFLINE OFFLINE
ora.slot8.ons application ONLINE ONLINE slot8
ora.slot8.vip ora....t1.type ONLINE ONLINE slot8
已经在rac其中一个节点上，看到了slot8的rac信息了；
[grid@slot6 bin]$ olsnodes
slot7
slot6
slot8
显示节点情况；
以上已经说明slot8的grid已经安装配置成功了；
在这个安装过程中，还遇到了另一个问题：
[grid@slot7 bin]$ ./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8,slot8 are available
............................................................... 100% Done.
Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
SEVERE:Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
后来查找原因就是因为在执行以上脚本之前，在rac节点执行过：
./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={slot6,slot7,slot8}"
这样一来，就误导了cluster以为slot8已经在rac中了，后执行：
./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={slot6,slot7}"
将slot8从cluster中清除掉，这样再执行以上的正常安装步骤就没有问题了。
17\ 安装rdbms--创建安装目录
mkdir -p /u01/product/oracle
mkdir -p /u01/product/oracle/11.2.0/db_1
chown -R oracle:oinstall /u01/product/oracle/
chmod -R 775 /u01/product/oracle/
18\ 安装rdbms--设置oracle环境变量
JAVA_HOME=/usr/local/java; export JAVA_HOME
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
ORACLE_SID=racdb3 ; export ORACLE_SID
ORACLE_BASE=/u01/product/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/11.2.0/db_1; export ORACLE_HOME
export PATH=$ORACLE_HOME/bin:$PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib;export LD_LIBRARY_PATH
PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin
PATH=${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/product/common/oracle/bin
export PATH
alias ll='ls -l';
alias ls='ls --color';
alias his='history';
umask 022
19\ 安装rdbms--节点间用户等效性
新节点slot8：
su - grid
mkdir .ssh
chmod 700 .ssh
ssh-keygen -t dsa
ssh-keygen -t rsa
slot8节点：
cd ~/.ssh
ssh slot6 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot6 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
ssh slot7 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot7 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
ssh slot8 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot8 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
chmod 600 authorized_keys
scp authorized_keys slot7:/home/oracle/.ssh/
scp authorized_keys slot6:/home/oracle/.ssh/
原rac节点：
cd /home/oracle/.ssh
chmod 600 authorized_keys
测试等效性：
ssh slot6 date
ssh slot7 date
ssh slot8 date
ssh slot6_pri date
ssh slot7_pri date
ssh slot8_pri date
20\ 安装rdbms--安装软件
./addNode.sh CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip}
WARNING:
Node "slot8" already appears to be part of cluster
[1793@slot7] [main] [ 2012-12-11 15:44:30.483 CST ] [NodeAddStage.setTaskList:457]
Node (slot8) already in cluster.
Cluster nodes: [slot7, slot6, slot8]
[1793@slot7] [main] [ 2012-12-11 15:44:30.483 CST ] [NodeAddStage.setTaskList:481] NO WORK!!
Parameters:
nodeList: slot8
validNodeList: slot8
newClusterNodes: slot7,slot6,slot8
validClusterNodes: slot6,slot7,slot8
validNewClusterNodes: slot6,slot7,slot8
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [ResultSet.traceResultSet:342]
*** results BEFORE node purge...
Overall Status->SUCCESSFUL
slot8-->SUCCESSFUL
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [ResultSet.traceResultSet:342]
*** results AFTER node purge...
Overall Status->SUCCESSFUL
slot8-->SUCCESSFUL
Pre-check for node addition was successful. -------> 此处因为成功了，所以能继续执行安装；
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [CluvfyDriver.main:325] ==== cluvfy exiting normally.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
Checking monitor: must be configured to display at least 256 colors
>>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set. Failed <<<<
Some requirement checks failed. You must fulfill these requirements before
continuing with the installation,
Continue? (y/n) [n] Y
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/product/oracle/11.2.0/db_1
New Nodes
Space Requirements
New Nodes
slot8
/: Required 3.95GB : Available 33.45GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.3.1.1
Oracle ODBC Driverfor Instant Client 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
SSL Required Support Files for InstantClient 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.2.0
Oracle Database Vault J2EE Application 11.2.0.2.0
Oracle Label Security 11.2.0.2.0
Oracle Data Mining RDBMS Files 11.2.0.2.0
Oracle OLAP RDBMS Files 11.2.0.2.0
Oracle OLAP API 11.2.0.2.0
Platform. Required Support Files 11.2.0.2.0
Oracle Database Vault option 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.2.0
Oracle SQL Developer 11.2.0.2.0
Oracle Application Express 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
SQLJ Runtime 11.2.0.2.0
Database Workspace Manager 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
Enterprise Manager Repository Core Files 10.2.0.4.3
Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
Enterprise Manager Grid Control Core Files 10.2.0.4.3
Enterprise Manager Common Core Files 10.2.0.4.3
Enterprise Manager Agent Core Files 10.2.0.4.3
RDBMS Required Support Files 11.2.0.2.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Parser Generator Required Support Files 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Multimedia Annotator 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Sample Schema Data 11.2.0.2.0
Oracle Starter Database 11.2.0.2.0
Oracle Message Gateway Common Files 11.2.0.2.0
Oracle XML Query 11.2.0.2.0
XML Parser for Oracle JVM 11.2.0.2.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.2.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.2.0
Oracle Containers for Java 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.2.0
Secure Socket Layer 11.2.0.2.0
Oracle Universal Connection Pool 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Precompiler Common Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
PL/SQL 11.2.0.2.0
Generic Connectivity Common Files 11.2.0.2.0
Oracle Database Gateway for ODBC 11.2.0.2.0
Oracle Programmer 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Call Interface (OCI) 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle XML Development Kit 11.2.0.2.0
Database Configuration and Upgrade Assistants 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Advanced Security 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Enterprise Manager Console DB 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Oracle Text 11.2.0.2.0
Oracle Net Services 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
Oracle OLAP 11.2.0.2.0
Oracle Spatial 11.2.0.2.0
Oracle Partitioning 11.2.0.2.0
Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Tuesday, December 11, 2012 3:47:29 PM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, December 11, 2012 3:47:33 PM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, December 11, 2012 3:50:42 PM CST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/product/oracle/11.2.0/db_1/root.sh #On nodes slot8
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/product/oracle/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[root@slot8 ~]# /u01/product/oracle/11.2.0/db_1/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/product/oracle/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
21\ 添加instance--dbca
在slot8节点上（rac其中一个节点也行）：
以下是选择步骤：
step1:To add new instances, launch DBCA from a pre-existing node. Example:
DISPLAY=ipaddress:0.0; export DISPLAY
dbca
step2:On the welcome screen, choose "Oracle Real Application Clusters",click next.
step3:Choose "Instance Management", click next.
step4:Choose "Add an Instance", click next.
step5:Choose the database you would like to add an instance to and specify a user with SYSDBA privileges, click next. Click next again...
step6:Choose the correct instance name and node, click next.
step7:Review the storage screen, click next.
step8:Review the summary screen, click OK and wait a few seconds for the progress bar to start.
step9:Allow the progress bar to finish. When asked if you want to perform. another operation, choose "No" to exit DBCA.
step10:To verify success, log into one of the instances and query from gv$instance, you should now see all nodes.
全部完成，以下命令验证：
[grid@slot6 ~]$ srvctl status instance -d racdb -i racdb1,racdb2,racdb3 -v
Instance racdb1 is running on node slot7. Instance status: Open.
Instance racdb2 is running on node slot6. Instance status: Open.
Instance racdb3 is running on node slot8. Instance status: Open.
[grid@slot6 bin]$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE slot6
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE slot6
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE slot7
ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE slot6
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE slot6
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE slot6
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE slot6
ora.oc4j ora.oc4j.type 0/5 0/0 ONLINE ONLINE slot8
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE slot6
ora.racdb.db ora....se.type 0/2 0/1 ONLINE ONLINE slot7
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE slot6
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE slot7
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE slot6
ora....T6.lsnr application 0/5 0/0 ONLINE ONLINE slot6
ora.slot6.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot6.ons application 0/3 0/0 ONLINE ONLINE slot6
ora.slot6.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot6
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE slot7
ora....T7.lsnr application 0/5 0/0 ONLINE ONLINE slot7
ora.slot7.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot7.ons application 0/3 0/0 ONLINE ONLINE slot7
ora.slot7.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot7
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE slot8
ora....T8.lsnr application 0/5 0/0 ONLINE ONLINE slot8
ora.slot8.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot8.ons application 0/3 0/0 ONLINE ONLINE slot8
ora.slot8.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot8
之前的部署环境
（RHEL5.8）：
iscsi服务器端 : 192.168.8.214
RAC端：
192.168.8.216 slot7
192.168.8.215 slot6
10.0.0.20 slot7_pri
10.0.0.10 slot6_pri
192.168.8.219 slot7_vip
192.168.8.218 slot6_vip
192.168.8.220 scanip
添加新节点后的部署规划
iscsi服务器端 : 192.168.8.214
RAC端：
192.168.8.216 slot7
192.168.8.215 slot6
192.168.8.217 slot8
10.0.0.20 slot7_pri
10.0.0.10 slot6_pri
10.0.0.30 slot8_pri
192.168.8.219 slot7_vip
192.168.8.218 slot6_vip
192.168.8.221 slot8_vip
192.168.8.220 scanip
1\ iscsi客户端（在rhel5.8中已经默认安装了iscsi服务）
[root@slot6 ~]# chkconfig --list|grep iscsi
iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
chkconfig iscsi on
/etc/init.d/iscsi start
[root@slot6 ~]# iscsiadm -m discovery -p 192.168.8.214 -t sendtargets
192.168.8.214:3260,1 iqn.2001-04.com.example:storage.disk2.sys1.xyz
[root@slot6 ~]# iscsiadm --mode node --targetname iqn.2001-04.com.example:storage.disk2.sys1.xyz --portal 192.168.8.214:3260 --login
3\ 安装grid--创建用户并设置密码
Grid Infrastructure 操作系统用户grid , 主组为oinstall， 辅助组为asmadmin, asmdba, asmoper
Oracle RAC 操作系统用户 oracle , 主组为oinstall , 辅助组为dba, oper , asmdba
groupadd -g 501 oinstall
groupadd -g 502 asmadmin
groupadd -g 503 asmdba
groupadd -g 504 asmoper
groupadd -g 505 dba
groupadd -g 506 oper
useradd -u 500 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
useradd -u 501 -g oinstall -G asmdba,dba,oper oracle
passwd grid
passwd oracle
4\ 安装grid--网络设置
eth0 public
eth1 private
具体ip规划见开始环境部署说明。
5\ 安装grid--各个节点时间同步
iscsi服务端作为ntp服务器；
ntp服务端：
[root@slot5 iscsi]# chkconfig --list |grep ntp
ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
chkconfig ntpd on
/etc/init.d/ntpd start
[root@slot5 iscsi]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*LOCAL(0) .LOCL. 10 l 38 64 377 0.000 0.000 0.001
ntp客户端（rac节点）：
chkconfig ntpd on
vi /etc/ntp.conf 将server 127.0.0.1 修改为：
server 192.168.8.214 # local clock
/etc/init.d/ntpd start
[root@slot6 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*192.168.8.214 LOCAL(0) 11 u 17 64 377 0.105 -0.322 0.004
过段时间节点间时间就会同步了。
另外，为防止在检测时报ntp时间同步错误：
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
– Cause: NTP daemon on one or more nodes lacked slewing option.
– Action: Shut down and restart the NTP daemon after setting the slewing option as follows:
For Linux, edit /etc/sysconfig/ntpd and add -x to the command line option.
For SUSE Linux, edit /etc/sysconfig/ntp and add -x to the OPTIONS variable.
For AIX, edit /etc/rc.tcpip and add -x to the command line option.
For HP-UX, edit /etc/rc.config.d/netdaemons and add -x to the command line option.
For Solaris, edit /etc/inet/ntp.conf and add ‘slewalways yes’ and ‘disable pll’ in ntp.conf.
需要 vi /etc/sysconfig/ntpd 将OPTIONS后添加 -x ，如下：
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
然后重启ntp服务；
/etc/init.d/ntpd restart
6\ 安装grid--配置内核参数
kernel.shmall=2097152
#kernel.shmmax=1073741824
kernel.shmmni=4096
kernel.sem=250 32000 100 128
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048576
fs.file-max = 6815744
fs.aio-max-nr = 1048576
然后执行 sysctl -p
7\ 安装grid--修改limit
vi /etc/security/limits.conf 添加
grid soft nproc 2047
grid hard nproc 32768
grid soft nofile 1024
grid hard nofile 250000
oracle soft nproc 2047
oracle hard nproc 32768
oracle soft nofile 1024
oracle hard nofile 250000
8\ 安装grid--修改/etc/pam.d/login
vi /etc/pam.d/login 添加
session required pam_limits.so
9\ 安装grid--创建目录
规划(grid的base目录和home目录不能是父子关系)：
ORACLE_BASE=/u01/product/grid/crs
ORACLE_HOME=/u01/product/grid/11.2.0
mkdir -p /u01/product/oraInventory
chown -R grid:oinstall /u01/product/oraInventory
chmod -R 775 /u01/product/oraInventory/
mkdir -p /u01/product/grid/crs
mkdir -p /u01/product/grid/11.2.0
chown -R grid.oinstall /u01/product/grid/crs
chown -R grid.oinstall /u01/product/grid/11.2.0
chmod -R 775 /u01/product/grid/crs
chmod -R 775 /u01/product/grid/11.2.0
10\ 安装grid--检查os rpm包
rpm -q binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel gcc gcc-c++ libaio libaio-devel libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel
11\ 安装grid--设置环境变量
vi ~/.bash_profile
ORACLE_SID=+ASM3; export ORACLE_SID
ORACLE_BASE=/u01/product/grid/crs; export ORACLE_BASE
ORACLE_HOME=/u01/product/grid/11.2.0; export ORACLE_HOME
#export PATH=$ORACLE_HOME/bin:$PATH
JAVA_HOME=/usr/local/java; export JAVA_HOME
PATH=${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/sbin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/product/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
在另一个节点，注意修改ORACLE_SID;
12\ 安装grid--安装cvuqdisk RPM
在grid安装包目录下，有该rpm包，路径为 grid/rpm/cvuqdisk-1.0.9-1.rpm
rpm -ivh grid/rpm/cvuqdisk-1.0.9-1.rpm
13\ 安装grid--节点间用户等效性
新节点slot8：
su - grid
mkdir .ssh
chmod 700 .ssh
ssh-keygen -t dsa
ssh-keygen -t rsa
slot8节点：
cd ~/.ssh
ssh slot6 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot6 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
ssh slot7 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot7 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
ssh slot8 cat /home/grid/.ssh/id_rsa.pub >>authorized_keys
ssh slot8 cat /home/grid/.ssh/id_dsa.pub >>authorized_keys
chmod 600 authorized_keys
scp authorized_keys slot7:/home/grid/.ssh/
scp authorized_keys slot6:/home/grid/.ssh/
原rac节点：
cd /home/grid/.ssh
chmod 600 authorized_keys
测试等效性：
ssh slot6 date
ssh slot7 date
ssh slot8 date
ssh slot6_pri date
ssh slot7_pri date
ssh slot8_pri date
14\ 安装grid--配置raw，使用udev管理【此步骤可选】
vi /etc/udev/rules.d/60-raw.rules 添加如下：
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="17", RUN+="/bin/raw /dev/raw/raw1 %M %m"
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="33", RUN+="/bin/raw /dev/raw/raw2 %M %m"
ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="49", RUN+="/bin/raw /dev/raw/raw3 %M %m"
KERNEL=="raw[1-3]", WNER="grid", GROUP="asmadmin", MODE="660"
start_udev
[root@slot6 ~]# ll /dev/raw/raw*
crw-rw---- 1 grid asmadmin 162, 1 Dec 3 13:59 /dev/raw/raw1
crw-rw---- 1 grid asmadmin 162, 2 Dec 3 13:59 /dev/raw/raw2
crw-rw---- 1 grid asmadmin 162, 3 Dec 3 16:59 /dev/raw/raw3
15\ 安装grid--在rac的其中一个节点执行
slot7上操作以下步骤：
--验证新节点
cd /u01/product/grid/11.2.0/oui/bin/
./addNode.sh -pre "CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip}"
也可以在现有rac其中一个节点上执行
./runcluvfy.sh stage -pre crsinst -n slot6,slot7,slot8 -fixup -verbose
16\ 安装grid--在rac的其中一个节点执行
slot7上操作以下步骤：
./addNode.sh -- 在10g中，这样会调用ui安装界面，从而将新节点安装上软件，并加入rac中；
./addNode.sh -- 在11g中，这样会调用check，从而验证新节点与之前的rac是否符合安装条件，并不会出现安装界面，下面是11g中addNode.sh脚本的内容：
#!/bin/sh
OHOME=/u01/product/grid/11.2.0
INVPTRLOC=$OHOME/oraInst.loc
ADDNODE="$OHOME/oui/bin/runInstaller -addNode -invPtrLoc $INVPTRLOC ORACLE_HOME=$OHOME $*"
if [ "$IGNORE_PREADDNODE_CHECKS" = "Y" -o ! -f "$OHOME/cv/cvutl/check_nodeadd.pl" ]
then
$ADDNODE
else
CHECK_NODEADD="$OHOME/perl/bin/perl $OHOME/cv/cvutl/check_nodeadd.pl -pre $*"
$CHECK_NODEADD
if [ $? -eq 0 ]
then
$ADDNODE
fi
fi
从该脚本入手，可以分析出为什么执行addNode.sh脚本时总是出现check的界面了，在check全部通过后，oracle才会进行安装步骤，而安装11grac时，需要配置dns，验证才会全部通过，所以如果这样操作，永远也到不了安装步骤，于是可以手动跳过该步骤，直接执行相应的 runInstaller脚本；
./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
正常情况下，此时应该能出现正常安装了，如下log（如果执行此步骤之前，已经在slot8上安装了grid软件，该脚本会覆盖掉之前的安装）：
[grid@slot7 bin]$ ./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/product/grid/11.2.0
New Nodes
Space Requirements
New Nodes
slot8
/: Required 4.82GB : Available 34.19GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.3
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Server) 11.2.0.2.0
Installation Plugin Files 11.2.0.2.0
Universal Storage Manager Files 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Automatic Storage Management Assistant 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.2.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Oracle Net Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Enterprise Manager plugin Common Files 11.2.0.2.0
Platform. Required Support Files 11.2.0.2.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.2.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Cluster Verification Utility Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle LDAP administration 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
PL/SQL 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Cluster Ready Services Files 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Tuesday, December 11, 2012 11:19:42 AM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, December 11, 2012 11:19:45 AM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, December 11, 2012 11:21:27 AM CST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/product/grid/11.2.0/root.sh #On nodes slot8
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/product/grid/11.2.0 was successful.
Please check '/tmp/silentInstall.log' for more details.
可以看到，已经将rac上一个节点的grid软件copy到slot8这个新节点上，请别忘记，需要再slot8上执行root.sh脚本：
/u01/product/grid/11.2.0/root.sh #On nodes slot8
[root@slot8 ~]# /u01/product/grid/11.2.0/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/product/grid/11.2.0
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/product/grid/11.2.0/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node slot7, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
脚本执行成功，查看crs情况：
[grid@slot6 bin]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE slot6
ora....ER.lsnr ora....er.type ONLINE ONLINE slot6
ora....N1.lsnr ora....er.type ONLINE ONLINE slot7
ora.OCR.dg ora....up.type ONLINE ONLINE slot6
ora.asm ora.asm.type ONLINE ONLINE slot6
ora.cvu ora.cvu.type ONLINE ONLINE slot7
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE slot6
ora.oc4j ora.oc4j.type ONLINE ONLINE slot7
ora.ons ora.ons.type ONLINE ONLINE slot6
ora.racdb.db ora....se.type ONLINE ONLINE slot7
ora....ry.acfs ora....fs.type ONLINE ONLINE slot6
ora.scan1.vip ora....ip.type ONLINE ONLINE slot7
ora....SM2.asm application ONLINE ONLINE slot6
ora....T6.lsnr application ONLINE ONLINE slot6
ora.slot6.gsd application OFFLINE OFFLINE
ora.slot6.ons application ONLINE ONLINE slot6
ora.slot6.vip ora....t1.type ONLINE ONLINE slot6
ora....SM1.asm application ONLINE ONLINE slot7
ora....T7.lsnr application ONLINE ONLINE slot7
ora.slot7.gsd application OFFLINE OFFLINE
ora.slot7.ons application ONLINE ONLINE slot7
ora.slot7.vip ora....t1.type ONLINE ONLINE slot7
ora....SM3.asm application ONLINE ONLINE slot8
ora....T8.lsnr application ONLINE ONLINE slot8
ora.slot8.gsd application OFFLINE OFFLINE
ora.slot8.ons application ONLINE ONLINE slot8
ora.slot8.vip ora....t1.type ONLINE ONLINE slot8
已经在rac其中一个节点上，看到了slot8的rac信息了；
[grid@slot6 bin]$ olsnodes
slot7
slot6
slot8
显示节点情况；
以上已经说明slot8的grid已经安装配置成功了；
在这个安装过程中，还遇到了另一个问题：
[grid@slot7 bin]$ ./runInstaller -addNode -invPtrLoc /u01/product/grid/11.2.0/oraInst.loc ORACLE_HOME=$ORACLE_HOME CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip} CLUSTER_NODES={slot6,slot7,slot8}
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8,slot8 are available
............................................................... 100% Done.
Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
SEVERE:Error ocurred while retrieving node numbers of the existing nodes. Please check if clusterware home is properly configured.
后来查找原因就是因为在执行以上脚本之前，在rac节点执行过：
./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={slot6,slot7,slot8}"
这样一来，就误导了cluster以为slot8已经在rac中了，后执行：
./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={slot6,slot7}"
将slot8从cluster中清除掉，这样再执行以上的正常安装步骤就没有问题了。
17\ 安装rdbms--创建安装目录
mkdir -p /u01/product/oracle
mkdir -p /u01/product/oracle/11.2.0/db_1
chown -R oracle:oinstall /u01/product/oracle/
chmod -R 775 /u01/product/oracle/
18\ 安装rdbms--设置oracle环境变量
JAVA_HOME=/usr/local/java; export JAVA_HOME
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export CLASSPATH
export TEMP=/tmp
export TMPDIR=/tmp
ORACLE_SID=racdb3 ; export ORACLE_SID
ORACLE_BASE=/u01/product/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/11.2.0/db_1; export ORACLE_HOME
export PATH=$ORACLE_HOME/bin:$PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib;export LD_LIBRARY_PATH
PATH=$PATH:$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin
PATH=${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=${PATH}:/u01/product/common/oracle/bin
export PATH
alias ll='ls -l';
alias ls='ls --color';
alias his='history';
umask 022
19\ 安装rdbms--节点间用户等效性
新节点slot8：
su - grid
mkdir .ssh
chmod 700 .ssh
ssh-keygen -t dsa
ssh-keygen -t rsa
slot8节点：
cd ~/.ssh
ssh slot6 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot6 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
ssh slot7 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot7 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
ssh slot8 cat /home/oracle/.ssh/id_rsa.pub >>authorized_keys
ssh slot8 cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
chmod 600 authorized_keys
scp authorized_keys slot7:/home/oracle/.ssh/
scp authorized_keys slot6:/home/oracle/.ssh/
原rac节点：
cd /home/oracle/.ssh
chmod 600 authorized_keys
测试等效性：
ssh slot6 date
ssh slot7 date
ssh slot8 date
ssh slot6_pri date
ssh slot7_pri date
ssh slot8_pri date
20\ 安装rdbms--安装软件
./addNode.sh CLUSTER_NEW_NODES={slot8} CLUSTER_NEW_VIRTUAL_HOSTNAMES={slot8_vip}
WARNING:
Node "slot8" already appears to be part of cluster
[1793@slot7] [main] [ 2012-12-11 15:44:30.483 CST ] [NodeAddStage.setTaskList:457]
Node (slot8) already in cluster.
Cluster nodes: [slot7, slot6, slot8]
[1793@slot7] [main] [ 2012-12-11 15:44:30.483 CST ] [NodeAddStage.setTaskList:481] NO WORK!!
Parameters:
nodeList: slot8
validNodeList: slot8
newClusterNodes: slot7,slot6,slot8
validClusterNodes: slot6,slot7,slot8
validNewClusterNodes: slot6,slot7,slot8
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [ResultSet.traceResultSet:342]
*** results BEFORE node purge...
Overall Status->SUCCESSFUL
slot8-->SUCCESSFUL
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [ResultSet.traceResultSet:342]
*** results AFTER node purge...
Overall Status->SUCCESSFUL
slot8-->SUCCESSFUL
Pre-check for node addition was successful. -------> 此处因为成功了，所以能继续执行安装；
[1793@slot7] [main] [ 2012-12-11 15:44:30.484 CST ] [CluvfyDriver.main:325] ==== cluvfy exiting normally.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 31267 MB Passed
Checking monitor: must be configured to display at least 256 colors
>>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set. Failed <<<<
Some requirement checks failed. You must fulfill these requirements before
continuing with the installation,
Continue? (y/n) [n] Y
>>> Ignoring required pre-requisite failures. Continuing...
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
Performing tests to see whether nodes slot6,slot8 are available
............................................................... 100% Done.
.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/product/oracle/11.2.0/db_1
New Nodes
Space Requirements
New Nodes
slot8
/: Required 3.95GB : Available 33.45GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.3.1.1
Oracle ODBC Driverfor Instant Client 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
SSL Required Support Files for InstantClient 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.2.0
Oracle Database Vault J2EE Application 11.2.0.2.0
Oracle Label Security 11.2.0.2.0
Oracle Data Mining RDBMS Files 11.2.0.2.0
Oracle OLAP RDBMS Files 11.2.0.2.0
Oracle OLAP API 11.2.0.2.0
Platform. Required Support Files 11.2.0.2.0
Oracle Database Vault option 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.2.0
Oracle SQL Developer 11.2.0.2.0
Oracle Application Express 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
SQLJ Runtime 11.2.0.2.0
Database Workspace Manager 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
Enterprise Manager Repository Core Files 10.2.0.4.3
Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
Enterprise Manager Grid Control Core Files 10.2.0.4.3
Enterprise Manager Common Core Files 10.2.0.4.3
Enterprise Manager Agent Core Files 10.2.0.4.3
RDBMS Required Support Files 11.2.0.2.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Parser Generator Required Support Files 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Multimedia Annotator 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Sample Schema Data 11.2.0.2.0
Oracle Starter Database 11.2.0.2.0
Oracle Message Gateway Common Files 11.2.0.2.0
Oracle XML Query 11.2.0.2.0
XML Parser for Oracle JVM 11.2.0.2.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.2.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.2.0
Oracle Containers for Java 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.2.0
Secure Socket Layer 11.2.0.2.0
Oracle Universal Connection Pool 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Precompiler Common Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
PL/SQL 11.2.0.2.0
Generic Connectivity Common Files 11.2.0.2.0
Oracle Database Gateway for ODBC 11.2.0.2.0
Oracle Programmer 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Call Interface (OCI) 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle XML Development Kit 11.2.0.2.0
Database Configuration and Upgrade Assistants 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Advanced Security 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Enterprise Manager Console DB 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Oracle Text 11.2.0.2.0
Oracle Net Services 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
Oracle OLAP 11.2.0.2.0
Oracle Spatial 11.2.0.2.0
Oracle Partitioning 11.2.0.2.0
Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------
Instantiating scripts for add node (Tuesday, December 11, 2012 3:47:29 PM CST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, December 11, 2012 3:47:33 PM CST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, December 11, 2012 3:50:42 PM CST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/product/oracle/11.2.0/db_1/root.sh #On nodes slot8
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The Cluster Node Addition of /u01/product/oracle/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
[root@slot8 ~]# /u01/product/oracle/11.2.0/db_1/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/product/oracle/11.2.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
21\ 添加instance--dbca
在slot8节点上（rac其中一个节点也行）：
以下是选择步骤：
step1:To add new instances, launch DBCA from a pre-existing node. Example:
DISPLAY=ipaddress:0.0; export DISPLAY
dbca
step2:On the welcome screen, choose "Oracle Real Application Clusters",click next.
step3:Choose "Instance Management", click next.
step4:Choose "Add an Instance", click next.
step5:Choose the database you would like to add an instance to and specify a user with SYSDBA privileges, click next. Click next again...
step6:Choose the correct instance name and node, click next.
step7:Review the storage screen, click next.
step8:Review the summary screen, click OK and wait a few seconds for the progress bar to start.
step9:Allow the progress bar to finish. When asked if you want to perform. another operation, choose "No" to exit DBCA.
step10:To verify success, log into one of the instances and query from gv$instance, you should now see all nodes.
全部完成，以下命令验证：
[grid@slot6 ~]$ srvctl status instance -d racdb -i racdb1,racdb2,racdb3 -v
Instance racdb1 is running on node slot7. Instance status: Open.
Instance racdb2 is running on node slot6. Instance status: Open.
Instance racdb3 is running on node slot8. Instance status: Open.
[grid@slot6 bin]$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE slot6
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE slot6
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE slot7
ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE slot6
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE slot6
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE slot6
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE slot6
ora.oc4j ora.oc4j.type 0/5 0/0 ONLINE ONLINE slot8
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE slot6
ora.racdb.db ora....se.type 0/2 0/1 ONLINE ONLINE slot7
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE slot6
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE slot7
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE slot6
ora....T6.lsnr application 0/5 0/0 ONLINE ONLINE slot6
ora.slot6.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot6.ons application 0/3 0/0 ONLINE ONLINE slot6
ora.slot6.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot6
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE slot7
ora....T7.lsnr application 0/5 0/0 ONLINE ONLINE slot7
ora.slot7.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot7.ons application 0/3 0/0 ONLINE ONLINE slot7
ora.slot7.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot7
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE slot8
ora....T8.lsnr application 0/5 0/0 ONLINE ONLINE slot8
ora.slot8.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.slot8.ons application 0/3 0/0 ONLINE ONLINE slot8
ora.slot8.vip ora....t1.type 0/0 0/0 ONLINE ONLINE slot8
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/10037372/viewspace-750952/ ，如需转载，请注明出处，否则将追究法律责任。
请登录后发表评论
登录
全部评论
