花近有10天的时间做了10G的RAC。由于OCFS2和ASM都熟悉时间长的点，不要笑我呀。

OS：Linux node1pub 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux

ORACLE：10.2.0.1

存储的布局：ocfs2 for OCR, CRS voting, ASM for data.

VMWARE:Server 1.0.5 build-80187

实验过程中严重参考：

http://blog.chinaunix.net/u/31/showart_518611.html

http://tolywang.itpub.net/post/48/454440

vmware-vdiskmanager 创建 pre-allocated并且是lsi contoller的硬盘 1GB一个 for ocfs2 for OCR +

CRS voting
vmware-vdiskmanager 创建 pre-allocated并且是lsi contoller的硬盘 1GB 四个for ASM for Oracle

data & flash recovery area (同上)

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:ocfs.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:ocfs.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm1.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm1.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm2.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm2.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>
C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm3.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm3.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm4.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm4.vmdk'
Create: 100% done.
Virtual disk creation successful.

分别在两个虚拟机的Red Hat Enterprise Linux 4.vmx文件里增加以下内容
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"

scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.filename = "D:Virtual Machinesocfs.vmdk"
scsi1:1.deviceType = "plainDisk"

scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.filename = "D:Virtual Machinesasm1.vmdk"
scsi1:2.deviceType = "plainDisk"

scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.filename = "D:Virtual Machinesasm2.vmdk"
scsi1:3.deviceType = "plainDisk"

scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.filename = "D:Virtual Machinesasm3.vmdk"
scsi1:4.deviceType = "plainDisk"

scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.filename = "D:Virtual Machinesasm4.vmdk"
scsi1:5.deviceType = "plainDisk"

disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.DataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"

2.配置两台机的IP及HOSTS
[root@node1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost

172.17.61.130 node1pub
172.17.61.131 node2pub

10.1.1.1 node1prv
10.1.1.2 node2prv

172.17.61.140 node1vip
172.17.61.141 node2vip

3.建立组和用户（两个node)
[root@node2 .ssh]# groupadd oinstall
[root@node2 .ssh]# groupadd dba
[root@node2 .ssh]# useradd oracle -g oinstall -G dba
[root@node2 .ssh]# mkdir -p /u01/app/oracle
[root@node2 .ssh]# chown -R oracle /u01/app/oracle
[root@node2 .ssh]# chmod -R 775 /u01/app/oracle
[root@node2 .ssh]# passwd oracle
这个目录给 ocfs2用来装OCR, CRS voting 的

4.配置SSH（rsa/dsa都要配）
而且必须SSH通过自己（注意是ORACLE用户）
测试一下：
ssh node1pub date
ssh node2pub date
ssh node1prv date
ssh node2prv date

5.修改/etc/sysctl.conf,添加参数
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

6.修改/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

7.修改 pam.d login
vi /etc/pam.d/login
最后添加 session required /lib/security/pam_limits.so

8. 增加 hancheck-timer 模块在2个node上

先看一下有没有 find /lib/modules -name "hangcheck-timer.ko"

然后加上去 echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >>

/etc/modprobe.conf
然后把这个模块调起来 modprobe hangcheck-timer
然后检查是否工作正常了 grep Hangcheck /var/log/messages | tail -2
看到类似这个的信息就表明工作正常了 Jan 18 01:08:19 n1pub kernel: Hangcheck: starting hangcheck

timer 0.5.0 (tick is 30 seconds, margin is 180 seconds).

9.安装ocfs2
[root@node1 rac package]# rpm -ivh ocfs2-tools-1.2.7-1.el4.i386.rpm
Preparing... ########################################### [100%]
1:ocfs2-tools ########################################### [100%]
[root@node1 rac package]# rpm -ivh ocfs2-2.6.9-42.ELsmp-1.2.8-2.el4.i686.rpm
Preparing... ########################################### [100%]
1:ocfs2-2.6.9-42.ELsmp ########################################### [100%]
[root@node1 rac package]# rpm -ivh ocfs2console-1.2.7-1.el4.i386.rpm
Preparing... ########################################### [100%]
1:ocfs2console ########################################### [100%]

10.进入X, 然后运行 ocfs2console, 把你的2个node都添加进去（两个节点）
[root@node1pub ocfs2]# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 172.17.61.130
number = 0
name = node1pub
cluster = ocfs2

node:
ip_port = 7777
ip_address = 172.17.61.131
number = 1
name = node2pub
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2

11.开启ocfs2(两个节点)
[root@node1pub ocfs2]# /etc/init.d/o2cb enable
Writing O2CB configuration: OK
O2CB cluster ocfs2 already online
[root@node1pub ocfs2]# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
O2CB cluster ocfs2 already online
[root@node1pub ocfs2]# /etc/init.d/o2cb stop
Stopping O2CB cluster ocfs2: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
[root@node1pub ocfs2]# /etc/init.d/o2cb start
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

12.格式化ocfs 磁盘（一个节点）
[root@node1pub init.d]# mkfs.ocfs2 -b 4k -C 32k -L oradatafiles /dev/sdb
mkfs.ocfs2 1.2.7
Filesystem label=oradatafiles
Block size=4096 (bits=12)
Cluster size=32768 (bits=15)
Volume size=1073741824 (32768 clusters) (262144 blocks)
2 cluster groups (tail covers 512 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

13.mount ocfs2 (两个节点)
mkdir -p /u02/oradata/orcl
mount -t ocfs2 -o datavolume /dev/sdb /u02/oradata/orcl
修改你的 /etc/fstab , 添加 类似这样的行
/dev/sdb /u02/oradata/orcl ocfs2 _netdev,datavolume 0 0
chown -R oracle.oinstall /u02/oradata/orcl/
chmod -R 775 /u02/oradata/orcl/
到这里，我们的ocfs2 for OCR, CRS voting 就OK了

14. 修改 /etc/sysconfig/o2cb(两个节点)
把threshhold 的 值设置成 601
O2CB_HEARTBEAT_THRESHOLD=601

15.安装ASM包(两个节点)
[root@node1pub rac package]# rpm -ivh oracleasm-support-2.0.3-1.i386.rpm
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]
[root@node1pub rac package]# rpm -ivh oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm
Preparing... ########################################### [100%]
1:oracleasm-2.6.9-42.ELsm############################################### [100%]

[root@node1pub rac package]# rpm -ivh oracleasmlib-2.0.2-1.i386.rpm
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]

16.然后运行 /etc/init.d/oracleasm configure(两个节点)

回答 oracle , dba, y, y 就可以了

[root@node1pub rac package]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Creating /dev/oracleasm mount point: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
[root@node1pub rac package]#

17.创建ASM
创建ASM之前，需要将几个磁盘fdisk成分区，否则ASM不认。
[root@node1pub ~]# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):
Using default value 130

Command (m for help): w
The partition table has been altered!

[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL2 /dev/sdd1
Marking disk "/dev/sdd1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL3 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL4 /dev/sdf1
Marking disk "/dev/sdf1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

在另外一个节点上
/etc/init.d/oracleasm scandisks
[root@node2pub ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@node2pub ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

18.修改 oracle用户目录下的 .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/crs
export ORACLE_SID=orcl1

export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp

第二个节点的 ORACLE_SID=orcl2 其他都一样

19.安装CRS
1）用oracle 用户login , 然后运行你的unzip 之后的那个clusterware目录下的runInsaller 进入图形安

装
2）确认你的安装目录是/u01/app/oracle/product/crs
3）修改 你的clustername 从crs到一个随便你想要的名字
然后增加
node1pub node1prv node1vip
node2pub node2prv node2vip
4）然后指定 eth0 的类型时public
5）然后制定你的OCR 和mirror
/u02/oradata/orcl/OCRFile
/u02/oradata/orcl/OCRFile_mirror
6）然后指定你的voting
/u02/oradata/orcl/VotingFile
/u02/oradata/orcl/VotingFile_mirror1
/u02/oradata/orcl/VotingFile_mirror2
7）然后就开始安装了，当众会让你用完全的root身份在每个节点上运行orainstRoot.sh，你就一个节点一

个节点运行，不要抢时间，一个个来
8）然后要求你在每个节点上用完全的root权限执行 root.sh,你也一样，一个个来，一定要等一个做好了，

然后做另外一个
9）做第二个时候，如果告诉你你的eth0不是public,你就在第一个节点上用ROOT帐户执行图形界面工

具/01/app/product/crs/bin/vipca
next-->选择eth0(public)-->加入两个VIP就可以了。
10)如安装时出现Remote 'AttachHome' failed on 'node2pub'.的错误，需要在第node2pub的节点上ORACLE

用户运行以下commond就可以了。

[oracle@node2pub bin]$ pwd
/u01/app/oracle/product/crs/oui/bin
[oracle@node2pub bin]$ ./runInstaller -attachHome -noClusterEnabled

ORACLE_HOME=/u01/app/oracle/product/crs ORACLE_HOME_NAME=OraCrs10g_home

CLUSTER_NODES=node1pub,node2pub CRS=true "INVENTORY_LOCATION=/u01/app/oracle/oraInventory"

LOCAL_NODE=node2pub

至此CRS安装完毕！
测试一下：
[root@node1pub oracle]# su - oracle
[oracle@node1pub ~]$ cd /u01/app/oracle/product/crs/bin
[oracle@node1pub bin]$ olsnodes -n
node1pub 1
node2pub 2

20.安装Oracle 10gR2 database
1)用oracle用户，运行你unzip之后的那个database目录下的runInstaller
2) ORACLE安装目录指定到 /u01/app/oracle/product/10.2.0/db_1
3) 把2个node选择上
4) 选择 Install database Software only
5) 会要求你用完全的root权限运行 root.sh ,分别在2个node上一一运行，不要抢时间
6) 安装完毕

21.配置netca
1) oracle 用户在一个node上运行 netca
2) 选择所有node
3) 选择 Listener configuration
4)添加一个LISTEN, 1521 port
5) 回到开始界面，选择Naming Methods configuration
6) 把Local Naming和Easy Naming 添加进去.然后结束配置
7. 确认一下
[root@node1pub logs]# ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_NODE1PUB

[root@node2pub db_1]# ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_NODE2PUB

22.安装database

1) 用oracle用户运行 dbca
2) 选择custom database
3) 输入数据库的全局名，比如orcl
4) 给你的系统的不同角色建立密码，我都选择same as ....
5) 选择用ASM管理数据库
6) 输入你前面设定的管理员密码，并把SPFILE 设定成/u02/oradata/orcl/dbs/spfile+ASM.ora
7)进入ASM配置界面, create NEW, 名字叫做DATA
然后下面你会看到你前面创建的4个ASM VOL1-4, 你选择 1 和2, "Redundancy" 选择 Normal.
这样 DATA diskgroup就创建好了
8) 回到ASM界面后. 仍旧选择create new, 然后名字叫做FLASH_RECOVERY_AREA, 选择第3和4两个卷，然

后"Redundancy" 选择External
9) 然后OK结束ASM配置
10)Database File Locations 选择DATA
11) Recovery Configuration 选择刚才创建的 FLASH_RECOVERY_AREA diskgroup
12) 接下来默认的就可以了
13. Database Services 这里，你选择Add你一个新的service, 随便叫名字，比如ractest
然后选择 TAF Policy,是Basic
14. 开始创建数据库

===================步骤六: 校验RAC===============
1.用oracle用户login, 运行

$ srvctl status database -d orcl
Instance orcl1 is running on node node1pub
Instance orcl2 is running on node node2pub

2.
$ srvctl status service -d orcl -s ractest
Service orcltest is running on instance(s) orcl2, orcl1

3.
$ srvctl status nodeapps -n node1pub
VIP is running on node: node1pub
GSD is running on node: node1pub
Listener is running on node: node1pub
ONS daemon is running on node: node1pub

4.
$ srvctl status asm -n node1pub
ASM instance +ASM1 is running on node node1pub.

5.
$ srvctl config nodeapps -n node1pub -a -g -s -l
VIP exists.: /node1vip/192.168.1.111/255.255.255.0/eth0:eth1
GSD exists.
ONS daemon exists.
Listener exists.

6. 运行 sqlplus /nolog

SQL> connect /as sysdba
SQL>

SELECT
inst_id
, instance_number inst_no
, instance_name inst_name
, parallel
, status
, database_status db_status
, active_state state
, host_name host
FROM gv$instance
ORDER BY inst_id;

INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST
-------- -------- ---------- --- ------- ------------ --------- -------
1 1 orcl1 YES OPEN ACTIVE NORMAL node1pub
2 2 orcl2 YES OPEN ACTIVE NORMAL node2pub

7.其他的测试，测试方法n多，你用sqlplus, asmcmd, 或者通过Web进 enterprise manager都可以.