搭建测试环境exadata一体机 （vm虚拟机redhat上配置）
搭建测试环境exadata一体机 （vm虚拟机redhat上配置）
环境：
cell节点：
192.168.9.245
redhat5.4 64
DB节点：
192.168.9.246
redhat5.4 64
cell节点：
1. 安装cell
[root@cell tmp]# unzip V33693-01.zip
[root@cell tmp]# tar -pxvf cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar
[root@cell tmp]# cd dl180/boot/cellbits/
[root@cell cellbits]# unzip cell.bin
Archive: cell.bin
warning [cell.bin]: 6408 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
inflating: jdk-1_5_0_15-linux-amd64.rpm
[root@cell cellbits]# ll
total 1692256
-rw-rw-r-- 1 root root 716 Jul 14 2012 c7rpms.tbz
-rw-r--r-- 1 root root 207051389 Jul 14 2012 cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
-rwxrwxr-x 1 root root 245118813 Jul 14 2012 cell.bin
-rw-rw-r-- 1 root root 12704742 Jul 14 2012 cellboot.tbz
-rw-rw-r-- 1 root root 140810602 Jul 14 2012 cellfw.tbz
-rw-rw-r-- 1 root root 141804661 Jul 14 2012 cellrpms.tbz
-rw-rw-r-- 1 root root 164829961 Jul 14 2012 commonos.tbz
-rw-rw-r-- 1 root root 377243862 Jul 14 2012 debugos.tbz
-rw-rw-r-- 1 root root 42948608 Jul 14 2012 diag.iso
-rw-rw-r-- 1 root root 55704927 Jul 14 2012 doclib.zip
-rw-rw-r-- 1 root root 199480899 Jul 14 2012 exaos.tbz
-rw-rw-r-- 1 root root 18177500 Jul 14 2012 hputils.tbz
-r--r--r-- 1 root root 43175156 Jul 14 2012 jdk-1_5_0_15-linux-amd64.rpm
-rw-rw-r-- 1 root root 53367136 Jul 14 2012 kernel.tbz
-rw-rw-r-- 1 root root 16165584 Jul 14 2012 ofed.tbz
-rw-rw-r-- 1 root root 12476276 Jul 14 2012 sunutils.tbz
[root@cell cellbits]#
2. 安装解压所得的两个rpm包
先安装jdk：
[root@cell cellbits]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm
[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
3. 提示N多条件无法满足安装
创建目录：
[root@cell cellbits]# mkdir -p /var/log/oracle
[root@cell cellbits]# chmod -R 755 /var/log/oracle/
[root@cell cellbits]#
[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
1:cell ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
/
/
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
WARNING: Using the current shell as root to restart cell services.
Restart the cell services using a new shell.
[root@cell cellbits]#
4. 在cell虚拟机中建立对应的虚拟的磁盘和闪盘
[root@cell oracle]# mkdir -p /opt/oracle/cell/disks/raw
[root@cell oracle]# cd /opt/oracle/cell/disks/raw/
[root@cell raw]# vi dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
~
~
~
~
~
~
~
"dd.sh" [New] 16L, 692C written
[root@cell raw]# cat dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
[root@cell raw]#
5. 执行dd.sh创建对应的磁盘和闪盘：其中磁盘12块，每块大小为1GB，闪盘4块，没块大小也是1GB。
[root@cell raw]# sh dd.sh
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.83828 seconds, 273 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.65106 seconds, 137 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.0758 seconds, 104 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.43671 seconds, 111 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.8433 seconds, 88.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 4.57079 seconds, 229 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.744 seconds, 97.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.7691 seconds, 97.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 13.8926 seconds, 75.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.3874 seconds, 92.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.5219 seconds, 83.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.068 seconds, 94.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 5.9232 seconds, 177 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.9895 seconds, 87.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.8567 seconds, 81.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.128 seconds, 115 MB/s
[root@cell raw]# ll -ctrl
total 16400068
-rw-rw---- 1 root root 692 Jan 20 18:24 dd.sh
-rw-r--r-- 1 root root 1048576000 Jan 20 18:24 disk01
-rw-r--r-- 1 root root 1048576000 Jan 20 18:24 disk02
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk03
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk04
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk05
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk06
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk07
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk08
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk09
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk10
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk11
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk12
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 FLASH01
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH02
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH03
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH04
[root@cell raw]# chmod 660 *
[root@cell raw]# ll -ctrl
total 16400068
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH04
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH03
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH02
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH01
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk12
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk11
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk10
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk09
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk08
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk07
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk06
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk05
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk04
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk03
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk02
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk01
-rw-rw---- 1 root root 692 Jan 20 18:27 dd.sh
[root@cell raw]#
6. 切换到celladmin用户，重新启动celld服务
[root@cell raw]# su - celladmin
[celladmin@cell trace]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services...
The STARTUP of MS services was successful.
[celladmin@cell trace]$
启动报错，查看启动日志
[celladmin@cell trace]$ pwd
/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace
[celladmin@cell trace]$ ll
total 3300
-rw-rw---- 1 root celladmin 3746 Jan 20 18:34 alert.log
-rw-r--r-- 1 root celladmin 279510 Jan 20 18:35 ms-odl.log
-rw-r--r-- 1 root celladmin 279510 Jan 20 18:35 ms-odl.trc
-rw-rw---- 1 root celladmin 4262 Jan 20 18:34 rstrc_5331_0.trc
-rw-rw---- 1 root celladmin 683 Jan 20 18:34 rstrc_5331_0.trm
-rw-rw---- 1 root celladmin 2836 Jan 20 18:33 rstrc_5331_3.trc
-rw-rw---- 1 root celladmin 896 Jan 20 18:33 rstrc_5331_3.trm
-rw-rw---- 1 root celladmin 2352658 Jan 20 18:33 rstrc_5331_4.trc
-rw-rw---- 1 root celladmin 288116 Jan 20 18:33 rstrc_5331_4.trm
-rw-rw---- 1 root celladmin 22859 Jan 20 18:33 rstrc_5331_5.trc
-rw-rw---- 1 root celladmin 5356 Jan 20 18:33 rstrc_5331_5.trm
-rw-rw---- 1 root celladmin 869 Jan 20 18:34 rstrc_5331_6.trc
-rw-rw---- 1 root celladmin 358 Jan 20 18:34 rstrc_5331_6.trm
-rw-rw---- 1 root celladmin 3450 Jan 20 18:33 rstrc_5340_1.trc
-rw-rw---- 1 root celladmin 526 Jan 20 18:33 rstrc_5340_1.trm
-rw-rw---- 1 root celladmin 1456 Jan 20 18:34 rstrc_5340_2.trc
-rw-rw---- 1 root celladmin 396 Jan 20 18:34 rstrc_5340_2.trm
-rw-rw---- 1 root celladmin 1561 Jan 20 18:34 rstrc_6681_0.trc
-rw-rw---- 1 root celladmin 255 Jan 20 18:34 rstrc_6681_0.trm
-rw-rw---- 1 root celladmin 815 Jan 20 18:34 rstrc_6681_3.trc
-rw-rw---- 1 root celladmin 325 Jan 20 18:34 rstrc_6681_3.trm
-rw-rw---- 1 root celladmin 4346 Jan 20 18:34 rstrc_6681_4.trc
-rw-rw---- 1 root celladmin 1491 Jan 20 18:34 rstrc_6681_4.trm
-rw-rw---- 1 root celladmin 9667 Jan 20 18:34 rstrc_6681_5.trc
-rw-rw---- 1 root celladmin 3244 Jan 20 18:34 rstrc_6681_5.trm
-rw-rw---- 1 root celladmin 771 Jan 20 18:34 rstrc_6681_6.trc
-rw-rw---- 1 root celladmin 190 Jan 20 18:34 rstrc_6681_6.trm
-rw-rw---- 1 root celladmin 1429 Jan 20 18:34 rstrc_6689_1.trc
-rw-rw---- 1 root celladmin 254 Jan 20 18:34 rstrc_6689_1.trm
-rw-rw---- 1 root celladmin 513 Jan 20 18:34 rstrc_6689_2.trc
-rw-rw---- 1 root celladmin 169 Jan 20 18:34 rstrc_6689_2.trm
[celladmin@cell trace]$ cat alert.log
Tue Jan 20 18:12:16 2015
RS version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_MAIN with pid 5331
[RS] Kill previous monitoring processes for backup RS, MS and Cellsrv
Tue Jan 20 18:12:16 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt with pid 5339
Tue Jan 20 18:12:16 2015
RSBK version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_BACKUP with pid 5340
[RS] Kill previous monitoring process for core RS
Tue Jan 20 18:12:16 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt with pid 5344
Tue Jan 20 18:12:17 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt with pid 5376
[RS] Started Service MS with pid 5377
Tue Jan 20 18:33:51 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt (pid: 5376) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:33:58 2015
[RS] Stopped Service MS with pid 5377
Tue Jan 20 18:33:58 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt (pid: 5339) received clean shutdown signal from pid: 5331, uid: 0
[RS] Stopped Service RS_BACKUP
[RS] Stopped Service RS_MAIN
Tue Jan 20 18:33:59 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbkm (pid: 5340) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:00 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt (pid: 5344) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:01 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssrm (pid: 5331) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:02 2015
RS version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_MAIN with pid 6681
[RS] Kill previous monitoring processes for backup RS, MS and Cellsrv
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt with pid 6688
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt with pid 6690
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt with pid 6691
[RS] Required IP parameters not configured in cellinit.ora. Err: 36
Tue Jan 20 18:34:02 2015
RSBK version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_BACKUP with pid 6689
[RS] Kill previous monitoring process for core RS
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt with pid 6696
Errors in file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_6681_4.trc (incident=1):
RS-7445 [Required IP parameters missing] [Check cellinit.ora] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/incident/incdir_1/rstrc_6681_4_i1.trc
Sweep [inc][1]: completed
[RS] Required IP parameters not configured in cellinit.ora. Err: 36
OS Hugepage status:
Total/free hugepages available=1/1; hugepage size=2048KB
[RS] Start service CELLSRV failed with error: -74.
Tue Jan 20 18:34:03 2015
Could not connect to MS socket. Communication with MS may be degraded. errno=115
[RS] monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt (pid: 0) returned with error: 124
[RS] Started Service MS with pid 6692
[celladmin@cell trace]$
启动报错，可以在trc文件17行中查看到详细报错信息
Flood ctrl function: Curtime: 1421751436, triggertime: 1421747836
[celladmin@cell trace]$ vi /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_8377_4.trc
1 Trace file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_8377_4.tr c
2 ORACLE_HOME = /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713
3 System name: Linux
4 Node name: cell
5 Release: 2.6.18-164.el5
6 Version: #1 SMP Tue Aug 18 15:51:48 EDT 2009
7 Machine: x86_64
8 CELL SW Version: OSS_11.2.3.2.0_LINUX.X64_120713
9
10 *** 2015-01-20 18:57:16.190
11 RSMain: skgznp_create on pid 8377 returned 0
12 serv_status - 1, exit_status - 0, cur_inc_num - 0, prev_inc_num - 0
13 service CELLSRV already has status 1, modify to 1
14 Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt with p id 8387
15 RSOMT: skgznp_create returned 0
16 RSOMT: skgznp_set_fasync returned 0
17 Required IP parameters not configured in cellinit.ora. Err: 36
18 Required IP parameters missing Check cellinit.ora
19 -------------------------------------------------------------------------------
20 Trace Bucket Dump Begin: default trace bucket
21 TIME(*=approx):SEQ: DATA
22 -------------------------------------------------------------------------------
23 2015-01-20 18:57:16.191617 :0000000A: serv_status - 1, exit_status - 0, cur_inc_num - 0, prev_inc_n um - 0
24 2015-01-20 18:57:16.191685 :0000000B: service CELLSRV already has status 1, modify to 1
25 2015-01-20 18:57:16.192110 :0000000C: Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X 64_120713/cellsrv/bin/cellrsomt with pid 8387
26 2015-01-20 18:57:16.194326 :0000000D: Required IP parameters not configured in cellinit.ora. Err: 3 6
27 2015-01-20 18:57:16.194348 :0000000E: Required IP parameters missing Check cellinit.ora
28 -------------------------------------------------------------------------------
29 Trace Bucket Dump End: default trace bucket
30 DDE: Flood control is not active
@
:set nu
17 Required IP parameters not configured in cellinit.ora. Err: 36 存储告警
修改系统打开文件参数
原来只能打开 1024个文件
[root@cell ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16736
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16736
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@cell ~]#
修改：
参数fs.file-max大小
[root@cell ~]# vi /etc/sysctl.conf
fs.file-max = 655363
生效：
[root@cell ~]# sysctl -p
open files大小修改：
[root@cell ~]# vi /etc/security/limits.conf
文件末尾加上：
* soft nofile 65536
* hard nofile 65536
该参数修改，必须重启才可以生效！！
查看
[root@cell ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16736
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16736
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@cell ~]#
修改成功！！！
再次重新启动celld服务
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 20:57:50 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI>
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
CELL-01509: Restart Server (RS) not responding.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
CellCLI>
RS, CELLSRV, and MS 服务全部启动成功！！
7. 在cellinit.ora中添加网卡的信息
/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora
[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0
CELL-01518: Stop CELLSRV. Create Cell cannot continue with CELLSRV running.
报错，先关闭 CELLSRV 进程
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:14:40 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> alter cell shutdown services cellsrv
Stopping CELLSRV services...
The SHUTDOWN of CELLSRV services was successful.
CellCLI> exit
quitting
[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0
Cell cell1 successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created...
CellDisk FD_00_cell1 successfully created
CellDisk FD_01_cell1 successfully created
CellDisk FD_02_cell1 successfully created
CellDisk FD_03_cell1 successfully created
Flash log cell1_FLASHLOG successfully created
Flash cache cell1_FLASHCACHE successfully created
[celladmin@cell ~]$
停启进程语法：
ALTER CELL { SHUTDOWN | RESTART | STARTUP } SERVICES { RS | MS | CELLSRV | ALL}
看到cellinit.ora文件中添加了一行ipaddress1=192.168.9.245/24类似的信息
[celladmin@cell ~]$ cat /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora
#CELL Initialization Parameters
version=0.0
HTTP_PORT=8888
bbuChargeThreshold=800
SSL_PORT=23943
RMI_PORT=23791
ipaddress1=192.168.9.245/24
bbuTempThreshold=60
DEPLOYED=TRUE
JMS_PORT=9127
BMC_SNMP_PORT=162
[celladmin@cell ~]$
8. 创建celldisk, griddisk, flashcache, flashlog
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:22:45 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
CellCLI> create celldisk all
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created
CellCLI> create flashcache all size=2G
CELL-02645: Flash cache already exists.
CellCLI> create griddisk all harddisk prefix=data
GridDisk data_CD_disk01_cell1 successfully created
GridDisk data_CD_disk02_cell1 successfully created
GridDisk data_CD_disk03_cell1 successfully created
GridDisk data_CD_disk04_cell1 successfully created
GridDisk data_CD_disk05_cell1 successfully created
GridDisk data_CD_disk06_cell1 successfully created
GridDisk data_CD_disk07_cell1 successfully created
GridDisk data_CD_disk08_cell1 successfully created
GridDisk data_CD_disk09_cell1 successfully created
GridDisk data_CD_disk10_cell1 successfully created
GridDisk data_CD_disk11_cell1 successfully created
GridDisk data_CD_disk12_cell1 successfully created
CellCLI> create flashlog all
CELL-02722: Flash log already exists.
CellCLI> list flashcache detail
name: cell1_FLASHCACHE
cellDisk: FD_01_cell1,FD_02_cell1,FD_00_cell1,FD_03_cell1
creationTime: 2015-01-20T22:15:22+08:00
degradedCelldisks:
effectiveCacheSize: 3.1875G
id: a97a7329-9169-4534-aac4-c28fac01a819
size: 3.1875G
status: normal
CellCLI> list flashlog detail
name: cell1_FLASHLOG
cellDisk: FD_02_cell1,FD_01_cell1,FD_00_cell1,FD_03_cell1
creationTime: 2015-01-20T22:15:20+08:00
degradedCelldisks:
effectiveSize: 512M
efficiency: 100.0
id: f4957f27-d1b6-463e-8d54-b1497cf90305
size: 512M
status: normal
CellCLI> list cell
cell1 online
CellCLI> list celldisk
CD_disk01_cell1 normal
CD_disk02_cell1 normal
CD_disk03_cell1 normal
CD_disk04_cell1 normal
CD_disk05_cell1 normal
CD_disk06_cell1 normal
CD_disk07_cell1 normal
CD_disk08_cell1 normal
CD_disk09_cell1 normal
CD_disk10_cell1 normal
CD_disk11_cell1 normal
CD_disk12_cell1 normal
FD_00_cell1 normal
FD_01_cell1 normal
FD_02_cell1 normal
FD_03_cell1 normal
CellCLI> list griddisk
data_CD_disk01_cell1 active
data_CD_disk02_cell1 active
data_CD_disk03_cell1 active
data_CD_disk04_cell1 active
data_CD_disk05_cell1 active
data_CD_disk06_cell1 active
data_CD_disk07_cell1 active
data_CD_disk08_cell1 active
data_CD_disk09_cell1 active
data_CD_disk10_cell1 active
data_CD_disk11_cell1 active
data_CD_disk12_cell1 active
CellCLI> list celldisk CD_disk01_cell1 detail
name: CD_disk01_cell1
comment:
creationTime: 2015-01-20T22:52:22+08:00
deviceName: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
devicePartition: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
diskType: HardDisk
errorCount: 0
freeSpace: 0
id: 572530b2-bed4-432d-8d46-093990067df8
interleaving: none
lun: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
physicalDisk: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
raidLevel: "RAID 0"
size: 992M
status: normal
CellCLI> list celldisk FD_00_cell1 detail
name: FD_00_cell1
comment:
creationTime: 2015-01-20T22:15:19+08:00
deviceName: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
devicePartition: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
diskType: FlashDisk
errorCount: 0
freeSpace: 0
id: d6346116-54fb-4501-bb44-dce5a8ad19c6
interleaving: none
lun: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
physicalDisk: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
raidLevel: "RAID 0"
size: 992M
status: normal
CellCLI> list griddisk data_CD_disk01_cell1 detail
name: data_CD_disk01_cell1
availableTo:
cachingPolicy: default
cellDisk: CD_disk01_cell1
comment:
creationTime: 2015-01-20T22:54:39+08:00
diskType: HardDisk
errorCount: 0
id: 5cfecc67-f540-41e1-bcab-36a64303abc2
offset: 48M
size: 944M
status: active
CellCLI>
cell节点创建完成！！！
DB节点：（参考rac安装博文http://blog.itpub.net/29500582/viewspace-1341453/）
1. 安装oracle数据库的基本步骤：
1.1 yum包安装
yum -y install binutils
yum -y install compat-libstdc++-33
yum -y install elfutils-libelf
yum -y install glibc
yum -y install glibc-common
yum -y install ksh
yum -y install libaio
yum -y install libgcc
yum -y install libstdc++
yum -y install make
yum -y install elfutils-libelf-devel
yum -y install glibc-headers
yum -y install glibc-devel
yum -y install gcc
yum -y install libstdc++-devel
yum -y install gcc-c++
yum -y install libaio-devel
yum -y install sysstat
yum -y install unixODBC
yum -y install unixODBC-devel
2. 系统初始化
2.1 配置系统参数
vi /etc/sysctl.conf
添加
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
kernel.shmmni = 4096
kernel.sem = 310 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
生效
sysctl -p
2.2 创建用户
groupadd -g 500 oinstall
groupadd -g 501 dba
groupadd -g 502 oper
groupadd -g 503 asmdba
groupadd -g 504 asmadmin
groupadd -g 505 asmoper
useradd -u 500 -g oinstall -G dba,oper,asmdba oracle
useradd -u 501 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper grid
passwd oracle
passwd grid
[root@rac1 tmp]# id oracle
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba),502(oper),503(asmdba)
[root@rac1 tmp]# id grid
uid=501(grid) gid=500(oinstall) groups=500(oinstall),501(dba),502(oper),503(asmdba),504(asmadmin),505(asmoper)
2.3 创建密码不过期
chage -M 99999 oracle
chage -M 99999 grid
2.4 创建目录给权限
mkdir -p /app/grid
mkdir -p /app/oracle
mkdir -p /app/oraInventory
mkdir -p /app/soft
mkdir -p /app/soft/grid
mkdir -p /app/soft/oracle
mkdir -p /app/soft/asmlib
mkdir -p /backup
mkdir -p /arch
chown -R oracle:oinstall /app
chown -R oracle:oinstall /arch
chown -R oracle:oinstall /backup
chown -R grid:oinstall /app/grid
chown -R oracle:oinstall /app/oracle
chown -R grid:oinstall /app/oraInventory
chown -R grid:oinstall /app/soft/grid
chown -R oracle:oinstall /app/soft/oracle
chmod -R 755 /app
2.5 修改Oracle用户shell权限
vi /etc/security/limits.conf
添加：
* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536
2.6 修改配置文件vi /etc/profile
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
2.7 关闭selinux
vi /etc/sysconfig/selinux
#modify by lcs for oracle
#old SELINUX=enforcing
SELINUX=disabled
2.8 关闭防火墙
service iptables stop
chkconfig iptables off
2.9 用户限制，登录
vi /etc/pam.d/login
添加：
session required pam_limits.so
3. 创建两个Exadata所需的配置文件。Exadata所需要的配置文件有两个一个是cellinit.ora，此文件中填入DB节点的ip地址和子网掩码。
另外一个是cellip.ora, 此文件填入Cell节点的ip地址
[root@exadata ~]# mkdir -p /etc/oracle/cell/network-config
[root@exadata ~]# chown -R grid:oinstall /etc/oracle/cell/network-config
[root@exadata ~]# chmod -R 775 /etc/oracle/cell/network-config
[root@exadata ~]# vi /etc/oracle/cell/network-config/cellinit.ora
ipaddress1=192.168.9.246/24
~
~
~
"/etc/oracle/cell/network-config/cellinit.ora" [New] 1L, 28C written
[root@exadata ~]# vi /etc/oracle/cell/network-config/cellip.ora
cell="192.168.9.245"
~
~
~
~
"/etc/oracle/cell/network-config/cellip.ora" [New] 1L, 21C written
[root@exadata ~]#
4. 安装asm管理
下载地址：http://www.oracle.com/technetwork/server-storage/linux/downloads/rhel5-084877.html
下载时注意版本内核
[root@rac1 yum.repos.d]# uname -a
Linux rac1 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@rac1 yum.repos.d]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
[root@rac1 yum.repos.d]#
内核参数： 2.6.18-164.el5
[root@rac1 oracleasm]# cd /tmp/oracleasm/
[root@rac1 oracleasm]# ll
total 536
-rw-r--r-- 1 root root 136910 Nov 19 13:51 oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 138539 Nov 19 13:51 oracleasm-2.6.18-164.el5debug-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 2807 Nov 19 13:51 oracleasm-2.6.18-164.el5-debuginfo-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 136839 Nov 19 13:51 oracleasm-2.6.18-164.el5xen-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 14176 Nov 19 13:51 oracleasmlib-2.0.4-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 90225 Nov 19 13:51 oracleasm-support-2.1.8-1.el5.x86_64.rpm
直接rpm包的安装
rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm
rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
初始化 asmlib，在所有节点执行；
[root@rac1 ~]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@rac1 ~]#
ASMlib 加载到内核，在所有节点执行；
[root@rac1 ~]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@rac1 ~]#
5. 配置oracle用户，grid用户环境变量
6. 安装grid软件
先配一些普通的裸盘（raw disk）做asm
这些以o/开头的ASM磁盘就是Exadata Cell端创建的griddisk。
当然在安装的时候可以选择普通的裸盘（raw disk）而非griddisk作为磁盘组，
到Grid Infrastructure安装完毕以后通过asmca来添加/新建新的磁盘组
7. 安装oracle软件
8. 安装oracle数据库
9. 配监听等等
10. 简单验证操作
[grid@exadata ~]:+ASM> asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 4194304 10236 10156 0 10156 0 N BACK/
MOUNTED EXTERN N 512 4096 4194304 944 864 0 864 0 N CELL1/
MOUNTED EXTERN N 512 4096 4194304 8496 6460 0 6460 0 N CELLDATA/
MOUNTED EXTERN N 512 4096 1048576 15358 15299 0 15299 0 N DATA/
MOUNTED NORMAL N 512 4096 4194304 10224 10064 0 5032 0 N OCR/
ASMCMD> lsattr -l -G data
Name Value
access_control.enabled FALSE
access_control.umask 066
au_size 1048576
cell.smart_scan_capable FALSE
compatible.asm 11.2.0.0.0
compatible.rdbms 10.1.0.0.0
disk_repair_time 3.6h
sector_size 512
ASMCMD> lsdsk -G data
Path
ORCL:DATA_VOL1
ASMCMD> lsdsk -G celldata
Path
o/192.168.9.245/data_CD_disk02_cell1
o/192.168.9.245/data_CD_disk03_cell1
o/192.168.9.245/data_CD_disk04_cell1
o/192.168.9.245/data_CD_disk05_cell1
o/192.168.9.245/data_CD_disk06_cell1
o/192.168.9.245/data_CD_disk07_cell1
o/192.168.9.245/data_CD_disk08_cell1
o/192.168.9.245/data_CD_disk09_cell1
o/192.168.9.245/data_CD_disk10_cell1
ASMCMD>
SQL> select disk_number, name, path from v$asm_disk where path like 'o/%' order by disk_number;
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
0 DATA_CD_DISK01_CELL1
o/192.168.9.245/data_CD_disk01_cell1
0 DATA_CD_DISK02_CELL1
o/192.168.9.245/data_CD_disk02_cell1
1 DATA_CD_DISK03_CELL1
o/192.168.9.245/data_CD_disk03_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
2 DATA_CD_DISK04_CELL1
o/192.168.9.245/data_CD_disk04_cell1
3 DATA_CD_DISK05_CELL1
o/192.168.9.245/data_CD_disk05_cell1
4 DATA_CD_DISK06_CELL1
o/192.168.9.245/data_CD_disk06_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
5 DATA_CD_DISK07_CELL1
o/192.168.9.245/data_CD_disk07_cell1
6 DATA_CD_DISK08_CELL1
o/192.168.9.245/data_CD_disk08_cell1
7
o/192.168.9.245/data_CD_disk12_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
7 DATA_CD_DISK09_CELL1
o/192.168.9.245/data_CD_disk09_cell1
8 DATA_CD_DISK10_CELL1
o/192.168.9.245/data_CD_disk10_cell1
10
o/192.168.9.245/data_CD_disk11_cell1
12 rows selected.
SQL>
SQL> select name, path from v$asm_disk;
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
BACK_VOL1
ORCL:BACK_VOL1
DATA_VOL1
ORCL:DATA_VOL1
OCR_VOL1
ORCL:OCR_VOL1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
OCR_VOL2
ORCL:OCR_VOL2
DATA_CD_DISK04_CELL1
o/192.168.9.245/data_CD_disk04_cell1
DATA_CD_DISK06_CELL1
o/192.168.9.245/data_CD_disk06_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK05_CELL1
o/192.168.9.245/data_CD_disk05_cell1
DATA_CD_DISK03_CELL1
o/192.168.9.245/data_CD_disk03_cell1
DATA_CD_DISK07_CELL1
o/192.168.9.245/data_CD_disk07_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK09_CELL1
o/192.168.9.245/data_CD_disk09_cell1
DATA_CD_DISK08_CELL1
o/192.168.9.245/data_CD_disk08_cell1
DATA_CD_DISK02_CELL1
o/192.168.9.245/data_CD_disk02_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK01_CELL1
o/192.168.9.245/data_CD_disk01_cell1
DATA_CD_DISK10_CELL1
o/192.168.9.245/data_CD_disk10_cell1
o/192.168.9.245/data_CD_disk11_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
o/192.168.9.245/data_CD_disk12_cell1
16 rows selected.
SQL>
SQL> select file#, name from v$datafile;
FILE#
----------
NAME
--------------------------------------------------------------------------------
1
+CELLDATA/exadata/datafile/system.261.869661803
2
+CELLDATA/exadata/datafile/sysaux.262.869661811
3
+CELLDATA/exadata/datafile/undotbs1.263.869661817
FILE#
----------
NAME
--------------------------------------------------------------------------------
4
+CELLDATA/exadata/datafile/users.265.869661827
SQL>
环境：
cell节点：
192.168.9.245
redhat5.4 64
DB节点：
192.168.9.246
redhat5.4 64
cell节点：
1. 安装cell
[root@cell tmp]# unzip V33693-01.zip
[root@cell tmp]# tar -pxvf cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar
[root@cell tmp]# cd dl180/boot/cellbits/
[root@cell cellbits]# unzip cell.bin
Archive: cell.bin
warning [cell.bin]: 6408 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
inflating: jdk-1_5_0_15-linux-amd64.rpm
[root@cell cellbits]# ll
total 1692256
-rw-rw-r-- 1 root root 716 Jul 14 2012 c7rpms.tbz
-rw-r--r-- 1 root root 207051389 Jul 14 2012 cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
-rwxrwxr-x 1 root root 245118813 Jul 14 2012 cell.bin
-rw-rw-r-- 1 root root 12704742 Jul 14 2012 cellboot.tbz
-rw-rw-r-- 1 root root 140810602 Jul 14 2012 cellfw.tbz
-rw-rw-r-- 1 root root 141804661 Jul 14 2012 cellrpms.tbz
-rw-rw-r-- 1 root root 164829961 Jul 14 2012 commonos.tbz
-rw-rw-r-- 1 root root 377243862 Jul 14 2012 debugos.tbz
-rw-rw-r-- 1 root root 42948608 Jul 14 2012 diag.iso
-rw-rw-r-- 1 root root 55704927 Jul 14 2012 doclib.zip
-rw-rw-r-- 1 root root 199480899 Jul 14 2012 exaos.tbz
-rw-rw-r-- 1 root root 18177500 Jul 14 2012 hputils.tbz
-r--r--r-- 1 root root 43175156 Jul 14 2012 jdk-1_5_0_15-linux-amd64.rpm
-rw-rw-r-- 1 root root 53367136 Jul 14 2012 kernel.tbz
-rw-rw-r-- 1 root root 16165584 Jul 14 2012 ofed.tbz
-rw-rw-r-- 1 root root 12476276 Jul 14 2012 sunutils.tbz
[root@cell cellbits]#
2. 安装解压所得的两个rpm包
先安装jdk：
[root@cell cellbits]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm
[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
3. 提示N多条件无法满足安装
创建目录：
[root@cell cellbits]# mkdir -p /var/log/oracle
[root@cell cellbits]# chmod -R 755 /var/log/oracle/
[root@cell cellbits]#
[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
1:cell ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
/
/
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
WARNING: Using the current shell as root to restart cell services.
Restart the cell services using a new shell.
[root@cell cellbits]#
4. 在cell虚拟机中建立对应的虚拟的磁盘和闪盘
[root@cell oracle]# mkdir -p /opt/oracle/cell/disks/raw
[root@cell oracle]# cd /opt/oracle/cell/disks/raw/
[root@cell raw]# vi dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
~
~
~
~
~
~
~
"dd.sh" [New] 16L, 692C written
[root@cell raw]# cat dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
[root@cell raw]#
5. 执行dd.sh创建对应的磁盘和闪盘：其中磁盘12块，每块大小为1GB，闪盘4块，没块大小也是1GB。
[root@cell raw]# sh dd.sh
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.83828 seconds, 273 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.65106 seconds, 137 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.0758 seconds, 104 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.43671 seconds, 111 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.8433 seconds, 88.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 4.57079 seconds, 229 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.744 seconds, 97.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 10.7691 seconds, 97.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 13.8926 seconds, 75.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.3874 seconds, 92.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.5219 seconds, 83.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.068 seconds, 94.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 5.9232 seconds, 177 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.9895 seconds, 87.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.8567 seconds, 81.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 9.128 seconds, 115 MB/s
[root@cell raw]# ll -ctrl
total 16400068
-rw-rw---- 1 root root 692 Jan 20 18:24 dd.sh
-rw-r--r-- 1 root root 1048576000 Jan 20 18:24 disk01
-rw-r--r-- 1 root root 1048576000 Jan 20 18:24 disk02
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk03
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk04
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk05
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk06
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk07
-rw-r--r-- 1 root root 1048576000 Jan 20 18:25 disk08
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk09
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk10
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk11
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 disk12
-rw-r--r-- 1 root root 1048576000 Jan 20 18:26 FLASH01
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH02
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH03
-rw-r--r-- 1 root root 1048576000 Jan 20 18:27 FLASH04
[root@cell raw]# chmod 660 *
[root@cell raw]# ll -ctrl
total 16400068
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH04
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH03
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH02
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 FLASH01
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk12
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk11
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk10
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk09
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk08
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk07
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk06
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk05
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk04
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk03
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk02
-rw-rw---- 1 root root 1048576000 Jan 20 18:27 disk01
-rw-rw---- 1 root root 692 Jan 20 18:27 dd.sh
[root@cell raw]#
6. 切换到celladmin用户，重新启动celld服务
[root@cell raw]# su - celladmin
[celladmin@cell trace]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services...
The STARTUP of MS services was successful.
[celladmin@cell trace]$
启动报错，查看启动日志
[celladmin@cell trace]$ pwd
/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace
[celladmin@cell trace]$ ll
total 3300
-rw-rw---- 1 root celladmin 3746 Jan 20 18:34 alert.log
-rw-r--r-- 1 root celladmin 279510 Jan 20 18:35 ms-odl.log
-rw-r--r-- 1 root celladmin 279510 Jan 20 18:35 ms-odl.trc
-rw-rw---- 1 root celladmin 4262 Jan 20 18:34 rstrc_5331_0.trc
-rw-rw---- 1 root celladmin 683 Jan 20 18:34 rstrc_5331_0.trm
-rw-rw---- 1 root celladmin 2836 Jan 20 18:33 rstrc_5331_3.trc
-rw-rw---- 1 root celladmin 896 Jan 20 18:33 rstrc_5331_3.trm
-rw-rw---- 1 root celladmin 2352658 Jan 20 18:33 rstrc_5331_4.trc
-rw-rw---- 1 root celladmin 288116 Jan 20 18:33 rstrc_5331_4.trm
-rw-rw---- 1 root celladmin 22859 Jan 20 18:33 rstrc_5331_5.trc
-rw-rw---- 1 root celladmin 5356 Jan 20 18:33 rstrc_5331_5.trm
-rw-rw---- 1 root celladmin 869 Jan 20 18:34 rstrc_5331_6.trc
-rw-rw---- 1 root celladmin 358 Jan 20 18:34 rstrc_5331_6.trm
-rw-rw---- 1 root celladmin 3450 Jan 20 18:33 rstrc_5340_1.trc
-rw-rw---- 1 root celladmin 526 Jan 20 18:33 rstrc_5340_1.trm
-rw-rw---- 1 root celladmin 1456 Jan 20 18:34 rstrc_5340_2.trc
-rw-rw---- 1 root celladmin 396 Jan 20 18:34 rstrc_5340_2.trm
-rw-rw---- 1 root celladmin 1561 Jan 20 18:34 rstrc_6681_0.trc
-rw-rw---- 1 root celladmin 255 Jan 20 18:34 rstrc_6681_0.trm
-rw-rw---- 1 root celladmin 815 Jan 20 18:34 rstrc_6681_3.trc
-rw-rw---- 1 root celladmin 325 Jan 20 18:34 rstrc_6681_3.trm
-rw-rw---- 1 root celladmin 4346 Jan 20 18:34 rstrc_6681_4.trc
-rw-rw---- 1 root celladmin 1491 Jan 20 18:34 rstrc_6681_4.trm
-rw-rw---- 1 root celladmin 9667 Jan 20 18:34 rstrc_6681_5.trc
-rw-rw---- 1 root celladmin 3244 Jan 20 18:34 rstrc_6681_5.trm
-rw-rw---- 1 root celladmin 771 Jan 20 18:34 rstrc_6681_6.trc
-rw-rw---- 1 root celladmin 190 Jan 20 18:34 rstrc_6681_6.trm
-rw-rw---- 1 root celladmin 1429 Jan 20 18:34 rstrc_6689_1.trc
-rw-rw---- 1 root celladmin 254 Jan 20 18:34 rstrc_6689_1.trm
-rw-rw---- 1 root celladmin 513 Jan 20 18:34 rstrc_6689_2.trc
-rw-rw---- 1 root celladmin 169 Jan 20 18:34 rstrc_6689_2.trm
[celladmin@cell trace]$ cat alert.log
Tue Jan 20 18:12:16 2015
RS version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_MAIN with pid 5331
[RS] Kill previous monitoring processes for backup RS, MS and Cellsrv
Tue Jan 20 18:12:16 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt with pid 5339
Tue Jan 20 18:12:16 2015
RSBK version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_BACKUP with pid 5340
[RS] Kill previous monitoring process for core RS
Tue Jan 20 18:12:16 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt with pid 5344
Tue Jan 20 18:12:17 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt with pid 5376
[RS] Started Service MS with pid 5377
Tue Jan 20 18:33:51 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt (pid: 5376) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:33:58 2015
[RS] Stopped Service MS with pid 5377
Tue Jan 20 18:33:58 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt (pid: 5339) received clean shutdown signal from pid: 5331, uid: 0
[RS] Stopped Service RS_BACKUP
[RS] Stopped Service RS_MAIN
Tue Jan 20 18:33:59 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbkm (pid: 5340) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:00 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt (pid: 5344) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:01 2015
[RS] Process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssrm (pid: 5331) received clean shutdown signal from pid: 5331, uid: 0
Tue Jan 20 18:34:02 2015
RS version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_MAIN with pid 6681
[RS] Kill previous monitoring processes for backup RS, MS and Cellsrv
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsbmt with pid 6688
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsmmt with pid 6690
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt with pid 6691
[RS] Required IP parameters not configured in cellinit.ora. Err: 36
Tue Jan 20 18:34:02 2015
RSBK version=11.2.3.2.0,label=OSS_11.2.3.2.0_LINUX.X64_120713,Fri_Jul_13_12:37:13_PDT_2012
[RS] Started Service RS_BACKUP with pid 6689
[RS] Kill previous monitoring process for core RS
Tue Jan 20 18:34:02 2015
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt with pid 6696
Errors in file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_6681_4.trc (incident=1):
RS-7445 [Required IP parameters missing] [Check cellinit.ora] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/incident/incdir_1/rstrc_6681_4_i1.trc
Sweep [inc][1]: completed
[RS] Required IP parameters not configured in cellinit.ora. Err: 36
OS Hugepage status:
Total/free hugepages available=1/1; hugepage size=2048KB
[RS] Start service CELLSRV failed with error: -74.
Tue Jan 20 18:34:03 2015
Could not connect to MS socket. Communication with MS may be degraded. errno=115
[RS] monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt (pid: 0) returned with error: 124
[RS] Started Service MS with pid 6692
[celladmin@cell trace]$
启动报错，可以在trc文件17行中查看到详细报错信息
Flood ctrl function: Curtime: 1421751436, triggertime: 1421747836
[celladmin@cell trace]$ vi /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_8377_4.trc
1 Trace file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_8377_4.tr c
2 ORACLE_HOME = /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713
3 System name: Linux
4 Node name: cell
5 Release: 2.6.18-164.el5
6 Version: #1 SMP Tue Aug 18 15:51:48 EDT 2009
7 Machine: x86_64
8 CELL SW Version: OSS_11.2.3.2.0_LINUX.X64_120713
9
10 *** 2015-01-20 18:57:16.190
11 RSMain: skgznp_create on pid 8377 returned 0
12 serv_status - 1, exit_status - 0, cur_inc_num - 0, prev_inc_num - 0
13 service CELLSRV already has status 1, modify to 1
14 Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrsomt with p id 8387
15 RSOMT: skgznp_create returned 0
16 RSOMT: skgznp_set_fasync returned 0
17 Required IP parameters not configured in cellinit.ora. Err: 36
18 Required IP parameters missing Check cellinit.ora
19 -------------------------------------------------------------------------------
20 Trace Bucket Dump Begin: default trace bucket
21 TIME(*=approx):SEQ: DATA
22 -------------------------------------------------------------------------------
23 2015-01-20 18:57:16.191617 :0000000A: serv_status - 1, exit_status - 0, cur_inc_num - 0, prev_inc_n um - 0
24 2015-01-20 18:57:16.191685 :0000000B: service CELLSRV already has status 1, modify to 1
25 2015-01-20 18:57:16.192110 :0000000C: Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X 64_120713/cellsrv/bin/cellrsomt with pid 8387
26 2015-01-20 18:57:16.194326 :0000000D: Required IP parameters not configured in cellinit.ora. Err: 3 6
27 2015-01-20 18:57:16.194348 :0000000E: Required IP parameters missing Check cellinit.ora
28 -------------------------------------------------------------------------------
29 Trace Bucket Dump End: default trace bucket
30 DDE: Flood control is not active
@
:set nu
17 Required IP parameters not configured in cellinit.ora. Err: 36 存储告警
修改系统打开文件参数
原来只能打开 1024个文件
[root@cell ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16736
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16736
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@cell ~]#
修改：
参数fs.file-max大小
[root@cell ~]# vi /etc/sysctl.conf
fs.file-max = 655363
生效：
[root@cell ~]# sysctl -p
open files大小修改：
[root@cell ~]# vi /etc/security/limits.conf
文件末尾加上：
* soft nofile 65536
* hard nofile 65536
该参数修改，必须重启才可以生效！！
查看
[root@cell ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16736
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16736
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@cell ~]#
修改成功！！！
再次重新启动celld服务
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 20:57:50 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI>
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
CELL-01509: Restart Server (RS) not responding.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
CellCLI>
RS, CELLSRV, and MS 服务全部启动成功！！
7. 在cellinit.ora中添加网卡的信息
/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora
[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0
CELL-01518: Stop CELLSRV. Create Cell cannot continue with CELLSRV running.
报错，先关闭 CELLSRV 进程
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:14:40 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> alter cell shutdown services cellsrv
Stopping CELLSRV services...
The SHUTDOWN of CELLSRV services was successful.
CellCLI> exit
quitting
[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0
Cell cell1 successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Flash cell disks, FlashCache, and FlashLog will be created...
CellDisk FD_00_cell1 successfully created
CellDisk FD_01_cell1 successfully created
CellDisk FD_02_cell1 successfully created
CellDisk FD_03_cell1 successfully created
Flash log cell1_FLASHLOG successfully created
Flash cache cell1_FLASHCACHE successfully created
[celladmin@cell ~]$
停启进程语法：
ALTER CELL { SHUTDOWN | RESTART | STARTUP } SERVICES { RS | MS | CELLSRV | ALL}
看到cellinit.ora文件中添加了一行ipaddress1=192.168.9.245/24类似的信息
[celladmin@cell ~]$ cat /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/config/cellinit.ora
#CELL Initialization Parameters
version=0.0
HTTP_PORT=8888
bbuChargeThreshold=800
SSL_PORT=23943
RMI_PORT=23791
ipaddress1=192.168.9.245/24
bbuTempThreshold=60
DEPLOYED=TRUE
JMS_PORT=9127
BMC_SNMP_PORT=162
[celladmin@cell ~]$
8. 创建celldisk, griddisk, flashcache, flashlog
[celladmin@cell ~]$ cellcli
CellCLI: Release 11.2.3.2.0 - Production on Tue Jan 20 22:22:45 CST 2015
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
CellCLI> create celldisk all
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created
CellCLI> create flashcache all size=2G
CELL-02645: Flash cache already exists.
CellCLI> create griddisk all harddisk prefix=data
GridDisk data_CD_disk01_cell1 successfully created
GridDisk data_CD_disk02_cell1 successfully created
GridDisk data_CD_disk03_cell1 successfully created
GridDisk data_CD_disk04_cell1 successfully created
GridDisk data_CD_disk05_cell1 successfully created
GridDisk data_CD_disk06_cell1 successfully created
GridDisk data_CD_disk07_cell1 successfully created
GridDisk data_CD_disk08_cell1 successfully created
GridDisk data_CD_disk09_cell1 successfully created
GridDisk data_CD_disk10_cell1 successfully created
GridDisk data_CD_disk11_cell1 successfully created
GridDisk data_CD_disk12_cell1 successfully created
CellCLI> create flashlog all
CELL-02722: Flash log already exists.
CellCLI> list flashcache detail
name: cell1_FLASHCACHE
cellDisk: FD_01_cell1,FD_02_cell1,FD_00_cell1,FD_03_cell1
creationTime: 2015-01-20T22:15:22+08:00
degradedCelldisks:
effectiveCacheSize: 3.1875G
id: a97a7329-9169-4534-aac4-c28fac01a819
size: 3.1875G
status: normal
CellCLI> list flashlog detail
name: cell1_FLASHLOG
cellDisk: FD_02_cell1,FD_01_cell1,FD_00_cell1,FD_03_cell1
creationTime: 2015-01-20T22:15:20+08:00
degradedCelldisks:
effectiveSize: 512M
efficiency: 100.0
id: f4957f27-d1b6-463e-8d54-b1497cf90305
size: 512M
status: normal
CellCLI> list cell
cell1 online
CellCLI> list celldisk
CD_disk01_cell1 normal
CD_disk02_cell1 normal
CD_disk03_cell1 normal
CD_disk04_cell1 normal
CD_disk05_cell1 normal
CD_disk06_cell1 normal
CD_disk07_cell1 normal
CD_disk08_cell1 normal
CD_disk09_cell1 normal
CD_disk10_cell1 normal
CD_disk11_cell1 normal
CD_disk12_cell1 normal
FD_00_cell1 normal
FD_01_cell1 normal
FD_02_cell1 normal
FD_03_cell1 normal
CellCLI> list griddisk
data_CD_disk01_cell1 active
data_CD_disk02_cell1 active
data_CD_disk03_cell1 active
data_CD_disk04_cell1 active
data_CD_disk05_cell1 active
data_CD_disk06_cell1 active
data_CD_disk07_cell1 active
data_CD_disk08_cell1 active
data_CD_disk09_cell1 active
data_CD_disk10_cell1 active
data_CD_disk11_cell1 active
data_CD_disk12_cell1 active
CellCLI> list celldisk CD_disk01_cell1 detail
name: CD_disk01_cell1
comment:
creationTime: 2015-01-20T22:52:22+08:00
deviceName: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
devicePartition: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
diskType: HardDisk
errorCount: 0
freeSpace: 0
id: 572530b2-bed4-432d-8d46-093990067df8
interleaving: none
lun: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
physicalDisk: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/disk01
raidLevel: "RAID 0"
size: 992M
status: normal
CellCLI> list celldisk FD_00_cell1 detail
name: FD_00_cell1
comment:
creationTime: 2015-01-20T22:15:19+08:00
deviceName: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
devicePartition: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
diskType: FlashDisk
errorCount: 0
freeSpace: 0
id: d6346116-54fb-4501-bb44-dce5a8ad19c6
interleaving: none
lun: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
physicalDisk: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/disks/raw/FLASH01
raidLevel: "RAID 0"
size: 992M
status: normal
CellCLI> list griddisk data_CD_disk01_cell1 detail
name: data_CD_disk01_cell1
availableTo:
cachingPolicy: default
cellDisk: CD_disk01_cell1
comment:
creationTime: 2015-01-20T22:54:39+08:00
diskType: HardDisk
errorCount: 0
id: 5cfecc67-f540-41e1-bcab-36a64303abc2
offset: 48M
size: 944M
status: active
CellCLI>
cell节点创建完成！！！
DB节点：（参考rac安装博文http://blog.itpub.net/29500582/viewspace-1341453/）
1. 安装oracle数据库的基本步骤：
1.1 yum包安装
yum -y install binutils
yum -y install compat-libstdc++-33
yum -y install elfutils-libelf
yum -y install glibc
yum -y install glibc-common
yum -y install ksh
yum -y install libaio
yum -y install libgcc
yum -y install libstdc++
yum -y install make
yum -y install elfutils-libelf-devel
yum -y install glibc-headers
yum -y install glibc-devel
yum -y install gcc
yum -y install libstdc++-devel
yum -y install gcc-c++
yum -y install libaio-devel
yum -y install sysstat
yum -y install unixODBC
yum -y install unixODBC-devel
2. 系统初始化
2.1 配置系统参数
vi /etc/sysctl.conf
添加
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
kernel.shmmni = 4096
kernel.sem = 310 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
生效
sysctl -p
2.2 创建用户
groupadd -g 500 oinstall
groupadd -g 501 dba
groupadd -g 502 oper
groupadd -g 503 asmdba
groupadd -g 504 asmadmin
groupadd -g 505 asmoper
useradd -u 500 -g oinstall -G dba,oper,asmdba oracle
useradd -u 501 -g oinstall -G dba,oper,asmdba,asmadmin,asmoper grid
passwd oracle
passwd grid
[root@rac1 tmp]# id oracle
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba),502(oper),503(asmdba)
[root@rac1 tmp]# id grid
uid=501(grid) gid=500(oinstall) groups=500(oinstall),501(dba),502(oper),503(asmdba),504(asmadmin),505(asmoper)
2.3 创建密码不过期
chage -M 99999 oracle
chage -M 99999 grid
2.4 创建目录给权限
mkdir -p /app/grid
mkdir -p /app/oracle
mkdir -p /app/oraInventory
mkdir -p /app/soft
mkdir -p /app/soft/grid
mkdir -p /app/soft/oracle
mkdir -p /app/soft/asmlib
mkdir -p /backup
mkdir -p /arch
chown -R oracle:oinstall /app
chown -R oracle:oinstall /arch
chown -R oracle:oinstall /backup
chown -R grid:oinstall /app/grid
chown -R oracle:oinstall /app/oracle
chown -R grid:oinstall /app/oraInventory
chown -R grid:oinstall /app/soft/grid
chown -R oracle:oinstall /app/soft/oracle
chmod -R 755 /app
2.5 修改Oracle用户shell权限
vi /etc/security/limits.conf
添加：
* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536
2.6 修改配置文件vi /etc/profile
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
2.7 关闭selinux
vi /etc/sysconfig/selinux
#modify by lcs for oracle
#old SELINUX=enforcing
SELINUX=disabled
2.8 关闭防火墙
service iptables stop
chkconfig iptables off
2.9 用户限制，登录
vi /etc/pam.d/login
添加：
session required pam_limits.so
3. 创建两个Exadata所需的配置文件。Exadata所需要的配置文件有两个一个是cellinit.ora，此文件中填入DB节点的ip地址和子网掩码。
另外一个是cellip.ora, 此文件填入Cell节点的ip地址
[root@exadata ~]# mkdir -p /etc/oracle/cell/network-config
[root@exadata ~]# chown -R grid:oinstall /etc/oracle/cell/network-config
[root@exadata ~]# chmod -R 775 /etc/oracle/cell/network-config
[root@exadata ~]# vi /etc/oracle/cell/network-config/cellinit.ora
ipaddress1=192.168.9.246/24
~
~
~
"/etc/oracle/cell/network-config/cellinit.ora" [New] 1L, 28C written
[root@exadata ~]# vi /etc/oracle/cell/network-config/cellip.ora
cell="192.168.9.245"
~
~
~
~
"/etc/oracle/cell/network-config/cellip.ora" [New] 1L, 21C written
[root@exadata ~]#
4. 安装asm管理
下载地址：http://www.oracle.com/technetwork/server-storage/linux/downloads/rhel5-084877.html
下载时注意版本内核
[root@rac1 yum.repos.d]# uname -a
Linux rac1 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@rac1 yum.repos.d]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
[root@rac1 yum.repos.d]#
内核参数： 2.6.18-164.el5
[root@rac1 oracleasm]# cd /tmp/oracleasm/
[root@rac1 oracleasm]# ll
total 536
-rw-r--r-- 1 root root 136910 Nov 19 13:51 oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 138539 Nov 19 13:51 oracleasm-2.6.18-164.el5debug-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 2807 Nov 19 13:51 oracleasm-2.6.18-164.el5-debuginfo-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 136839 Nov 19 13:51 oracleasm-2.6.18-164.el5xen-2.0.5-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 14176 Nov 19 13:51 oracleasmlib-2.0.4-1.el5.x86_64.rpm
-rw-r--r-- 1 root root 90225 Nov 19 13:51 oracleasm-support-2.1.8-1.el5.x86_64.rpm
直接rpm包的安装
rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm
rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
初始化 asmlib，在所有节点执行；
[root@rac1 ~]# /usr/sbin/oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@rac1 ~]#
ASMlib 加载到内核，在所有节点执行；
[root@rac1 ~]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@rac1 ~]#
5. 配置oracle用户，grid用户环境变量
6. 安装grid软件
先配一些普通的裸盘（raw disk）做asm
这些以o/开头的ASM磁盘就是Exadata Cell端创建的griddisk。
当然在安装的时候可以选择普通的裸盘（raw disk）而非griddisk作为磁盘组，
到Grid Infrastructure安装完毕以后通过asmca来添加/新建新的磁盘组
7. 安装oracle软件
8. 安装oracle数据库
9. 配监听等等
10. 简单验证操作
[grid@exadata ~]:+ASM> asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 4194304 10236 10156 0 10156 0 N BACK/
MOUNTED EXTERN N 512 4096 4194304 944 864 0 864 0 N CELL1/
MOUNTED EXTERN N 512 4096 4194304 8496 6460 0 6460 0 N CELLDATA/
MOUNTED EXTERN N 512 4096 1048576 15358 15299 0 15299 0 N DATA/
MOUNTED NORMAL N 512 4096 4194304 10224 10064 0 5032 0 N OCR/
ASMCMD> lsattr -l -G data
Name Value
access_control.enabled FALSE
access_control.umask 066
au_size 1048576
cell.smart_scan_capable FALSE
compatible.asm 11.2.0.0.0
compatible.rdbms 10.1.0.0.0
disk_repair_time 3.6h
sector_size 512
ASMCMD> lsdsk -G data
Path
ORCL:DATA_VOL1
ASMCMD> lsdsk -G celldata
Path
o/192.168.9.245/data_CD_disk02_cell1
o/192.168.9.245/data_CD_disk03_cell1
o/192.168.9.245/data_CD_disk04_cell1
o/192.168.9.245/data_CD_disk05_cell1
o/192.168.9.245/data_CD_disk06_cell1
o/192.168.9.245/data_CD_disk07_cell1
o/192.168.9.245/data_CD_disk08_cell1
o/192.168.9.245/data_CD_disk09_cell1
o/192.168.9.245/data_CD_disk10_cell1
ASMCMD>
SQL> select disk_number, name, path from v$asm_disk where path like 'o/%' order by disk_number;
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
0 DATA_CD_DISK01_CELL1
o/192.168.9.245/data_CD_disk01_cell1
0 DATA_CD_DISK02_CELL1
o/192.168.9.245/data_CD_disk02_cell1
1 DATA_CD_DISK03_CELL1
o/192.168.9.245/data_CD_disk03_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
2 DATA_CD_DISK04_CELL1
o/192.168.9.245/data_CD_disk04_cell1
3 DATA_CD_DISK05_CELL1
o/192.168.9.245/data_CD_disk05_cell1
4 DATA_CD_DISK06_CELL1
o/192.168.9.245/data_CD_disk06_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
5 DATA_CD_DISK07_CELL1
o/192.168.9.245/data_CD_disk07_cell1
6 DATA_CD_DISK08_CELL1
o/192.168.9.245/data_CD_disk08_cell1
7
o/192.168.9.245/data_CD_disk12_cell1
DISK_NUMBER NAME
----------- ------------------------------
PATH
--------------------------------------------------------------------------------
7 DATA_CD_DISK09_CELL1
o/192.168.9.245/data_CD_disk09_cell1
8 DATA_CD_DISK10_CELL1
o/192.168.9.245/data_CD_disk10_cell1
10
o/192.168.9.245/data_CD_disk11_cell1
12 rows selected.
SQL>
SQL> select name, path from v$asm_disk;
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
BACK_VOL1
ORCL:BACK_VOL1
DATA_VOL1
ORCL:DATA_VOL1
OCR_VOL1
ORCL:OCR_VOL1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
OCR_VOL2
ORCL:OCR_VOL2
DATA_CD_DISK04_CELL1
o/192.168.9.245/data_CD_disk04_cell1
DATA_CD_DISK06_CELL1
o/192.168.9.245/data_CD_disk06_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK05_CELL1
o/192.168.9.245/data_CD_disk05_cell1
DATA_CD_DISK03_CELL1
o/192.168.9.245/data_CD_disk03_cell1
DATA_CD_DISK07_CELL1
o/192.168.9.245/data_CD_disk07_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK09_CELL1
o/192.168.9.245/data_CD_disk09_cell1
DATA_CD_DISK08_CELL1
o/192.168.9.245/data_CD_disk08_cell1
DATA_CD_DISK02_CELL1
o/192.168.9.245/data_CD_disk02_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
DATA_CD_DISK01_CELL1
o/192.168.9.245/data_CD_disk01_cell1
DATA_CD_DISK10_CELL1
o/192.168.9.245/data_CD_disk10_cell1
o/192.168.9.245/data_CD_disk11_cell1
NAME
------------------------------
PATH
--------------------------------------------------------------------------------
o/192.168.9.245/data_CD_disk12_cell1
16 rows selected.
SQL>
SQL> select file#, name from v$datafile;
FILE#
----------
NAME
--------------------------------------------------------------------------------
1
+CELLDATA/exadata/datafile/system.261.869661803
2
+CELLDATA/exadata/datafile/sysaux.262.869661811
3
+CELLDATA/exadata/datafile/undotbs1.263.869661817
FILE#
----------
NAME
--------------------------------------------------------------------------------
4
+CELLDATA/exadata/datafile/users.265.869661827
SQL>
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/29500582/viewspace-1410794/ ，如需转载，请注明出处，否则将追究法律责任。
请登录后发表评论
登录
全部评论
