Purpose

In this article we will demonstrate how to set up the Openfiler based server as an iSCSI storage device. We will then show how to use that iSCSI storage with an Oracle 10gR2 database with Real Application Clusters option.

Openfiler (http://www.openfiler.com) is a powerful, intuitive browser-based network storage software distribution. Openfiler delivers file-based Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. Openfiler sits atop of CentOS Linux and it is distributed as a stand-alone Linux distribution. The entire software stack interfaces with third-party software that is all open source.

File-based networking protocols supported by Openfiler include: NFS, SMB/CIFS, HTTP/WebDAV and FTP. Network directories supported by Openfiler include NIS, LDAP (with support for SMB/CIFS encrypted passwords), Active Directory (in native and mixed modes) and Hesiod. Authentication protocols include Kerberos 5. Openfiler includes support for volume-based partitioning, iSCSI (target and initiator), scheduled snapshots, resource quota, and a single unified interface for share management.

SCSI (and iSCSI) uses a client-server architecture. A 'client' is an initiator - it initiates requests. A 'server' is a target - it has something you want and answers requests from the initiators. Initiators come in two varieties: software and hardware. A software initiator is just a driver that handles all requests and pairs the network interfaces driver and the SCSI drivers together to make it all work. Using a software initiator any system with an Ethernet card can act as an iSCSI initiator. (From Ben Rockwood's 'A Quick Guide to iSCSI on Linux', available at http://www.cuddletech.com/articles/iscsi/index.html).

Openfiler builds on the existing iSCSI and logical volume management capabilities of the Linux OS. It provides easy setup and management via browser based interface, and stable services on the server end.

Hardware setup

For the Openfiler server (named pixel3) we used Dell DX270 system, with the following specifications:
- Intel Pentium 4 CPU (2.6GHz) with HT disabled in the BIOS
- 1GB RAM
- 80GB internal IDE hard drive
- Built in 1Gb Ethernet card

For the storage, we used two external Maxtor OneTouch II, 250GB drives, connected to the Openfiler server via USB 2.0 interface. We also had one partition on the internal IDE hard drive, left over after the Openfiler installation.

For the Oracle 10gR2 database servers (named pixel1 and pixel2), we used two Dell DX280 machines, each with the following configuration:
- Intel Pentium 4 CPU (3GHz) with HT enabled in the BIOS.
- 2GB RAM
- 80GB internal SATA hard drive
- Built in 1Gb Ethernet card (for the private network - to connect to the Openfiler server and for the cluster interconnect traffic)
- Add on 100Mb Ethernet PCI card (to connect to the public network)

Linux Red Hat 4 update 2 was installed on these two servers and they were setup in a cluster configuration, to run Oracle 10gR2 database with Real Application Clusters option. Database servers were connected to the public network via 100Mb Ethernet cards.

Private network was built around D-Link DGS-1008D, 8 port Gigabit switch, to which Openfiler server (pixel3) and database servers (pixel1 and pixel2) were connected via their 1Gb Ethernet cards. The same private network was used for the Oracle 10gR2 RAC interconnect traffic.

Note that in our setup, the Openfiler server was not connected to the public network.

Scope and Application

Once setup, Openfiler iSCSI based storage can be used like any other (local) hard drives, and with any Oracle database version. In this article we chose to use Oracle database 10gR2 with RAC option.

The main purpose of this article is to show how to setup iSCSI with the Openfiler. While this article does not talk about particulars of Oracle database installation, we do demonstrate how to use the storage with some of the Oracle components - namely Oracle Cluster Ready Services (CRS) and Oracle Automatic Storage Management (ASM).

This article is provided for educational purposes, so the setup is kept simple to demonstrate ideas and concepts. For example, the disk mirroring was setup on one physical disk only, while in practice that should be done on at least two physical drives.

Using Openfiler iSCSI with an Oracle RAC database on Linux

1. Openfiler server

Openfiler 2.0 beta 1 (openfiler-2.0.beta1-i386.iso from http://www.openfiler.com/download) was installed with all default options. We only had to specify the local network relevant information (time zone, server name, and IP address). For the install instructions please consult the Openfiler install guide available at http://www.openfiler.com/docs/install/graphical_install.html.

Once the install completed, server was rebooted to make sure all required components, services and drivers were started and recognised. After the reboot, external Maxtor hard drives were discovered by Openfiler server as /dev/sda and /dev/sdb.

We have also ran 'yum upate' to get the latest Openfiler version and packages. To run yum from behind the company firewall, we had to set HTTP_PROXY environment variable and then run yum:

At the end of the yum update, Openfiler was patched to version 2.0-beta2.961.1, and the Linux kernel was updated to version 2.6.9-34.EL.

1.1. Administration

Openfiler administration is performed via Openfiler Storage Control Center, a browser based tool, over https connection on port 446. Admin user name is 'openfiler' with the initial password of 'password':

To use the Openfiler as the storage server, we have to set up services, network access and the physical storage. Other components (user accounts, quotas etc) are not mandatory and can be setup as required.

1.2. Services

To control services, we login to the Openfiler Storage Control Center (in our case via https://pixel3:446), with user name openfiler. That takes us to the Home/Administration section. We click on the Services tab, that shows the 'Enable/Disable services' option:

We are only interested in iSCSI service, so to enable it we just click on the 'Enable' under the 'iSCSI target' service name. After that, 'iSCSI target' status should change to 'Enabled'.

1.3. Physical storage

Storage devices - internal IDE/SATA/SCSI disks, external USB or firewire drives, or any other storage can be connected to the Openfiler server, and served to the clients. Once these devices are discovered at the OS level, Openfiler Storage Control Center can be used to set up and manage all that storage.

In our case, we had an internal 60GB disk (partition) and two external USB drives, for our storage needs. On the Openfiler server these drives were seen as /dev/hda (internal IDE drive), /dev/sda (external Maxtor drive) and /dev/sdb (second external Maxtor drive).

Physical storage is managed via Volumes/Physical Storage Mgmt tab, in the Openfiler Storage Control Center. In our case that showed:

1.3.1. Partitioning the physical disks

We will show how to setup the iSCSI storage on one of the external Maxtor hard drives, and the process would be the same for any other disks available via the Physical Storage Management screen.

By clicking on /dev/sda we are presented with the options to Edit or Create a partition:

To create a partition, we specify/select:

Mode: Primary
Partition Type: Physical Volume
Starting Cylinder: 1
Ending Cylinder: 8000

The size now shows 61.28GB. To accept that we click on the Create button. This results in a new partition on our external hard drive:

One partition is all we need for now, and more partitions can be created later as per our storage requirements.

1.3.2 Volume group management

To create a volume group, in this newly created partition, we go to the Volumes/Volume Group Mgmt tab. There we would see any existing volume groups, or none as in our case:

We name the volume group ("data1" sounds like a good name), tick the box in front of /dev/sda1, to select that partition, and click on the 'Add volume group' button.

After that we are presented with the list, that now shows our newly created volume group data1:

Note that the volume group can be deleted ('Delete VG' column) because it does not contain any logical volumes.

1.3.3. Logical volumes

We can now create logical volumes in this new volume group (data1), via 'Create New Volume' tab.

From the 'Select Volume Group' list, we choose 'data1' and click on the 'Change' button. That presents the information about available space in the volume group 'data1':

We now create three logical volumes - ocr (1GB), vote (1GB) and asm, that takes the rest of the available space (60GB). For each logical volume we specify its name, a short description, required size and iSCSI for the filesystem type.

After that, we can go to the 'List of Existing Volumes' tab, and select 'data1'. That presents us the pie chart diagram of the data1 volume group, and the table with all the information about our logical volumes:

In effect we have created three iSCSI disks, that can now be presented to iSCSI clients (pixel1 and pixel2) on the network.

1.3.4. ietd - iSCSI Enterprise Target Daemon

The ietd program implements the user level part of iSCSI Enterprise Target software for building an iSCSI storage system on Linux. With the iSCSI target service enabled, we should see the ietd daemon running on the Openfiler server:

1.3.5. /etc/ietd.conf - ietd configuration file

This file gets created by and is maintained by Openfiler. Initially the file is empty, and entries get added as iSCSI targets are configured. This was the content of /etc/ietd.conf, after our iSCSI devices were configured:

We should now edit /etc/ietd.conf file, to add SCSI IDs, that can later be used to uniquely identify iSCSI disks on the clients. SCSI IDs are added to the Lun lines, and can be any string of up to 16 characters. In our case we chose to use volume names for SCSI IDs. After these changes, the file /etc/ietd.conf was:

1.3.6. Make iSCSI targets available to clients

Every time we add logical volumes, we need to restart the associated service on the Openfiler server. In our case we created iSCSI logical volumes, so we have to restart iscsi-target service. That makes new iSCSI targets available to all clients on the network:

The same (iscsi-target service restart) can be achieved via Services tab in the Openfiler Storage Control Center. We would first disable iSCSI target service, then enable it again.

2. iSCSI clients

An iSCSI client can be any system (Linux, Unix, MS Window$, Apple Mac, etc) for which iSCSI support (driver) is available. In our case, clients were two Linux servers (pixel1 and pixel2), running Red Hat 4, as mentioned earlier in the 'Hardware setup' section.

2.1. iSCSI (initiator) service

On each client machine, we have to make sure iSCSI (initiator) service is up and running. If not installed as part of the OS setup, iscsi-initiator-utils rpm should be downloaded from one of the Internet rpm resources, and installed on the clients.

The only configuration required on the clients is to specify the iSCSI server in the /etc/iscsi.conf file. The only required option is DiscoveryAddress, for the name of the Openfiler storage server. In our case that was:

DiscoveryAddress=pixel3

After that we can start the iscsi initiator service:

pixel1 # service iscsi restart
Stopping iscsid: [ OK ]
Removing iscsi driver: [ OK ]
Checking iscsi config: [ OK ]
Loading iscsi driver: [ OK ]
Starting iscsid: [ OK ]

We should also configure the service to be active across machine reboots. Linux command chkconfig can be used to achieve that, as follows:

pixel1 # chkconfig --level 345 iscsi on

2.2. Discovering iSCSI targets

When Openfiler server publishes available iSCSI targets (that happens when iscsi-target service get started/restarted on the Openfiler server), or when the iscsi initiator service is started/restarted on the client, configured clients get the message that new iSCSI disks are now available. We would see something like this in the client's /var/log/messages file:

May 24 16:41:25 pixel1 iscsid[6071]: version 4:0.1.11 variant (12-Jan-2005)
May 24 16:41:25 pixel1 iscsi: iscsid startup succeeded
May 24 16:41:25 pixel1 iscsid[6075]: Connected to Discovery Address 10.10.0.30
May 24 16:41:25 pixel1 kernel: iscsi-sfnet:host8: Session established
May 24 16:41:25 pixel1 kernel: scsi8 : SFNet iSCSI driver
May 24 16:41:25 pixel1 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0
May 24 16:41:25 pixel1 kernel: Type: Direct-Access ANSI SCSI revision: 04
May 24 16:41:25 pixel1 kernel: SCSI device sdb: 124256256 512-byte hdwr sectors (63619 MB)
May 24 16:41:25 pixel1 kernel: SCSI device sdb: drive cache: write back
May 24 16:41:25 pixel1 kernel: iscsi-sfnet:host9: Session established
May 24 16:41:25 pixel1 kernel: sdb:<6>scsi9 : SFNet iSCSI driver
May 24 16:41:25 pixel1 kernel: unknown partition table
May 24 16:41:25 pixel1 kernel: Attached scsi disk sdb at scsi8, channel 0, id 0, lun 0
May 24 16:41:25 pixel1 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0
May 24 16:41:25 pixel1 kernel: Type: Direct-Access ANSI SCSI revision: 04
May 24 16:41:25 pixel1 kernel: iscsi-sfnet:host10: Session established
May 24 16:41:25 pixel1 kernel: scsi10 : SFNet iSCSI driver
May 24 16:41:25 pixel1 kernel: Vendor: IET Model: VIRTUAL-DISK Rev: 0
May 24 16:41:25 pixel1 kernel: Type: Direct-Access ANSI SCSI revision: 04
May 24 16:41:25 pixel1 kernel: SCSI device sdc: 2097152 512-byte hdwr sectors (1074 MB)
May 24 16:41:25 pixel1 scsi.agent[6126]: disk at /devices/platform/host8/target8:0:0/8:0:0:0
May 24 16:41:25 pixel1 kernel: SCSI device sdc: drive cache: write back
May 24 16:41:25 pixel1 kernel: sdc: unknown partition table
May 24 16:41:25 pixel1 kernel: Attached scsi disk sdc at scsi9, channel 0, id 0, lun 0
May 24 16:41:25 pixel1 kernel: SCSI device sdd: 2097152 512-byte hdwr sectors (1074 MB)
May 24 16:41:25 pixel1 kernel: SCSI device sdd: drive cache: write back
May 24 16:41:25 pixel1 kernel: sdd: unknown partition table
May 24 16:41:25 pixel1 kernel: Attached scsi disk sdd at scsi10, channel 0, id 0, lun 0
May 24 16:41:25 pixel1 scsi.agent[6187]: disk at /devices/platform/host10/target10:0:0/10:0:0:0
May 24 16:41:25 pixel1 scsi.agent[6182]: disk at /devices/platform/host9/target9:0:0/9:0:0:0
...

The above entries show that the client (pixel1) was able to establish an iSCSI sessions with the iSCSI storage server (pixel3 at 10.10.0.30)

We also see how iSCSI targets' host IDs and LUNs associate to local SCSI device names:

Attached scsi disk sdb at scsi8, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi9, channel 0, id 0, lun 0
Attached scsi disk sdd at scsi10, channel 0, id 0, lun 0

To work out mapping of iSCSI targets to the local SCSI device names (on pixel1), we run iscsi-ls command on the client:

pixel1 # iscsi-ls
*******************************************************************************
SFNet iSCSI Driver Version ...4:0.1.11(12-Jan-2005)
*******************************************************************************
TARGET NAME : iqn.2006-01.com.openfiler:data1.ocr
TARGET ALIAS :
HOST ID : 10
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 10.10.0.30:3260,1
SESSION STATUS : ESTABLISHED AT Wed May 24 16:41:26 EST 2006
SESSION ID : ISID 00023d000001 TSIH 300
*******************************************************************************
TARGET NAME : iqn.2006-01.com.openfiler:data1.asm
TARGET ALIAS :
HOST ID : 8
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 10.10.0.30:3260,1
SESSION STATUS : ESTABLISHED AT Wed May 24 16:41:26 EST 2006
SESSION ID : ISID 00023d000001 TSIH 100
*******************************************************************************
TARGET NAME : iqn.2006-01.com.openfiler:data1.vote
TARGET ALIAS :
HOST ID : 9
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 10.10.0.30:3260,1
SESSION STATUS : ESTABLISHED AT Wed May 24 16:41:26 EST 2006
SESSION ID : ISID 00023d000001 TSIH 200
*******************************************************************************

So we see that iSCSI target iqn.2006-01.com.openfiler:data1.asm has the host id 8, which maps to /dev/sdb on pixel1. Similarly, we see that iqn.2006-01.com.openfiler:data1.vote maps to /dev/sdc, and that iqn.2006-01.com.openfiler:data1.ocr maps to /dev/sdd.

While the above may seem rather complex, it needs to be done only the very first time an iSCSI target is made available to a client(s).

2.3. iSCSI disks partitioning

Available iSCSI disks can now be used on client systems like any other (local) SCSI disks. They can be partitioned, used for local (e.g. ext3), shared (e.g. smb) or cluster (e.g. OCFS2) file systems, as raw devices, for ASM based storage etc.

In our case, we wanted to use data1.ocr for Oracle Cluster Registry (component of Oracle 10g CRS stack), data1.vote for shared voting disk (again a CRS stack component) and data1.asm for the ASM based database storage.

We used fdisk to partition /dev/sdb (asm), /dev/sdc (vote) and /dev/sdd (ocr) as follows.

For the ASM based storage we created ten 6GB partitions:

pixel1 # fdisk -l /dev/sdb

Disk /dev/sdb: 63.6 GB, 63619203072 bytes
64 heads, 32 sectors/track, 60672 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 6068 6213616 83 Linux
/dev/sdb2 6069 12136 6213632 83 Linux
/dev/sdb3 12137 18204 6213632 83 Linux
/dev/sdb4 18205 60672 43487232 5 Extended
/dev/sdb5 18205 24272 6213616 83 Linux
/dev/sdb6 24273 30340 6213616 83 Linux
/dev/sdb7 30341 36408 6213616 83 Linux
/dev/sdb8 36409 42476 6213616 83 Linux
/dev/sdb9 42477 48544 6213616 83 Linux
/dev/sdb10 48545 54612 6213616 83 Linux
/dev/sdb11 54613 60672 6205424 83 Linux

For the voting disks, we created three 340MB partitions:

pixel1 # fdisk -l /dev/sdc

Disk /dev/sdc: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 337 349438+ 83 Linux
/dev/sdc2 338 674 349469 83 Linux
/dev/sdc3 675 1011 349469 83 Linux

For the OCR we created two 500MB partitions:

pixel1 # fdisk -l /dev/sdd

Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 500 518469+ 83 Linux
/dev/sdd2 501 1011 529907 83 Linux

Note that the partitioning needs to be performed from one client machine only (e.g. from pixel1), and will be visible on both pixel1 and pixel2 as this is the shared storage.

2.4. Consistent/persistent storage device mapping solutions

Device mapping at boot time need to be persistent accross reboots. It further need to be the same on all RAC nodes.

2.4.1 udev option

udev utility can be used for disk mounting consistency between the nodes. It is the preferred tool.
Note:371814.1 explains how to use the udev option.

2.4.2 devlabel - using symlinking

devlabel is a feature that exist on some Linux system, but has been depreciated in n EL4 & EL5. However, when it exists, it can be used the following way:

To ensure consistent iSCSI device to SCSI disk mappings across reboots, we should label SCSI disks (or disk partitions) on the client machines. Unlike partitioning, this needs to be performed on each client, as iSCSI devices can map to different SCSI disk names on each client machine. If not installed on the client machines, devlabel can be obtained from one of the Internet rpm resources or from http://linux.dell.com/.

Since we will be using disk partitions, we will label the partitions of interest (/dev/sdcn and /dev/sddn). First we check that our partitions do return valid labels:

# devlabel printid -d /dev/sdc1
S83.1:4f70656e66696c006461746131766f7465000000000000OpenfileVirtualdisksector61

Note that our SCSI ID string "data1vote" is seen by devlabel (hex value 6461746131766f7465 in the middle of the id) and it now uniquely identifies this disk/partition.

We can now label the partitions:

pixel1 # devlabel add -d /dev/sdc1 -s /dev/vote1
SYMLINK: /dev/vote1 -> /dev/sdc1
Added /dev/vote1 to /etc/sysconfig/devlabel
...

pixel1 # devlabel add -d /dev/sdc11 -s /dev/asm10
SYMLINK: /dev/asm10 -> /dev/sdc11
Added /dev/asm10 to /etc/sysconfig/devlabel

After device labeling the content of /etc/sysconfig/devlabel was:

# devlabel configuration file
#
# This file should generally not be edited by hand.
# Instead, use the /sbin/devlabel program to make changes.
# devlabel by Gary Lerhaupt gary_lerhaupt@dell.com
#
# format:
# or format:

/dev/vote1 /dev/sdc1 S83.1:4f70656e66696c006461746131766f7465310000000000OpenfileVirtualdisksector61
/dev/vote2 /dev/sdc2 S83.1:4f70656e66696c006461746131766f7465320000000000OpenfileVirtualdisksector1555508
...
/dev/asm10 /dev/sdb11 S83.1:4f70656e66696c00646174613161736d31300000000000OpenfileVirtualdisksector1037005

It should be noted that the above symbolic names were entirely arbitrary, and that we could have used any names we wanted.

While devlabel should be set up on all clients, we don't have to work out iSCSI device mappings to SCSI disk names on other clients. We can just copy /etc/sysconfig/devlabel file from the first client to all other clients, and restart the devlabel. That would create the symbolic links to the correct SCSI disk names on the client machines.

As we have already setup devlabel on pixel1, we will make use of devlabel restart facility on pixel2.

We first copy /etc/sysconfig/devlabel from pixel1 to pixel2:

pixel2 # cd /etc/sysconfig
pixel2 # scp pixel1:/etc/sysconfig/devlabel .

And restart the devlabel:

pixel2 # devlabel restart
SYMLINK: /dev/vote1 -> /dev/sdc1
SYMLINK: /dev/vote2 -> /dev/sdc2
SYMLINK: /dev/vote3 -> /dev/sdc3
SYMLINK: /dev/ocr1 -> /dev/sdd1
...

If the disk mapping were to change on reboot, devlabel would fix that automatically:

Device name inconsistency detected for symlink /dev/ocr1!
The device /dev/sdd1 is now /dev/sde1.
The symlink /dev/ocr1 will now point to the new device name.
SYMLINK: /dev/ocr1 -> /dev/sde1

We should put 'devlabel restart' in one of the startup scripts (e.g. /etc/rc.local) to make sure it runs on system reboot.

3. Using iSCSI storage with an Oracle database

Lets see how we can use the configured iSCSI storage for Oracle Cluster Ready Services and Oracle 10gR2 database. While the installation of Oracle Cluster Ready Services, Oracle database and Oracle Automatic Storage Management is outside the scope of this article, we will demonstrate how the iSCSI storage can be used for those components.

3.1. Oracle Cluster Ready Services

Oracle Cluster Ready Services (CRS) has two components that require shared disk storage - the Oracle Cluster Registry (OCR) and the Voting disk. In addition to that, the recommendation is to have at least two (mirror) copies of the OCR and at least three copies of the Voting disks.

The OCR and the Voting disks can be either on a supported shared file system (e.g. OCFS2), or implemented via raw devices. We will use shared iSCSI storage to create raw devices for the OCR and the Voting disks.

We have already partitioned /dev/sdd for OCR and /dev/sdc for the Voting disk, and created symbolic names using devlabel. We can now use those partitions to create raw devices. Note that this needs to be performed on all nodes in a cluster (in our case on both pixel1 and pixel2).

Bind raw devices to /dev/ocr? for the OCR and its mirror:

# /usr/bin/raw /dev/raw/raw11 /dev/ocr1
# /usr/bin/raw /dev/raw/raw12 /dev/ocr2

Bind raw devices to /dev/vote? for the voting disk and its mirrors:

# /usr/bin/raw /dev/raw/raw21 /dev/vote1
# /usr/bin/raw /dev/raw/raw22 /dev/vote2
# /usr/bin/raw /dev/raw/raw23 /dev/vote3

Red Hat 4 used udev to manage raw (and other) devices. To set the correct ownership and permissions on these raw devices, we have to update udev's permissions file /etc/udev/permissions.d/50-udev.permissions.

The following changes were made:

1. Original raw setup line

raw/*:root:disk:0660

was commented out, i.e. modified to:

#raw/*:root:disk:0660

2. Two lines were added.

One for the OCR raw devices:

raw/raw1[1-2]:root:dba:640

And another one for the Voting disks:

raw/raw2[1-3]:oracle:dba:640

The raw section of /etc/udev/permissions.d/50-udev.permissions was now:

It should be noted that we didn't have any other raw devices on the system. In case more raw devices are required, they would also need to be setup in this file.

Since we are relying on devlabel to create the symbolic links to /dev/ocrn and /dev/voten, we should put the above commands in the /etc/rc.local file, after the devlabel restart command:

# cat /etc/rc.local#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
# Persistent iSCSI based device names across reboots - for OCR, voting disk and ASM disks.
# See /etc/sysconfig/devlabel for mappings
devlabel restart

# OCR copy 1, /dev/ocr1 on Maxtor disk1
/usr/bin/raw /dev/raw/raw11 /dev/ocr1

# Mirror copy of OCR, /dev/ocr2, on Maxtor disk 2
/usr/bin/raw /dev/raw/raw12 /dev/ocr2

# Voting disk /dev/vote1, on Maxtor 1
/usr/bin/raw /dev/raw/raw21 /dev/vote1

# First mirror copy of the voting disk, /dev/vote2 also on Maxtor 1
/usr/bin/raw /dev/raw/raw22 /dev/vote2

# Second mirror copy of the voting disk, /dev/vote3 on Maxtor 2
/usr/bin/raw /dev/raw/raw23 /dev/vote3

# Hangcheck timer module
/sbin/insmod /lib/modules/2.6.9-22.ELsmp/kernel/drivers/char/hangcheck-timer.ko

3.2. Oracle Automatic Storage Management

Oracle Automatic Storage Management (ASM) is an integrated file system and volume manager expressly built for Oracle database files. Oracle files that can be stored in the ASM include datafiles, control files, redo log files, archive log files, Oracle Recovery Manager files etc.

3.2.1. Oracle asmlib

The Automatic Storage Management library driver (ASMlib) simplifies the configuration and management of the disk devices by eliminating the need to rebind raw devices used with ASM each time the system is restarted.

We downloaded the following ASMlib rpm packages from OTN (http://www.oracle.com/technology/software/tech/linux/asmlib/rhel4.html):

Library and Tools:

oracleasm-support-2.0.1-1.i386.rpm
oracleasmlib-2.0.1-1.i386.rpm

Driver for our kernel version (2.6.9-22.ELsmp):

oracleasm-2.6.9-22.ELsmp-2.0.0-1.i686.rpm

ASMlib packages need to be installed on both Oracle database servers (pixel1 and pixel2):

# rpm -Uvh oracleasm-support-2.0.1-1.i386.rpm oracleasmlib-2.0.1-1.i386.rpm oracleasm-2.6.9-22.ELsmp-2.0.0-1.i686.rpm
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [ 33%]
2:oracleasm-2.6.9-22.ELsm########################################### [ 67%]
3:oracleasmlib ########################################### [100%]

We also need to configure asmlib on both servers:

# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: [ OK ]
Scanning system for ASM disks: [ OK ]

If asmlib configure fails, that could be due to a Red Hat problem with SELinux. To get around that run /usr/bin/system-config-securitylevel, and disable SELinux option.

We can now create ASM disks, using /dev/asm* devices we configured earlier. Note that ASM disks need to created on one server only.

To do this we will make use of oracleasm createdisk command:

pixel1 # /etc/init.d/oracleasm createdisk DISK01 /dev/asm01
Marking disk "/dev/asm01" as an ASM disk: [ OK ]
...
pixel1 # /etc/init.d/oracleasm createdisk DISK10 /dev/asm10
Marking disk "/dev/asm10" as an ASM disk: [ OK ]

Check that all disks are recognized by asmlib:

pixel1 # /etc/init.d/oracleasm listdisks
DISK01
DISK02
...
DISK10

On the other cluster nodes (in our case the second server, pixel2), we merely need to scan for ASM disks:

pixel2 # /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]

After that we should be able to see our disks on the second node:

pixel2 # /etc/init.d/oracleasm listdisks
DISK01
DISK02
...
DISK10

3.2.2. Create an ASM disk group

When ASM is used for Oracle 10g database storage, data files are stored in an ASM disk group(s). To create an ASM disk group, using some or all available iSCSI disks, we configured earlier, we connect to an ASM instance, on either pixel1 or pixel2 server, and run the create diskgroup command:

create diskgroup DG_OPEN1
normal redundancy
failgroup F1 disk 'ORCL:DISK01', 'ORCL:DISK02'
failgroup F2 disk 'ORCL:DISK03', 'ORCL:DISK04'
;

This way we created the 12GB disk group DG_OPEN1 (each ASM disk was 6GB), with additional 12GB for the data mirroring (note the two fail groups).

3.2.3. Create a tablespace

Database can now use the diskgroup DG_OPEN1 for its tablespaces. We connect to the database instance, on either pixel1 or pixel2 server, and create the tablespace:

SQL> create tablespace TBS2 datafile '+DG_OPEN1';

Without specifying the size, this creates tablespace TBS2, with a single 100MB data file in the disk group DG_OPEN1:

SQL> select name, bytes/1024/1024 mb
from v$datafile;

NAME MB
------------------------------------------------ ----------
+GROUPA/v10/datafile/system.268.564600555 520
...
+DG_OPEN1/v10/datafile/tbs2.256.589462741 100

8 rows selected.

4. Open issues

Openfiler is a solid product with rich set of features. In this article we only explored the iSCSI functionality and we did not encounter any major problems.

4.1. Oracle Cluster Verification Utility (CVU) fails the shared storage check

Part of Oracle Cluster Ready Services setup, is the check for the configured shared storage. While the disks are visible and shared from our cluster nodes, the check itself fails:

$ runcluvfy.sh stage -post hwos -n pixel1,pixel2 -verbose
...

Checking shared storage accessibility...

WARNING: Unable to determine the sharedness of /dev/sdc on nodes:
pixel2,pixel1

Shared storage check failed on nodes "pixel2,pixel1".

This is because CVU calls smartctl on Linux, and the problem is that smartctl does not return the serial number from our iSCSI devices. For exampl, check against /dev/sdc shows:

# /usr/sbin/smartctl -i /dev/sdc
smartctl version 5.33 [i386-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: IET VIRTUAL-DISK Version: 0
Serial number:
Device type: disk
Local Time is: Tue May 30 11:33:35 2006 EST
Device supports SMART and is Disabled
Temperature Warning Disabled or Not Supported

I am yet to hear from the Openfiler developers, if there are any plans to fix this.

4.2. Minor bugs

This version of Openfiler was still in beta release and we have seen some minor issues.

4.2.1. Physical volume space

Physical volumes would not show released space after the volume group was dropped. We had to manually remove the volume group (using vgremove) on the Openfiler server.

4.2.2. Loosing iSCSI targets after Openfiler server reboot

After rebooting the Openfiler server, iSCSI clients were not able to see any targets. This was because logical volumes were not activated on reboot.

The problem was due to a timing issue with activating USB on server startup. Openfiler activates logical volumes on reboot via /etc/rc.sysinit script. But at the time call was made to activate the LVM, USB wasn't initialised and external USB drives were not visible to the kernel via /dev/sda and /dev/sdb.

The solution was to modify the /etc/rc.sysinit script and introduce a wait on devices, just before the lvm activation. Before the script was modified, the LVM part was:

After the change, the relevant part was:

References

NOTE:371814.1 - Using udev with Oracle Architecture (RAC & ASM) - Red Hat 4.0
[@more@]