Large SGA On Linux

On 32-bit Linux the total SGA size is limited to around 2Gig. There are several ways to allocate more memory than this, but the most robust and scalable method requires the use of a shared memory file, like shmfs in RHEL 2.1 or ramfs in RHEL 3-5. This article presents the methods necessary to create very large buffer caches on Red Hat Enterprise Linux and its clones.

• Red Hat Enterprise Linux (RHEL) 2.1
  • Create a shared memory file system (shmfs)
  • Enabling big pages
• Red Hat Enterprise Linux (RHEL) 3, 4 and 5
• Setting the SHMMAX value
• Instance Parameters

Red Hat Enterprise Linux (RHEL) 2.1

Create a shared memory file system (shmfs)

The shmfs is a memory file system so it can be as large as the maximum allowable virtual memory supported by Red Hat Linux AS2.1, currently 16 GB, although the enterprise kernel theoretically supports up to 64 GB of RAM.

The shmfs is created using the following command as the root user:

mount -t shm shmfs -o size=3g /dev/shm

The shared memory file system can be mounted automatically by adding the following line into /etc/fstab file:

shmfs /dev/shm shm size=3g 0 0

In the above example I've created the shmfs with a size of 3G as that is the size of the buffer cache I am planning to use. The other elements of the SGA are placed in regular memory, not this shared memory file system, so they should not be included when deciding on the size of the shmfs. It is advisable to size this slightly bigger than the actual size needed, but in this example I've used a 3G shmfs for a 3G buffer cache.

Enabling big pages

Big pages are enabled by adding the bigpages=xMB to the relevant kernel entry in the boot loader file /boot/grub/grub.conf file, where "x" is calculated as follows:

(Total SGA size in Gig) x 1024

Then round this value to the nearest hundredth. So for a 4G SGA we would do the following:

4 x 1024 = 4096 = 4100

So the /boot/grub/grub.conf file entry might look like this:

kernel /vmlinuz-2.4.9-e.40enterprise ro root=/dev/cciss/c0d0p2 bigpages=4100MB

With this entry saved the system should be rebooted. Once the system is available you must perform the following command as the root user:

echo 2 > /proc/sys/kernel/shm-use-bigpages

Alternatively you can add the following entry into the /etc/sysctl.conf file so this value persists between reboots:

kernel.shm-use-bigpages = 2

Red Hat Enterprise Linux (RHEL) 3, 4 and 5

Mount the ramfs filesystem and make sure it is usable by Oracle.

umount /dev/shm 
mount -t ramfs ramfs /dev/shm 
chown oracle:oinstall /dev/shm

Add these commands into the "/etc/rc.local" file. As with the RHEL 2.1 method, only the buffer cache is stored in this location.

For Oracle to lock shared memory for the shared pool, we must adjust the "memlock" parameter in the "/etc/security/limits.conf". The "memlock" figure is specified in Kb, so for a 2G shared pool we would need to set at least (2 * 1024 * 1024 = 2097152), as shown below.

oracle           soft    memlock         2097152
oracle           hard    memlock         2097152

In reality, it is better to oversize it a little.

Setting the SHMMAX value

For 32 bit architecture, the shmmax value should be set at half the physical memory up to a maximum of 4294967295. For a server with 6G of memory we can set this value to 3G (half physical memory) using the following command as the root user:

echo 3221225472 > /proc/sys/kernel/shmmax

Alternatively it can be set in the /etc/sysctl.conf file with the following entry:

kernel.shmmax = 3221225472

The contents of your /etc/sysctl.conf file may look something like this:

kernel.shmmax = 3221225472
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.sem = 1000 32000 100 150
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
kernel.shm-use-bigpages = 2

Alterations to the /etc/sysctl.conf file can be applied without a reboot by issuing the following command as root:

/sbin/sysctl -p

Instance Parameters

Some instance parameter changes are necessary to allow the Oracle instance to use the shared memory file system. The spfile parameters can be manipulated using the ALTER SYSTEM SET command in a running instance, or by modifying the spfile contents offline:

-- Change the parameter value in the spfile directly.
ALTER SYSTEM SET parameter = value SCOPE=spfile

-- Create a pfile with the contents of the current spfile.
CREATE PFILE='/tmp/pfile' FROM SPFILE;

-- Manually manipulate the contents of the pfile.

-- Recreate the spfile from the amended pfile.
CREATE SPFILE FROM PFILE='/tmp/pfile';

The following parameter should be added to the spfile or pfile:

use_indirect_data_buffers=true

Also, any references to db_cache_size and db_xK_cache_size parameters should be removed and replaced with the old style db_block_buffers parameter entry:

# 3Gig for an 8K db_block_size. ((3 * 1024 * 1024) / 8) = 393216
db_block_buffers = 393216

This means that the multiple block size feature is not available when using this method. Remember that the buffer cache is only one part of the SGA.[@more@]