Automatic Storage Manager (ASM) Enhancements in Oracle Database 11g Release 1

This article provides an overview of the main Automatic Storage Manager (ASM) enhancements in Oracle Database 11g Release 1, including:

• Oracle Database Storage Administrator's Guide
• New Disk Group Compatibility Attributes
• Fast Mirror Resync
• Rolling Upgrade
• SYSASM Privilege and OSASM OS Group
• Scalability and Performance Enhancements
• New ASMCMD Commands and Options
• Preferred Read Failure Groups
• Fast Rebalance
• Miscellaneous Disk Group Maintenance Enhancements

Related articles.

• Automatic Storage Management (ASM) in Oracle Database 10g
• ASM using ASMLib and Raw Devices
• Using NFS with ASM

Oracle Database Storage Administrator's Guide

The documentation for this release now includes a Oracle Database Storage Administrator's Guide for the first time.

New Disk Group Compatibility Attributes

Oracle 11g ASM includes two new compatibility attributes that determine the version of the ASM and database software that can use specific disk groups:

• COMPATIBLE.ASM - The minimum version of the ASM software that can access the disk group. In 11g, the default setting is 10.1.
• COMPATIBLE.RDBMS - The minimum COMPATIBLE database initialization parameter setting for any database instance that uses the disk group. In 11g, the default setting is 10.1.

The compatibility versions of a disk group can only be increased, not decreased. If you have increased the version by mistake, you will need to create a new disk group.

The disk group compatibility attributes can be set during disk group creation by adding the ATTRIBUTE clause to the CREATE DISKGROUP command.

CREATE DISKGROUP data DISK '/dev/raw/*'
  ATTRIBUTE 'compatible.asm' = '11.1';

CREATE DISKGROUP data DISK '/dev/raw/*' 
  ATTRIBUTE 'compatible.rdbms' = '11.1', 'compatible.asm' = '11.1';

The disk group compatibility attributes for existing disk groups can be altered using the SET ATTRIBUTE clause to the ALTER DISKGROUP command.

ALTER DISKGROUP data SET ATTRIBUTE 'compatible.asm' = '11.1';
ALTER DISKGROUP data SET ATTRIBUTE 'compatible.rdbms' = '11.1';

The current compatibility settings are available from the V$ASM_DISKGROUP and V$ASM_ATTRIBUTE views.

COLUMN name FORMAT A10
COLUMN compatibility FORMAT A20
COLUMN database_compatibility FORMAT A20

SELECT group_number, name, compatibility, database_compatibility FROM v$asm_diskgroup;

GROUP_NUMBER NAME       COMPATIBILITY        DATABASE_COMPATIBILI
------------ ---------- -------------------- --------------------
           1 DATA       11.1.0.0.0           11.1.0.0.0

1 row selected.

SQL>

COLUMN name FORMAT A20
COLUMN value FORMAT A20

SELECT group_number, name, value FROM v$asm_attribute ORDER BY group_number, name;

GROUP_NUMBER NAME                 VALUE
------------ -------------------- --------------------
           1 au_size              1048576
           1 compatible.asm       11.1.0.0.0
           1 compatible.rdbms     11.1
           1 disk_repair_time     3.6h

4 rows selected.

SQL>

Fast Mirror Resync

During transient disk failures within a failure group, ASM keeps track of the changed extents that need to be applied to the offline disk. Once the disk is available, only the changed extents are written to resynchronize the disk, rather than overwriting the contents of the entire disk. This can speed up the resynchronization process considerably.

Fast mirror resync is only available when the disk groups compatibility attributes are set to 11.1 or higher.

ALTER DISKGROUP disk_group_1 SET ATTRIBUTE 'compatible.asm' = '11.1';
ALTER DISKGROUP disk_group_1 SET ATTRIBUTE 'compatible.rdbms' = '11.1;

ASM drops disks if they remain offline for more than 3.6 hours. The disk groups default time limit is altered by changing the DISK_REPAIR_TIME parameter with a unit of minutes (M or m) or hours (H or h).

-- Set using the hours unit of time.
ALTER DISKGROUP disk_group_1 SET ATTRIBUTE 'disk_repair_time' = '4.5h';

-- Set using the minutes unit of time.
ALTER DISKGROUP disk_group_1 SET ATTRIBUTE 'disk_repair_time' = '300m';

The DROP AFTER clause of the ALTER DISKGROUP command is used to override the disk group default DISK_REPAIR_TIME.

-- Use the default DISK_REPAIR_TIME for the diskgroup.
ALTER DISKGROUP disk_group_1 OFFLINE DISK D1_0001;

-- Override the default DISK_REPAIR_TIME.
ALTER DISKGROUP disk_group_1 OFFLINE DISK D1_0001 DROP AFTER 30m;

If a disk goes offline during a rolling upgrade, the timer is not started until after the rolling upgrade is complete.

Rolling Upgrade

Clustered ASM instances for 11g onwards can be upgraded using a rolling upgrade. The ASM cluster is placed in rolling upgrade mode by issuing the following command from one of the nodes.

ALTER SYSTEM START ROLLING MIGRATION TO 11.2.0.0.0;

Once the cluster is in rolling upgrade mode each node in turn can be shutdown, upgraded and started. The cluster runs in a mixed version environment until the upgrade is complete. In this state, the cluster is limited to the following operations:

• Mount and dismount of the disk groups.
• Open, close, resize, and delete of database files.
• Access to local fixed views and fixed packages.

The current status of the ASM cluster can be determined using the following query.

SELECT SYS_CONTEXT('sys_cluster_properties', 'cluster_state') FROM dual;

Once the last node is upgraded, the rolling upgrade is stopped by issuing the following command, which checks all ASM instances are at the appropriate version, turns off rolling upgrade mode and restarts any pending rebalance operations.

ALTER SYSTEM STOP ROLLING MIGRATION;

Restrictions and miscellaneous points about the rolling upgrade process include:

• The Oracle clusterware must be fully patched before an ASM rolling upgrade is started.
• Rolling upgrades are only available from 11g onwards, so this method is not suitable for 10g to 11g upgrades.
• This method can be used to rollback to the previous version if the rolling upgrade fails before completion.
• If the upgrade fails, any rebalancing operations must complete before a new upgrade can be attempted.
• New instances joining the cluster during a rolling upgrade are automatically placed in rolling upgrade mode.
• If all instances in a cluster are stopped during a rolling upgrade, once the instances restart they will no longer be in rolling upgrade mode. The upgrade must be initiated as if it were a new process.

SYSASM Privilege and OSASM OS Group

The introduction of ASM moved the management of storage away from system adminstrators and into the DBA territory. Unfortunately this isn't how every company operates. As a result, some system administrators were required to have access to privileged users to handle disk storage. Oracle addresses this issue with the introduction of the SYSASM privilege and the OSASM operating system group, which provide two mechanisms to enable the separation of storage and database administration duties.

Users can be created in the ASM instance in a similar manner to database users. Granting these users the SYSASM privilege allows them to connect to the ASM instance and perform. administration tasks. First, connect to the ASM instance.

$ export ORACLE_SID=+ASM
$ sqlplus / as sysasm

Next, create a new user in the ASM instance and grant it the SYSASM privilege. The user is now able to connect using SYSASM.

SQL> CREATE USER asm_user_1 IDENTIFIED by asm_password;

User created.

SQL> GRANT SYSASM TO asm_user_1;

Grant succeeded.

SQL> CONN asm_user_1/asm_password AS SYSASM
Connected.
SQL>

Alternatively, assigning an operating system user to the OSASM group (asmadmin) allows then to connect as SYSASM using OS authentication. The following example creates a new OS user assigned to the OSASM group (asmadmin), which is immediately able to connect using SYSASM.

# useradd tim_hall -G asmadmin
# su - tim_hall
$ export ORACLE_SID=+ASM
$ export ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1
$ $ORACLE_HOME/bin/sqlplus / as sysasm

SQL*Plus: Release 11.1.0.6.0 - Production on Fri Aug 8 16:48:37 2008

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>

Scalability and Performance Enhancements

ASM files are stored in a disk group as a collection of extents. In Oracle 10g each extent mapped directly to one allocation unit (AU), but in Oracle 11g an extent can be made up of one or more allocation units. As files get larger, the extent size can grow to reduce the size of the extent map describing the file, thus saving memory.

When the disk group compatibility attributes are set to 11.1 or higher, the extent size will automatically grow as the file grows. The first 20,000 extents match the allocation unit size (1*AU). The next 20,000 extents are made up of 8 allocation units (8*AU). Beyond that point, the extent size becomes 64 allocation units (64*AU).

In addition to the automatic expansion of the extent sizes, Oracle 11g also allows control over the allocation unit size using the ATTRIBUTE clause in the CREATE DISKGROUP statement, with values ranging from 1M to 64M.

CREATE DISKGROUP disk_group_2
  EXTERNAL REDUNDANCY
  DISK '/dev/sde1'
  ATRRIBUTE 'au_size' = '32M';

The combination of expanding extent sizes and larger allocation units should result in increased I/O performance for very large databases.

New ASMCMD Commands and Options

The ASM command line utility includes several new commands. The following example output assumes you have already started the utility using the "-p" option.

$ asmcmd -p
ASMCMD [+] >

The usage notes for each command is available by issuing the "help " command from within the asmcmd utility, so I will avoid displaying all this information here.

The lsdsk command lists information about ASM disks from the V$ASM_DISK_STAT and V$ASM_DISK views. The summary usage is shown below.

lsdsk [-ksptcgHI] [-d ] [pattern]

An example of the output from the basic command and the "-k" option are shown below.

ASMCMD [+] > lsdsk
Path
/dev/sdc1
/dev/sdd1
/dev/sde1
ASMCMD [+] > lsdsk -d data -k
Total_MB  Free_MB  OS_MB  Name       Failgroup  Library  Label  UDID  Product  Redund   Path
    8189     6961   8189  DATA_0000  DATA_0000  System                         UNKNOWN  /dev/sdc1
    8189     6961   8189  DATA_0001  DATA_0001  System                         UNKNOWN  /dev/sdd1
    8189     6950   8189  DATA_0002  DATA_0002  System                         UNKNOWN  /dev/sde1
ASMCMD [+] >

The cp command allows files to be copied between ASM and local or remote destinations. The summary usage is shown below.

cp [-ifr] <[\@connect_identifier:]src> <[\@connect_identifier:]tgt>

The following example copies the current USERS datafile from ASM to the local file system.

ASMCMD [+] > cp +DATA/db11g/datafile/users.273.661514191 /tmp/users.dbf
source +DATA/db11g/datafile/users.273.661514191
target /tmp/users.dbf
copying file(s)...
file, /tmp/users.dbf, copy committed.
ASMCMD [+] >

There seems to be some concern that the cp command doesn't work for control files (here).

The md_backup command makes a copy of the metadata for one or more disk groups. The summary usage is shown below.

md_backup [-b location_of_backup] [-g dgname [-g dgname …]]

An example of the command is shown below. The resulting file contains all the metadata needed to recreate the ASM setup.

ASMCMD [+] > md_backup -b /tmp/backup.txt -g data
Disk group to be backed up: DATA
ASMCMD [+] >

The md_restore command allows you to restore a disk group from the metadata created by the md_backup command. It also allows a certain amount of manipulation of the final disk groups during the restore. The summary usage is shown below.

md_restore -b  [-li]
                   [-t (full)|nodg|newdg] [-f ]
                   [-g ',,...']
                   [-o ':,...']

A straight restore of the backup shown previously is shown below.

ASMCMD [+] > md_restore -b /tmp/backup.txt -t full -g data

The remap command repairs a range of physical blocks on disk. The contents of each block is not validated, so only blocks exhibiting read errors are repaired. The summary usage is shown below.

remap   

An example of the command is show below.

ASMCMD [+] > remap data data_0001 5000-5999

For detailed usage information see the ASMCDM Command Reference.

Preferred Read Failure Groups

In Oracle 10g, ASM always reads the primary copy of the mirrored extent set. This isn't a problem when both nodes and both failure groups are all located in the same site, but it can be inefficient for extended clusters, causing needless network traffic. Oracle 11g allows each node to define a preferred failure group, allowing nodes in extended clusters to access local failure groups in preference to remote ones.

To configure preferred read failure groups the disk group compatibility attributes must be set to 11.1 or higher. Once the compatibility options are correct, the ASM_PREFERRED_READ_FAILURE_GROUPS parameter is set to the preferred failure groups for each node.

SELECT name, failgroup FROM v$asm_disk;

NAME                           FAILGROUP
------------------------------ ------------------------------
DATA_0000                      DATA_0000
DATA_0001                      DATA_0001
DATA_0002                      DATA_0002

3 rows selected.

SQL>

ALTER SYSTEM SET ASM_PREFERRED_READ_FAILURE_GROUPS = 'data.data_0000', 'data.data_0001', 'data.data_0002';

Fast Rebalance

The "ALTER DISKGROUP ... MOUNT" statement allows disk groups to be mounted in restricted mode.

SQL> SELECT name FROM v$asm_diskgroup;

NAME
------------------------------
DATA

SQL> ALTER DISKGROUP data DISMOUNT;

Diskgroup altered.

SQL> ALTER DISKGROUP data MOUNT RESTRICTED;

Diskgroup altered.

SQL> ALTER DISKGROUP data DISMOUNT;

Diskgroup altered.

SQL> ALTER DISKGROUP data MOUNT;

Diskgroup altered.

SQL>

In a RAC environment, a disk group mounted in RESTRICTED mode can only be accessed by a single instance. The restricted disk group is not available to any ASM clients, even on the node where it is mounted.

Using RESTRICTED mode improves the performance of rebalance operations in a RAC environment as it elimitates the need for lock and unlock extent map messaging that occurs between ASM instances. Once the rebalance operation is complete, the disk group should be dismounted then mounted in NORMAL mode (the default).

Miscellaneous Disk Group Maintenance Enhancements

• The CREATE DISKGROUP and ALTER DISKGROUP commands include a new ATTRIBUTE clause (compatible, disk_repair_time, au_size).
• The CHECK clause of the ALTER DISKGROUP command has been simplified so there are only two options, NOREPAIR and REPAIR, available, with NOREPAIR as the default. Summarized errors are displayed, with full error messages writen to the alert log.
  

  ALTER DISKGROUP data CHECK; -- Like NOREPAIR
  ALTER DISKGROUP data CHECK NOREPAIR;
  ALTER DISKGROUP data CHECK REPAIR;

• Disk groups can now be mounted in restricted mode, which can improve performance of some maintenance tasks.
• The ALTER DISKGOUP command now includes ONLINE and OFFLINE clauses so disks can be taken offline for repair before being brought back online.
  

  -- Individual disks.
  ALTER DISKGROUP data OFFLINE DISK 'disk_0000', 'disk_0001';
  ALTER DISKGROUP data ONLINE DISK 'disk_0000', 'disk_0001';
  
  -- Failure groups.
  ALTER DISKGROUP data OFFLINE DISKS IN FAILGROUP 'fg_0000';
  ALTER DISKGROUP data ONLINE DISKS IN FAILGROUP 'fg_0000';
  
  -- Bring online all disks in disk group.
  ALTER DISKGROUP data ONLINE ALL;

• Disk groups that can't be mounted by ASM can now be dropped using the FORCE keyword of the DROP DISKGROUP command.
  

  DROP DISKGROUP data FORCE;