遭遇aix bug

上周客户进行数据库移植，user表空间增长迅速，200gb的空间就剩下2m了，我们给扩了10gb的空间上去，用的裸设备。没想到问题出现了，各一个节点的oracle alert日志不断报错
ORA-27063: Message 27063 not found; No message file for product=RDBMS, facility=ORA
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: -1
Additional information: 8192
Wed Jan 21 10:10:50 2009
Errors in file /opt/app/oracle/admin/yb/bdump/yb2_ckpt_573768.trc:
ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=ORA; arguments: [125] [/dev/rS_user23_10g]
ORA-01115: Message 1115 not found; No message file for product=RDBMS, facility=ORA; arguments: [125] [1]
ORA-27063: Message 27063 not found; No message file for product=RDBMS, facility=ORA
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: -1
Additional information: 8192

发现这个问题后 第一感觉是不是命令用错了，或者裸设备划的有问题，进行下列操作开始排除
$dbv file=/dev/rS_user23_10g blocksize=8192
在一节点执行dbv 正常 在二节点执行就报io有问题
排除裸设备存在问题的可能性
检查570，存储，b16等相关硬件及线缆正常，无硬件告警。
在metalink上查询oracle相关报错 确定是不是遇到数据库的bug
检查2台570 的errpt 及相关配置参数
发现二节点errpt 里有如下报错
LABEL: LVM_IO_FAIL
IDENTIFIER: E86653C3

Date/Time: Wed May 21 12:25:49 2008
Sequence Number: 64953
Machine Id: 00CB0ADE000
Node Id: mytesthost
Class: H
Type: PERM
Resource Name: LVDD
Resource Class: NONE
Resource Type: NONE
Location:

Description
I/O ERROR DETECTED BY LVM

搜索报错发现 遇到了aix bug ：IZ23018: CONC VG REMOTE NODE MAY GET LVM_IO_FAIL FOR A PV WITH SIZE >1TB APPLIES TO AIX 5300-07
http://www-01.ibm.com/support/docview.wss?uid=isg1IZ23018

IZ23018: CONC VG REMOTE NODE MAY GET LVM_IO_FAIL FOR A PV WITH SIZE >1TB APPLIES TO AIX 5300-07

A fix is available
Obtain fix for this APAR

APAR status
Closed as program error.

Error description
In a Concurrent Volume Group (VG) setup, a remote
node may get IO failures (LVM_IO_FAIL) for a disk (PV)
which is more than 1TB in size. The "LVM_IO_FAIL"
entry in error report will have error code as 22 with
a negative block number. This happens after an LV is
created or extended to PPs which reside beyond 1TB on
the disk and an IO occurs to that region.

Sample errpt entry may look like this (on remote node )

LABEL: LVM_IO_FAIL
IDENTIFIER: E86653C3

Date/Time: Wed May 21 12:25:49 2008
Sequence Number: 64953
Machine Id: 00CB0ADE000
Node Id: mytesthost
Class: H
Type: PERM
Resource Name: LVDD
Resource Class: NONE
Resource Type: NONE
Location:

Description
I/O ERROR DETECTED BY LVM

Probable Causes
POWER, DRIVE, ADAPTER, OR CABLE FAILURE

Recommended Actions
RUN DIAGNOSTICS AGAINST THE FAILING DEVICE

Detail Data
PHYSICAL VOLUME DEVICE MAJOR/MINOR
8000 0013 0000 0028
ERROR CODE AS DEFINED IN sys/errno.h
22
BLOCK NUMBER
-624811008
LOGICAL VOLUME DEVICE MAJOR/MINOR
8000 0024 0000 0001
PHYSICAL BUFFER TRANSACTION TIME
8296
RESIDUAL COUNT
4096
NUMBER OF BLOCKS
4096
I/O TYPE
LVM META DATA
SENSE DATA
FFFF FFFF FFED 6112 00CB 0ADE 0000 4C00 0000
011A 0B8B 9E67 00CB 0ADE 60EC D30D 0000 0000
0000 0000
Local fix
On the remote node varyoff and varyon the VG.
Problem summary
When an LV with partitions allocated on blocks further than
1 TB into a disk is created in a concurrent VG, the remote
nodes may update their LVM drivers with an incorrect block
number for those partitions.
.
This may cause I/O failures from the remote node to this block
or possibly LVM command failures if the LV's lvcb is allocated
on one of these blocks.
Problem conclusion
Correct variable types in code that updates remote nodes with
new LV partitions to account for large block addresses.
Temporary fix
Comments
5200-10 - use AIX APAR IZ28288
5300-06 - use AIX APAR IZ25834
5300-07 - use AIX APAR IZ23018
5300-08 - use AIX APAR IZ24640
5300-09 - use AIX APAR IZ23868
6100-00 - use AIX APAR IZ27349
6100-01 - use AIX APAR IZ27211
6100-02 - use AIX APAR IZ24099
APAR information
APAR number IZ23018
Reported component name AIX 5.3
Reported component ID 5765G0300
Reported release 530
Status CLOSED PER
PE NoPE
HIPER NoHIPER
Submitted date 2008-05-22
Closed date 2008-06-02
Last modified date 2008-11-18

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:
IZ23868 IZ24099 IZ24589 IZ24640 IZ25834 IZ27211 IZ27349 IZ28288

Publications Referenced

Fix information
Fixed component name AIX 5.3
Fixed component ID 5765G0300

Applicable component levels
R530 PSY U819294 UP08/07/24 I 1000

PTF to Fileset Mapping
U819294 bos.clvm.enh 5.3.7.3


下载补丁 晚上9点申请停机后 开始打补丁
#installp -a -d . all 安装补丁
#installp -c . all 提交
#installp -r . all 回退
#installp -s 查看升级成功的包
#installp -l 看介质中可以在当前版本中安装

打完补丁 问题解决。以后划lun的时候，不敢偷懒了，多划一些小的，也不遭这罪了。