客户10.2.0.4 RAC环境出现ORA-600[13310]错误。

错误信息如下：

Sun Aug 1 04:08:24 2010

Errors in file /oracle/admin/ORCL/udump/orcl1_ora_14964.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27544: Failed to map memory region for export
ORA-27300: OS system dependent operation:socket failed with status: 23
ORA-27301: OS failure message: File table overflow
ORA-27302: failure occurred at: sskgxpcre1
Sun Aug 1 04:08:24 2010
Trace dumping is performing id=[cdmp_20100801040824]
Sun Aug 1 16:08:25 2010
Trace dumping is performing id=[cdmp_20100801160825]
Sun Aug 1 16:08:27 2010
Errors in file /oracle/admin/ORCL/bdump/orcl1_diag_17090.trc:
ORA-27050: function called with invalid FIB/IOV structure
Additional information: 2
ORA-27041: unable to open file
HPUX-ia64 Error: 23: File table overflow
Additional information: 3
Sun Aug 1 04:08:42 2010
Errors in file /oracle/admin/ORCL/udump/orcl1_ora_15065.trc:
ORA-00603: Message 603 not found; No message file for product=RDBMS, facility=ORA
ORA-27544: Message 27544 not found; No message file for product=RDBMS, facility=ORA
ORA-27300: Message 27300 not found; No message file for product=RDBMS, facility=ORA; arguments: [socket] [23]
ORA-27301: Message 27301 not found; No message file for product=RDBMS, facility=ORA; arguments: [File table overflow]
ORA-27302: Message 27302 not found; No message file for product=RDBMS, facility=ORA; arguments: [sskgxpcre1]
Sun Aug 1 04:08:48 2010
Trace dumping is performing id=[cdmp_20100801160843]
Sun Aug 1 04:08:48 2010
Trace dumping is performing id=[cdmp_20100801160843]
Sun Aug 1 04:08:56 2010
ospid 17096: network interface with IP address 192.168.1.81 no longer operational
Sun Aug 1 04:08:58 2010
Errors in file /oracle/admin/ORCL/bdump/orcl1_ora_15148.trc:
ORA-00600: internal error code, arguments: [13310], [], [], [], [], [], [], []
Sun Aug 1 04:08:58 2010
Errors in file /oracle/admin/ORCL/bdump/orcl1_ora_15148.trc:
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/dev/vgpmcs11/rr01G1001'
ORA-27041: unable to open file
HPUX-ia64 Error: 23: File table overflow
Additional information: 3
ORA-00600: internal error code, arguments: [13310], [], [], [], [], [], [], []
Sun Aug 1 04:09:03 2010
Trace dumping is performing id=[cdmp_20100801040859]

根据MOS文档ORA-600 [13310] Reported When Obtaining An OS Unique Identifier [ID 1234083.1]，ORA-600[13310]错误出现的原因是无法获取操作系统上的唯一标识信息，而Oracle文档给出的解决方案是释放或者增加系统资源，比如内存或SWAP空间。

由于出现600错误之前出现了ORA-2730N的错误，说明确实出现了操作系统层面的问题。根据ORA-27544错误以及File table overflow信息进行定位，确认是HP-UX上10.2.0.4的bug：hp-ux: File handles not released after upgrade to 10.2.0.3 CRS Bundle#2 or 10.2.0.4 [ID 739557.1]。

这个问题是10.2.0.3的CRS Bundle补丁引入，由mmap引发的错误，影响版本包括10.2.0.3、10.2.0.4和11.1.0.6。Oracle在10.2.0.5、11.1.0.7中解决了这个问题，在10.2.0.4 CRS Bundle Patch #2以及10.2.0.4最新的CRS PSU中同样修正了该错误。