环境:oracle 11.2.0.3 rac ,AIX6.1
第一节点ASM实例报04031,赶到单位检查日志:
ORA-04031: unable to allocate 3896 bytes of shared memory ("shared pool","select name_kfgrp, number_kf...","sga heap(1,0)","kglsim object batch")
登录ASM实例,检查SGA
SQL> select bytes,name,pool from v$sgastat where name like '%free memory%';
60501728 free memory shared pool
未发现异常
设置事件检查TRACE
SQL >alter session set events '4031 trace name heapdump level 536870914';
SQL >alter session set events '4031 trace name context off;
检查TRACE文件
1、Memory Utilization of Subpool 1中未发现异常
2、LIBRARY CACHE STATISTICS--SQL AREA reload较高,其他正常。
MOS上记录了一个11.2.0.3的BUG:
Oracle Server - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.
Symptoms
11.2.0.3.0 ASM instance crash after giving the following errors:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Tue Mar 27 14:09:16 2012
Errors in file /app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_2966.trc (incident=206840):
ORA-04031: unable to allocate 3896 bytes of shared memory ("shared pool","DECLARE
hdl number; blk_nu...","sga heap(1,0)","kglsim object batch")
Incident details in: /app/oracle/diag/asm/+asm/+ASM1/incident/incdir_206840/+ASM1_ora_2966_i206840.trc
...
...
...
Tue Mar 27 15:11:28 2012
DDE: Problem Key 'ORA 4031' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Tue Mar 27 15:15:25 2012
Errors in file /app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_2967.trc (incident=211937):
ORA-00600: internal error code, arguments: [ksxp_rm_check0], [0xFFFFFFFF79122998], [0], [0x517854080], [], [], [], [], [], [], [], []
Incident details in: /app/oracle/diag/asm/+asm/+ASM1/incident/incdir_211937/+ASM1_ora_2967_i211937.trc
...
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
NOTE: Deferred communication with ASM instance
Errors in file /app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21400.trc:
ORA-15055: unable to connect to ASM instance
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [ksxp_rm_check0],
[0xFFFFFFFF79122998], [0], [0x517854080], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Cause
Non-published Bug:12925089 which fix is included starting 11.2.0.4
根据 ID 1370925.1推荐ASM配置:
References
NOTE:437924.1 - ASM & Shared Pool (ORA-4031)
NOTE:1363369.1 - Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM
Hdr: 13605735 11.2.0.3 PCW 11.2.0.3 PRODID-5 PORTID-46
Abstract: SET ASM MEMORY_TARGET TO BE AT LEAST 1536M WHILE UPGRADING TO 11.2.0.3
*** 01/18/12 02:34 pm REQUEST TEXT ***
In 11.2.0.3, init.ora parameter "processes" will be default to "available CPU
cores * 80 + 40". As the default value for "memory_target" is based on
"processes", it can be insufficient if there's large number of CPU cores or
large number of diskgroups which could cause issues, it's recommended to
increase the value of memory_max_target and memory_target while/before
upgrading to 11.2.0.3 to the minimal of 1536m or the default.
*** 01/18/12 02:34 pm BUSINESS NEED ***
*** 01/19/12 01:42 am ***
*** 02/03/12 02:18 pm ***
*** 02/03/12 02:18 pm RESPONSE ***
ER is forwarded for further consideration
修改相关参数
bjschxdbcx01:/u01/app/grid/diag/asm/+asm/+ASM1/trace$vmstat
System configuration: lcpu=40 mem=81920MB ent=10.00
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
6 1 9409443 1227672 0 0 0 215 399 0 4608 71811 23295 6 2 89 2 1.37 13.7
1.修改PROCESSES=CPU.CORE*80+40
2.metalink推荐memory_max_target = 4096m ,memory_target = 1536m
3.修改_library_cache_advice=false
SQL> alter system set "_library_cache_advice"=false scope = spfile;
System altered.
SQL> alter system set memory_max_target = 4096m scope=spfile;
System altered.
SQL> alter system set memory_target = 1536m scope=spfile;
System altered.
SQL> alter system set processes = 3240
2 scope = spfile;
System altered.
--重启第一节点集群后现象消失
crsctl stop cluster -n bjschxdbcx01
crsctl start cluster -n bjschxdbcx01