ITPub博客

首页 > 数据库 > Oracle > EM 12c Read-errors ORA-27603 ORA-27626报错总结

EM 12c Read-errors ORA-27603 ORA-27626报错总结

原创 Oracle 作者:Curry_0316 时间:2016-01-07 11:35:56 0 删除 编辑
EM 12C报数据库报磁盘read errors的错误:
EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46) 

oracle官方分析认为是触发bug导致:
    当cell节点出现内存溢出,ORA-27626: Exadata error: 215 (Cell failed to allocate memory)错误就会出在数据库的alert日志中产生。 
    这个错误发生在大量的smart scan操作需要分配更多的内存用于smart scan。这不是说cell节点需增加更多的内存用于smart scan,而是说在这一点上的smart scan需要内存比较高。 
    这是一个与I/O相关的错误。所以在trace文件中可以看到很多类似的以下信息 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
所以在相同的时间里,在EM12c中可以看到以下的read-error相关的信息。 
EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46) 

这个错误可以打patch 18169273来解决,这样ORA-27603和ORA-27626就不会再报出来,EM12c中报出的错误也不会报出来了。
=== ODM Cause Determination === 

The issue is caused by Bug 18169273 - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY


=== ODM Cause Justification === 

On 11.2.0.3 Exadata, ORA-27626: Exadata error: 215 (Cell failed to allocate memory) generated in db alert log. 
when the cell runs out of memory, for that scan it returns error 215 (OSS_ERRCODE_NOCELLMEM) to RDBMS. Once received the IO result, we print that in alert.log. Those errors mean that some smart scans didn't actually go through smart scan but did get serviced through Block IO. This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan. 
It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point

ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 

Above version, symptom matches Bug 18169273. 
估计和sga比较小也是有关系的


下面是oracle的分析思路
1、出现这个问题可能需要分析的日志
+ 所有数据库节点ASM和所有RDBMS实例的alert log 
+ GI/RDBMS的opatch输出 - ‘opatch lsinventory -detail' 
+ Exadata image版本 - imageinfo 
+ 所有cell节点sundiag输出 
+ ASM中的以下输出: 
select name, read_errs from v$asm_disk; 

2、结合em的报错信息,查看alert日志时发现db节点1 alert日志有报错
FileName = alert_reportdb1.log 

Wed Nov 25 11:28:33 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_90598.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 
... 
Wed Nov 25 11:43:49 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_95186.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 
... 
Wed Nov 25 11:43:50 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_94153.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 
... 
Wed Nov 25 12:30:21 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_127407.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 
... 
Wed Nov 25 12:52:51 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_124095.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 
... 
Wed Nov 25 12:52:52 2015 
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_71945.trc: 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)  

3、分析trace文件
FileName = reportdb1_ora_90598.trc 

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production 
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, 
Data Mining and Real Application Testing options 
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1 
System name: Linux 
Node name: fd02db01.deppon.com 
Release: 2.6.39-400.128.17.el5uek 
Version: #1 SMP Tue May 27 13:20:24 PDT 2014 
Machine: x86_64 
Instance name: reportdb1 
Redo thread mounted by this instance: 1 
Oracle process number: 359 
Unix process pid: 90598, image: oracle@fd02db01.deppon.com 


*** 2015-11-25 09:56:43.270 
*** SESSION ID:(4162.57941) 2015-11-25 09:56:43.270 
*** CLIENT ID:(192.168.4.221) 2015-11-25 09:56:43.270 
*** SERVICE NAME:(report) 2015-11-25 09:56:43.270 
*** MODULE NAME:(JDBC Thin Client) 2015-11-25 09:56:43.270 
*** ACTION NAME:() 2015-11-25 09:56:43.270 

=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 

*** 2015-11-25 09:56:46.861 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 1: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 2: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 3: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 4: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 5: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 7: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 8: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 9: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 10: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 11: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 12: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 

*** 2015-11-25 09:56:47.636 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 6: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 8: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 9: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675 
Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 8: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 

*** 2015-11-25 09:56:48.971 
Start of KCFIS reconnect diagnostics      ---KCFIS is responsible for ExaData-optimized I/O. It is related to smart scan.
Appstate: 0x7f7e9e0c82d8 Cell: 172.16.247.110/, Device: /box/predicate1600118 
Printing reconnect diagnostics: app_type 1 tsn 16 objd 1674375 objn 1674375 
Dumping sqltext for sqlid: 5dmxfk16wnkav 
kxsDumpSqlTextFromSQLID SQL= 
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME, 
FBASE_CODE, 
FORG_NAME, 
FORG_CODE, 
FBIG_NAME, 
FBIG_CODE, 
... 

+++++++++++++++++ Dumping sqltext in chunks of 2048 bytes ++++++++ 
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME, 
FBASE_CODE, 
FORG_NAME, 
FORG_CODE, 
FBIG_NAME, 
FBIG_CODE, 
FSMALL_NAME, 
FSMALL_CODE, 
FDEPT_NAME, 
...
...

TO_CHAR(T.FTOTAL_AMOUNT, 'fm999999990.00') FTOTAL_AMOUNT FROM t_etl_papers t LEFT JOIN crm.t_cust_custbasedata CUS ON t.fnumber = cus.fcustnumber AND cus.fcuststatus <> '2' WHERE 1 = 1 and t.FDEPT_CODE IN (SELECT T.FDEPTCODE CODE FROM DPRP.V_BUSINESS_ORGANIZATION T WHERE T.FDEPTCODE = 'W011302040502' OR T.FSMALLAREACODE = 'W011302040502' OR T.FBIGAREACODE = 'W011302040502' OR T.FORGCODE = 'W011302040502' OR T.FBASECODE ='W011302040502' OR t.fbosscode='W011302040502' OR T.fex_boss_code='W011302040502' union all select w.FDEPTCODE from DPRP.v_operation_organization w WHERE w.FDEPTCODE = 'W011302040502' OR w.FSMALLAREACODE = 'W011302040502' OR w.FBIGAREACODE = 'W011302040502' OR w.FORGCODE = 'W011302040502' OR w.FBASECODE = 'W011302040502' OR w.fbosscode = 'W011302040502' OR w.fex_boss_code='W011302040502' ) and t.FDATE = to_date('2015-11-24','yyyy-mm-dd') ORDER BY t.FDEPT_CODE,T.FNUMBER,T.FCHARGE_TYPE,T.FPAYMENT_TYPE,T.FBILL_TYPE) m ) ) 
+++++++++++++ End of SQl dump +++++++++++++ 
Dumping session 
sid: 4162 ser: 57941 audsid: 231250857 user: 52/DPRP 
flags: (0x100045) USR/- flags_idl: (0x1) BSY/-/-/-/-/- 
flags2: (0x40009) -/-/INC 
pid: 359 O/S info: user: grid, term: UNKNOWN, ospid: 90598 
image: oracle@fd02db01.deppon.com 
client details: 
O/S info: user: root, term: unknown, ospid: 1234 
machine: fspbaobiao-221 program: JDBC Thin Client 
client info: 192.168.4.221 
application name: JDBC Thin Client, hash value=2546894660 
current SQL: 
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME, 
FBASE_CODE, 
FORG_NAME, 
FORG_CODE, 
FBIG_NAME, 
FBIG_CODE, 
FSMALL_NAME, 
FSMALL_CODE, 
FDEPT_NAME, 


... 
*** 2015-11-25 09:56:49.621 
=======Dumping Fail Data blocks======= 
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987 
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353 
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141 
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 4: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729 
Fail Block 5: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098 
====End of Dumping Fail Data blocks=== 
kcfis: Exadata error: Device 'o/172.16.247.111//box/predicate' error code 215 (Cell failed to allocate memory) 

*** 2015-11-25 11:28:33.613 
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0 
ORA-27626: Exadata error: 215 (Cell failed to allocate memory) 

3、然后通过相关报错查看文档确定是bug
-- Search -- 

Note: This is INTERNAL ONLY research. No action should be taken by the customer on this information. This is research only, and may NOT be applicable to your specific situation. 

Searched on: Cell failed to allocate memory 

Bug 20310116 - ORA-27603 AND ORA-27626 IN ALERT LOG 
Bug 18169273 - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY 

4、采用打one-off-patch的方式来避免这个告警,或者忽略
=== ODM Proposed Solution === 

Apply one-off patch 18169273 on 11.2.0.3.28 Exadata Database.

=== ODM Proposed Solution Justif === 

Bug 18169273 is fixed in 12.1.0.2 and above. 
There is a one-off patch on 11.2.0.3.28 Exadata Database. 
These messages should not be considered buggy but only informative.

=== ODM Solution / Action Plan === 

This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan. 
It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point. 

To resolve the issue, please apply the patch 18169273. 

Download and review the readme and pre-requisites for Patch 18169273. 
Download link: 
 
Select a Release: Oracle 11.2.0.3.28 Exadata Database

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/29953799/viewspace-1973100/,如需转载,请注明出处,否则将追究法律责任。

下一篇: em 12c硬件需求
请登录后发表评论 登录
全部评论

注册时间:2014-10-11

  • 博文量
    88
  • 访问量
    253319