EM 12C报数据库报磁盘read errors的错误：

EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46)

oracle官方分析认为是触发bug导致：

当cell节点出现内存溢出，ORA-27626: Exadata error: 215 (Cell failed to allocate memory)错误就会出在数据库的alert日志中产生。
这个错误发生在大量的smart scan操作需要分配更多的内存用于smart scan。这不是说cell节点需增加更多的内存用于smart scan，而是说在这一点上的smart scan需要内存比较高。
这是一个与I/O相关的错误。所以在trace文件中可以看到很多类似的以下信息
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
所以在相同的时间里，在EM12c中可以看到以下的read-error相关的信息。
EM12C-Warning:+ASM_fd02-cluster:Disk-DATA_FD02.DATA_FD02_CD_08_FD02CEL06-has-279-Read-errors.:Nov-25,-2015-11:47:04-AM-CST(2015-11-25-11:46)

这个错误可以打patch 18169273来解决，这样ORA-27603和ORA-27626就不会再报出来，EM12c中报出的错误也不会报出来了。

=== ODM Cause Determination ===

The issue is caused by Bug 18169273 - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY

=== ODM Cause Justification ===

On 11.2.0.3 Exadata, ORA-27626: Exadata error: 215 (Cell failed to allocate memory) generated in db alert log.
when the cell runs out of memory, for that scan it returns error 215 (OSS_ERRCODE_NOCELLMEM) to RDBMS. Once received the IO result, we print that in alert.log. Those errors mean that some smart scans didn't actually go through smart scan but did get serviced through Block IO. This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan.
It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point.

ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)

Above version, symptom matches Bug 18169273.

估计和sga比较小也是有关系的

下面是oracle的分析思路

1、出现这个问题可能需要分析的日志

+ 所有数据库节点ASM和所有RDBMS实例的alert log
+ GI/RDBMS的opatch输出 - ‘opatch lsinventory -detail'
+ Exadata image版本 - imageinfo
+ 所有cell节点sundiag输出
+ ASM中的以下输出：
select name, read_errs from v$asm_disk;

2、结合em的报错信息，查看alert日志时发现db节点1 alert日志有报错

FileName = alert_reportdb1.log

Wed Nov 25 11:28:33 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_90598.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)
...
Wed Nov 25 11:43:49 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_95186.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)
...
Wed Nov 25 11:43:50 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_94153.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)
...
Wed Nov 25 12:30:21 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_127407.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.110//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)
...
Wed Nov 25 12:52:51 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_124095.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)
...
Wed Nov 25 12:52:52 2015
Errors in file /u01/app/oracle/diag/rdbms/reportdb/reportdb1/trace/reportdb1_ora_71945.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.107//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)

3、分析trace文件

FileName = reportdb1_ora_90598.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0.3/dbhome_1
System name: Linux
Node name: fd02db01.deppon.com
Release: 2.6.39-400.128.17.el5uek
Version: #1 SMP Tue May 27 13:20:24 PDT 2014
Machine: x86_64
Instance name: reportdb1
Redo thread mounted by this instance: 1
Oracle process number: 359
Unix process pid: 90598, image: oracle@fd02db01.deppon.com

*** 2015-11-25 09:56:43.270
*** SESSION ID:(4162.57941) 2015-11-25 09:56:43.270
*** CLIENT ID:(192.168.4.221) 2015-11-25 09:56:43.270
*** SERVICE NAME:(report) 2015-11-25 09:56:43.270
*** MODULE NAME:(JDBC Thin Client) 2015-11-25 09:56:43.270
*** ACTION NAME:() 2015-11-25 09:56:43.270

=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 1: RDBA: -1328288896, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 2: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 3: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 4: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 5: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 6: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 7: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 8: RDBA: -1328289152, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 9: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 10: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 11: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 12: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 13: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 14: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 15: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===

*** 2015-11-25 09:56:46.861
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 1: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 2: RDBA: -1328252672, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 3: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 4: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 5: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 7: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 8: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 9: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 10: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 11: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 12: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===

*** 2015-11-25 09:56:47.636
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 6: RDBA: -1329264256, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 8: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 9: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 4: RDBA: -1628508668, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 5: RDBA: -1628508544, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437782538, Disknum: 0, AUNUM: 417675
Fail Block 6: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 7: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 8: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===

*** 2015-11-25 09:56:48.971
Start of KCFIS reconnect diagnostics ---KCFIS is responsible for ExaData-optimized I/O. It is related to smart scan.
Appstate: 0x7f7e9e0c82d8 Cell: 172.16.247.110/, Device: /box/predicate1600118
Printing reconnect diagnostics: app_type 1 tsn 16 objd 1674375 objn 1674375
Dumping sqltext for sqlid: 5dmxfk16wnkav
kxsDumpSqlTextFromSQLID SQL=
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,
FBASE_CODE,
FORG_NAME,
FORG_CODE,
FBIG_NAME,
FBIG_CODE,
...

+++++++++++++++++ Dumping sqltext in chunks of 2048 bytes ++++++++
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,
FBASE_CODE,
FORG_NAME,
FORG_CODE,
FBIG_NAME,
FBIG_CODE,
FSMALL_NAME,
FSMALL_CODE,
FDEPT_NAME,
...
...
TO_CHAR(T.FTOTAL_AMOUNT, 'fm999999990.00') FTOTAL_AMOUNT FROM t_etl_papers t LEFT JOIN crm.t_cust_custbasedata CUS ON t.fnumber = cus.fcustnumber AND cus.fcuststatus <> '2' WHERE 1 = 1 and t.FDEPT_CODE IN (SELECT T.FDEPTCODE CODE FROM DPRP.V_BUSINESS_ORGANIZATION T WHERE T.FDEPTCODE = 'W011302040502' OR T.FSMALLAREACODE = 'W011302040502' OR T.FBIGAREACODE = 'W011302040502' OR T.FORGCODE = 'W011302040502' OR T.FBASECODE ='W011302040502' OR t.fbosscode='W011302040502' OR T.fex_boss_code='W011302040502' union all select w.FDEPTCODE from DPRP.v_operation_organization w WHERE w.FDEPTCODE = 'W011302040502' OR w.FSMALLAREACODE = 'W011302040502' OR w.FBIGAREACODE = 'W011302040502' OR w.FORGCODE = 'W011302040502' OR w.FBASECODE = 'W011302040502' OR w.fbosscode = 'W011302040502' OR w.fex_boss_code='W011302040502' ) and t.FDATE = to_date('2015-11-24','yyyy-mm-dd') ORDER BY t.FDEPT_CODE,T.FNUMBER,T.FCHARGE_TYPE,T.FPAYMENT_TYPE,T.FBILL_TYPE) m ) )
+++++++++++++ End of SQl dump +++++++++++++
Dumping session
sid: 4162 ser: 57941 audsid: 231250857 user: 52/DPRP
flags: (0x100045) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 359 O/S info: user: grid, term: UNKNOWN, ospid: 90598
image: oracle@fd02db01.deppon.com
client details:
O/S info: user: root, term: unknown, ospid: 1234
machine: fspbaobiao-221 program: JDBC Thin Client
client info: 192.168.4.221
application name: JDBC Thin Client, hash value=2546894660
current SQL:
select count(*) recNum from (select rownum 序号,m.* from ((SELECT FBASE_NAME,
FBASE_CODE,
FORG_NAME,
FORG_CODE,
FBIG_NAME,
FBIG_CODE,
FSMALL_NAME,
FSMALL_CODE,
FDEPT_NAME,

...
*** 2015-11-25 09:56:49.621
=======Dumping Fail Data blocks=======
Fail Block 0: RDBA: -1328252800, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x437774f78, Disknum: 2, AUNUM: 119987
Fail Block 1: RDBA: 1103997444, Diskname: o/172.16.247.110/DATA_FD02_CD_03_fd02cel06, FOB: 0x856eb2c8, Disknum: 2, AUNUM: 407353
Fail Block 2: RDBA: -1328289280, Diskname: o/172.16.247.110/DATA_FD02_CD_08_fd02cel06, FOB: 0x437774f78, Disknum: 0, AUNUM: 277141
Fail Block 3: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 4: RDBA: -1329264622, Diskname: o/172.16.247.106/DATA_FD02_CD_05_fd02cel02, FOB: 0x437774f78, Disknum: 0, AUNUM: 276729
Fail Block 5: RDBA: -1624484864, Diskname: o/172.16.247.111/DATA_FD02_CD_09_fd02cel07, FOB: 0x437776f58, Disknum: 2, AUNUM: 418098
====End of Dumping Fail Data blocks===
kcfis: Exadata error: Device 'o/172.16.247.111//box/predicate' error code 215 (Cell failed to allocate memory)

*** 2015-11-25 11:28:33.613
ORA-27603: Cell storage I/O error, I/O failed on disk o/172.16.247.111//box/predicate at offset 0 for data length 0
ORA-27626: Exadata error: 215 (Cell failed to allocate memory)

3、然后通过相关报错查看文档确定是bug

-- Search --

Note: This is INTERNAL ONLY research. No action should be taken by the customer on this information. This is research only, and may NOT be applicable to your specific situation.

Searched on: Cell failed to allocate memory

Bug 20310116 - ORA-27603 AND ORA-27626 IN ALERT LOG
Bug 18169273 - FLOOD OF ALERT WHEN CELL GOES TO PASS THROUGH MODE DUE TO NO CELL MEMORY

4、采用打one-off-patch的方式来避免这个告警，或者忽略

=== ODM Proposed Solution ===

Apply one-off patch 18169273 on 11.2.0.3.28 Exadata Database.

=== ODM Proposed Solution Justif ===

Bug 18169273 is fixed in 12.1.0.2 and above.
There is a one-off patch on 11.2.0.3.28 Exadata Database.
These messages should not be considered buggy but only informative.

=== ODM Solution / Action Plan ===

This happens when there is a surge in smart scan sessions which need more memory to be allocated for the scan.
It is not just number of smart scan sessions being more but number of smart scan sessions which need more memory are high at that point.

To resolve the issue, please apply the patch 18169273.

Download and review the readme and pre-requisites for Patch 18169273.
Download link:
https://updates.oracle.com/download/10098816.html
Select a Release: Oracle 11.2.0.3.28 Exadata Database

ITPub博客

EM 12c Read-errors ORA-27603 ORA-27626报错总结

最新文章