In this Document
  Symptoms
  Changes
  Cause
  Solution
  References

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.7 - Release: 10.2 to 11.1
Information in this document applies to any platform.
ASM instance crash but can likely occur for RDBMS instances

Symptoms

CRS starts up successfully on the newly added node but the ASM instance fails to start and causes all ASM instances in the cluster to crash with

Maximum Tranmission Unit (mtu) of the ether adapter is different
on the node running instance 4, and this node.
Ether adapters connecting the cluster nodes must be configured
with identical mtu on all the nodes, for Oracle.
Please ensure the mtu attribute of the ether adapter on all
nodes are identical, before running Oracle.
Tue Aug 5 12:54:28 2008
Errors in file /opt/oracle/app/oracle/product/10.2.0/asm/admin/+ASM/bdump/+asm1_lmon_2499.trc:
ORA-27550: Target ID protocol check failed. tid vers=%d, type=%d, remote instance number=%d, local
instance number=%d

Changes

1. Newly added node with incorrect NIC settings
   or
2. Modification took place with regards to the interconnect Network Card Interface Settings for an existing node.

Cause

ORA-27550 can occur for various reasons:

• MTU size on interfaces are different
• MTU size changed on OS level but instance didn't restart so it's still using old value
• Switch is unable to handle the MTU size specified on the NIC
• Instances may use different interfaces due to incorrect cluster_interconnects setting in the init.ora
• Not all nodes private interface are on the same subnet

Solution

• Check MTU sizes on all interfaces that correspond to the private network. If part of a link aggregation setup (ipmp, bonding) ensure you check all active/standby NICs
• MTU change requires restart of instance to be effective, if the change happens on all nodes, a downtime is needed to restart all instances

• Confirm the switch can handle that MTU-size, you can do this with traceroute sending an equivalent frame. size.
• Ensure the other instances are using the same private network, as 'ORA-27550' can occur if the subsequent nodes in the cluster are using a different network. Check the alert log  for the setting of cluster_interconnect.
• Confirm the private interfaces for each node is on the same subnet.

For example:

Modify the netmask for the bcme0 (interconnect) interface to be 255.255.255.0 same across all nodes.
Use ifconfig to modify the netmask and ensure its consistent with the current nodes already in the cluster.

Login as root.

% cd $ORA_CRS_HOME/bin
% crsctl stop crs
% ifconfig bcme0 down
% ifconfig bcme0 192.168.1.31 netmask 255.255.255.0
% ifconfig bcme0 up
% crsctl start crs