Description
Summary of typical IPMP Configurations:

Steps to Follow
Contents:
1.  Production and test interfaces in the same IP subnet
    1.1  With default router
    1.2  Without default router
    1.3  With dedicated hosts acting as test targets with "host-routes"
    1.4  Configuration example for 1.1, 1.2 and 1.3
   
2.  Production and test interfaces in different IP subnets but the same
    physical network
    2.1  With default router in production subnet and test subnet
    2.2  With default router in production subnet but without default router in test subnet
    2.3  With dedicated hosts acting as test targets with "host-routes"
    2.4  Configuration example for 2.1, 2.2 and 2.3
    2.5  Configuration example for 2.2 if you use IPMP on the test subnet

A.  Start script. for adding static "host routes" on Solaris 8 and 9

B.  Adding persistent "host routes" via 'route -p' option on Solaris 10   

C.  Summary

Note: This document does not cover the basic knowledge of IP Multipathing. If you would need it, please search  in Oracle Documentation.    

All IPMP patches mentioned below should be installed.

  OS
Release       Sparc      Intel      Comments
-----------------------------------------------------------------------------------
7 or earlier   N/S        N/S       IPMP was introduced with Solaris[TM] 8 U2
-----------------------------------------------------------------------------------
8             108528     108529     kernel update patch (in.mpathd, ip, icmp, ifconfig)
              116965     116966     ip/arp/tcp/udp/tun patch
              109900     109901     /etc/init.d/network and /sbin/ifparse patch
              109902     109903     in.ndpd patch
              110378     110396     mipagent patch Mobile IP
              116991     116992     rcm_daemon patch
-----------------------------------------------------------------------------------
9             112233     112234     Kernel Patch
              114344     119435     arp, dlcosmk, ip, and ipgpc Patch
                "        114348     /usr/sbin/in.routed patch
              112912      N/P       libinetcfg Patch
              113073     114733     ufs and fsck patch
-----------------------------------------------------------------------------------
10            118833-36  118855-36  kernel patch
              120011-14  120012-14  kernel patch (min. required for IPMP patch 141532) 
              141532     141533     in.mpathd patch 
10 U4 (8/07)  127111     127112     kernel patch 
10 U5 (5/08)  137111     137112     kernel patch
10 U6 (10/08) 138888     138889     kernel patch
10 U7 (5/09)  141414     141415     kernel patch
10 U8 (10/09) 142900     142901     kernel patch
10 U9 (9/10)  144488     144489     kernel patch
-----------------------------------------------------------------------------------

Key
N/S - Product not supported on this OS 
N/P - No patch applicable for this OS 


Legend:
IPMP    system with IPMP group
T       target host
p       network/host partof IP address of interface in production subnet
t       network/host part of IP address of interface in test subnet
---     data link subnet (i.e broadcast domain) with one IP subnet 
===     data link subnet (i.e broadcast domain) with two (or more) IP subnets

Good to know:
The operation of IP Multipathing (in.mpathd) depends on the routing configuration. Therefore in.mpathd always refers to the routing-table (IRE-cache) to distinguish which test partner(s) are going to be used. Test partners are required for deciding if the interface is working properly.

in.mpathd by default chooses the default router as single test-target for probing. If no default router exists for the test-interface ip address, arbitrary hosts on the link are detected by sending out "all hosts" multicast packets (224.0.0.1) on the wire to detect its test-partners. An "all routers" multicasts (224.0.0.2) will never be sent! The first five hosts that are responding to the echo packets are chosen as targets for probing. In such a non-default router environment, the in.mpathd always tries to find five probe targets via an "all hosts" multicast.

The in.mpathd detects failures and repairs by sending out 'icmp-echo' probes (like pinging) from all interfaces that are part of the IPMP group. If there are no replies to five consecutive probes, the interface is considered to have failed and performs a failover of the network access to another interface in the IPMP group. The probing rate depends on the failure detection time which is defined in /etc/default/mpathd. By default, failure detection time is 10 seconds. Thus the five probes will be sent within the failure detection time.

1.  Production and test interfaces in the same IP subnet

1.1 With default router

                                      
                                   +----------------+                                   
                                   | default router |                                      
                                   +-------o--------+                                            
                                           | p=t:172.20.20.1                                            
                                           |                                            
                                           |        
      ----------+-+------------------------+-------------------------                  
                | |                                 p=t:172.20.20/24
                | | 
p:172.20.20.10  | |
t:172.20.20.210 | | t:172.20.20.220           
            +---o-o---+              
            |   IPMP  |               
            +---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will be NO "all hosts" multicast (224.0.0.1) sent out.

Advantages:

• Easiest configuration for IPMP.
  

Disadvantages:

• If the default router is down then IPMP failover does not work anymore. The in.mpathd does NOT send out multicasts to get other probe targets, therefore all interfaces in the IPMP group get the state "failed". You can ignore this bug/feature when you have a default router which is 100% online! Please look to RFE 4431511 and 4489960 for further information.
• If you have a lot of IPMP groups, the default router has to reply to many ICMP requests. Take care about the default router. Do not overload the default router.
• The default router has to reliably answer ICMP echo requests. (e.g.  firewalls sometimes do not)

1.2 Without default router

                              +-----+               +-----+                      
                              | T 1 |     ...       | T 5 |
                              +--o--+               +--o--+                             
                                 | p=t:172.20.20.110   | p=t:172.20.20.150                                            
                                 |                     |        
                                 |                     |
     -----------+-+--------------+---------------------+-------
                | |                                 p=t:172.20.20.0/24
                | | 
p:172.20.20.10  | |
t:172.20.20.210 | | t:172.20.20.220          
            +---o-o---+              
            |   IPMP  |               
            +---------+

IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.

Advantages:

• easiest configuration for IPMP in a subnet without a default router.
• very reliable due to the five targets.
  

Disadvantages:

• a subnet without an default router is very rare.
  

1.3 With dedicated hosts acting as test targets with "host-routes"

          +----------------+   +-----+               +-----+                      
          | default router |   | T 1 |     ...       | T 5 |
          +-------o--------=   +--o--+               +--o--+                             
                  |              | p=t:172.20.20.110   | p=t:172.20.20.150                                            
                  |              |                     |        
                  |              |                     |
     -----------+-+--------------+---------------------+-------
                | |                                 p=t:172.20.20.0/24
                | |
p:172.20.20.10  | |
t:172.20.20.210 | | t:172.20.20.220           
            +---o-o---+              
            |   IPMP  |               
            +---------+

Solaris 8 and 9 - Some "host routes" will be defined with a start script. in /etc/rc2.d/S70staticroutes (refer to appendix A).  When IPMP refers to the routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the default router in the routing table. If you have less than five "host routes", the default router (when available) will be used as a probe target too.

Examples:

1. Configuration with host1, host2 ... hostN (with N=5 or N>5), defaultrouter :
   ==> The first five hosts (host1 ... host5) will be defined as target, not the default router
   
   
2. Configuration with less than 5 hosts : for instance, host1, host2, defaultrouter :
   ==> The three systems (host1, host2, defaultrouter) will be defined as target.
   
   The in.mpathd continuously tries to get five probe targets from the routing table but it will NOT send "all hosts" multicasts!
   

Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script. will be necessary.

Advantages:

• The default router is not a single point of failure anymore because you still have some other "host routes" as probe targets.                             
• Higher availability.
  

Disadvantages:

• More administrative work to do.
• Due to static configuration you should regularly check that the probe targets are available.

1.4 Configuration examples for 1.1, 1.2 and 1.3

/etc/hosts
127.0.0.1       localhost       
172.20.20.10    host10 loghost
172.20.20.210   host10-test-qfe0
172.20.20.220   host10-test-qfe4

/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up

/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:
qfe0: flags=9040843 mtu 1500 index 3
        inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
        groupname ipmp0
        ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843 mtu 1500 index 3
        inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4: flags=9040843 mtu 1500 index 4
        inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
        groupname ipmp0
        ether 8:0:20:e8:89:34 

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More Details available in Technical Instruction Document: 1012450.1.
                                    

2.  Production and test interfaces in different IP subnets

If you don't have enough additional IP addresses available, you can configure the IPMP test interfaces to a different IP subnet than your production network (e.g. 192.168., 10. ..). But you have to make sure that there are always enough test partners in that test network to respond to the ICMP probes. In such a configuration in.mpathd will use the test IP addresses only as source address for the outgoing probe packets.

Note: The in.mpathd only checks the test network. If there is no responding probe target in that IP subnet the IPMP group will fail even though the production subnet is still available.

2.1 With default router in production subnet and test subnet

                                      
                                   +----------------+                                   
                                   | default router |                                      
                                   +-------o--------+                                            
                                           | p:172.20.20.1                                            
                                           | t:192.168.1.1                                           
                                           |        
      ----------+-+------------------------+-------------------------                  
                | |                                 p:172.20.20.0/24
                | |                                 t:192.168.1.0/24
p:172.20.20.10  | |                                 
t:192.168.1.210 | | t:192.168.1.220           
            +---o-o---+              
            |   IPMP  |               
            +---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group will send ICMP requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of the /etc/defaultrouter file. There will be NO "all hosts" multicast (224.0.0.1) sent out.

Advantages:

• No additional production IP addresses required for the test interfaces
  

Disadvantages:

• The interface of the default router has to reside in both the production AND test subnet.
• Exceptional configuration of default router.
• All other disadvantages already mentioned in section 1.1
  

2.2 With default router in production subnet net but without default router in test subnet

          +----------------+   +-----+               +-----+                      
          | default router |   | T 1 |     ...       | T 5 |
          +-------o--------=   +--o--+               +--o--+                             
    p:172.20.20.1 |              | p:172.20.20.110     | p:172.20.20.150                                            
                  |              | t:192.168.1.110     | t:192.168.1.150       
                  |              |                     |
     -----------+-+--------------+---------------------+-------
                | |                                 p:172.20.20.0/24
                | |                                 t:192.168.1.0/24
p:172.20.20.10  | |
t:192.168.1.210 | | t:192.168.1.220           
            +---o-o---+              
            |   IPMP  |               
            +---------+

IPMP dynamically determines five arbitrary hosts on the link via an "all hosts" multicast (224.0.0.1). At least, you will need one probe target to get IPMP working but be aware that one probe target is not reliable enough. As long as there are less than five targets available, the in.mpathd daemon will continuously sent out "all hosts" multicasts to get a complete list of five probe targets.

Advantages:

• easiest IPMP configuration if you don't have enough additional IP addresses available in the production subnet.
• very reliable due to the five targets.
  

Disadvantages:

• the probe targets must be available before you can setup the IPMP host.
• more administrative work because you have to setup the probe targets as an additional interface in the test subnet.
  
  Recommendation: Setup an additional logical network interface on target host.
  (e.g. add a new interface to /etc/hostname.qfe0 with the 'addif' option)
  
  If your test partners are on systems which run also IPMP you must add an logical network interface NOT flagged deprecated and NOT flagged nofailover. An address associated with this interface will then be used as source address by responding properly to "all hosts" multicast packets which are used for the automatic IPMP test partner detection. Be aware that Solaris 8 does NOT require the additional logical network interface in the test subnet on the target host. So, in case of an upgrade from Solaris 8 to Solaris[TM] 9 or higher you have to change your configuration.
  
  
• Please refer to 2.5 for a detailed example.

2.3 with dedicated hosts acting as test targets with "host-routes"

          +----------------+   +-----+               +-----+                      
          | default router |   | T 1 |     ...       | T 5 |
          +-------o--------=   +--o--+               +--o--+                             
    p:172.20.20.1 |              | p:172.20.20.110     | p:172.20.20.150                                            
                  |              | t:192.168.1.110     | t:192.168.1.150       
                  |              |                     |
     -----------+-+--------------+---------------------+-------
                | |                                 p:172.20.20.0/24
                | |                                 t:192.168.1.0/24
p:172.20.20.10  | |
t:192.168.1.210 | | t:192.168.1.220           
            +---o-o---+              
            |   IPMP  |               
            +---------+ 

Solaris 8 and 9 - Some "host routes" will be defined with a start script. in /etc/rc2.d/S70staticroutes (refer to appendix A).  When IPMP refers to the routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the "host routes" are before the default router in the routing table. If you have less than five "host routes", the default router (when available in the test subnet) will be used as a probe target too.

Examples:

Please refer to the examples of section 1.3.

Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script. will be necessary.

Advantages:

• test interfaces don't need IP addresses of the production subnet
• all others already mentioned in section 1.3

Disadvantages:

• all others already mentioned in section 1.3
• all others already mentioned in section 2.2

2.4 Configuration examples for 2.1, 2.2 and 2.3

/etc/hosts
127.0.0.1       localhost       
172.20.20.10    host10 loghost
192.168.1.210   host10-test-qfe0
192.168.1.220   host10-test-qfe4

/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up

/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:
qfe0: flags=9040843 mtu 1500 index 3
        inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
        groupname ipmp0
        ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843 mtu 1500 index 3
        inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4: flags=9040843 mtu 1500 index 4
        inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 8:0:20:e8:89:34 

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an active-standby configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More Details available in Technical Instruction Document: 1012450.1.
 

2.5  Configuration example for 2.2 if you use IPMP on the test subnet

/etc/hosts
127.0.0.1       localhost       
#
# IPMP active IP address:
#
172.20.20.10    host10 loghost
#
# IPMP test interfaces
#
192.168.1.210   host10-test-ce0
192.168.1.220   host10-test-ce1
#
# additional active interface in the 'test subnet'
# to be able to respond to probe packets from other 
# IPMP setups (See 2.2). Will also failover.
#
192.168.1.211   host10-prod-ce0

/etc/hostname.ce0
host10 netmask + broadcast + group ipmp0 up \
addif host10-prod-ce0 netmask + broadcast + up \
addif host10-test-ce0 netmask + broadcast + deprecated -failover up

/etc/hostname.ce1
host10-test-ce1 netmask + broadcast + deprecated -failover group ipmp0 up


ifconfig output:
ce0: flags=1000843 mtu 1500 index 2
        inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
        groupname ipmp0
        ether 8:0:20:e2:da:d5
ce0:1: flags=1000843 mtu 1500 index 2
        inet 192.168.1.211 netmask ffffff00 broadcast 192.168.1.255
ce0:2: flags=9040843 mtu 1500 index 2
        inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=9040843 mtu 1500 index 3
        inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 8:0:20:e2:da:d6

A. Start script. for adding static "host routes" on Solaris 8 and 9

----------- Begin of start script. /etc/init.d/ipmp.targets --------------

#!/sbin/sh
# /etc/rc2.d/S70ipmp.targets /etc/init.d/ipmp.targets
# Copyright (c) 2005 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident  "@(#)ipmp.targets     1.0.0    
#
# Edit the following IPMP test TARGETS to suit your needs.
# To install:
# 1) cp ipmp.targets /etc/init.d    
# 2) perform. edits on the script. as required (e.g: add TARGETS)
# 3) chmod 744 /etc/init.d/ipmp.targets
# 4) chown root:sys /etc/init.d/ipmp.targets
# 5) ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
# 
TARGETS="192.168.85.117 192.168.85.127 192.168.85.137"

case "$1" in
        'start')
            /usr/bin/echo "Adding static routes for use as IPMP targets"
		for target in $TARGETS; do
	  /usr/sbin/route add -host $target $target
		done
                  ;;
        'stop')
              /usr/bin/echo "Removing static routes for use as IPMP targets"
		 for target in $TARGETS; do
		/usr/sbin/route delete -host $target $target
		 done
                  ;;
esac   
----------- End of start script. /etc/init.d/ipmp.targets --------------

See also:
System Administration Guide: IP Services > Configuring IPMP Groups  > Configuring Target Systems

B. Adding persistent "host routes" via 'route -p' option on Solaris 10

Example:
In this example, the active interface on the system is 10.8.118.104
The test host targets are 5 systems that are external to this physical system.
The IP address used for the default router is the same IP address that was in the /etc/defaultrouter file.

# rm /etc/defaultrouter

# route -p add -host 10.8.118.1 10.8.118.1 -static
# route -p add -host 10.8.118.3 10.8.118.3 -static
# route -p add -host 10.8.118.40 10.8.118.40 -static
# route -p add -host 10.8.118.41 10.8.118.41 -static
# route -p add -host 10.8.118.193 10.8.118.193 -static
# route -p add default 10.8.118.248 -static

Note: If you define 5 host routes then the default router will NOT be used as a probe target, if you define less than 5 host routes the default router will be used as an additional probe target.

C. Summary

It's not easy to give a general recommendation because it depends on the network infrastructure which the customers have. Therefore you have to discuss the various possibilities with your customer.  Maybe the most used setups are 1.1 and 2.2.

 

Product
Solaris 8 Operating System
Solaris 9 Operating System
Solaris 10 Operating System