PURPOSE
=========

Below is a discussion of how Data Guard reacts and recovers when a network
connection between the primary and the standby is disconnected or broken.  This
discussion applies to both physical and logical standby databases. 


Data Guard and Network Disconnects:
=========================================

In a Data Guard configuration the primary establishes a network connection to the 
standby by either the LGWR or ARCH processes.  When the standby recieves the network
connection request from the primary, the listener on the standby site spawns the RFS
process.  This RFS process on the standby reads from the networks and sends
acknowledgement messages back to the primary when done processing any request.

During normal Data Guard operations, when redo data is transmitted from the primary 
to the standby, network messages are initiated from the primary database (the network
`client'), and always acknowledged by the standby database (the network `server'). 
In this case, the LGWR and ARCH processes are the network clients, and the RFS 
process is the network server. 

Consider the simple scenario where the network between the primary and standby 
systems is disconnected. This results in what is known as a dead connection. A dead 
connection indicates that there is no physical connection, but the connection appears
to still be there to the processes on each system. 

When the LGWR process attempts to send a new message to the RFS process over the 
dead connection it will notice that the connection appears to be broken.  At this
point LGWR will wait on the TCP layer to timeout on the network session between the
primary and standby before establishing that network connectivity has indeed been lost.
The TCP timeout, as defined by TCP kernel parameter settings, is key to how long either 
LGWR or ARCH will remain in a wait state before abandoning the network connection.
On some platforms the default for TCP timeout can be as high as 2 hours.
 
As of 9.2 new functionality was added to Data Guard so as to better control LGWR 
timeouts for broken network connections. The [NO]MAX_FAILURE, [NO]REOPEN and 
[NO]NET_TIMEOUT attributes to the LOG_ARCHIVE_DEST_n parameter were added to give
LGWR the desired flexibility to control timeout intervals and the number of retries
when a network connection is not responding.

On the standby side, the RFS process is always synchronously waiting for new information
to arrive from the LGWR or ARCH process on the primary.  The RFS process that is
doing the network read operation is blocked until more data arrives or until the 
operating system's network software determines that the connection is dead.

Once the RFS process receives notification of the dead network connection, it will 
terminate itself. However, until such time as the RFS process terminates itself, it 
will retain lock information on the archivelog on the standby site, or the standby 
redo log, whose redo information was being received from the primary database. 

Any attempt to perform a failover using the RECOVER MANAGED STANDBY DATABASE FINSIH
command will fail while the RFS process maintains a lock on the standby redo
log.  The recover command will fail with the following errors:

    ORA-00283: recovery session canceled due to errors
    ORA-00261: log 4 of thread 1 is being archived or modified
    ORA-00312: online log 4 thread 1: '/database/mtsmith/920DG/srl1.dbf'

One must wait for either the operating system network software to clean up the dead
connection or kill the RFS process before the failover attempt will succeed.  One 
method to decrease the time for the operating system network software to clean up the
dead connection is the use of Oracle's Dead Connection Detection feature.

With Oracle's Dead Connection Detection feature Oracle Net periodically sends a
network probe to verify that a client/server connection is still active. This ensures
that connections are not left open indefinitely due to an abnormal client termination.
If the probe finds a dead connection or a connection that is no longer in use, it
returns an error that causes the RFS process to exit. However, as we are still 
dependant on the operation system network software for timeouts and retries it can
take Dead Connection Detection up to 9 minutes to terminate the RFS network connection.
The time can be expressed as:

     time to kill dead connection=
       SQLNET.EXPIRE_TIME (Dead Connection Detection)
     + tcp_ip_abort_interval (default=480000, 8 mins. on Solaris) 
     + tcp_rexmit_interval_max (def.=1 min)

Changing the tcp_keep_alive setting to a lower number will improve this timing 
but is usually not feasible to apply to the whole system. 

Once the network problem is resolved, and the primary database processes are again 
able to establish network connections to the standby database, a new RFS process 
will automatically be spawned on the standby database for each new network connection.
These new RFS processes will resume the reception of redo data from the primary 
database.