ORA-16166 LGWR network server failed to send remote message
Cause: The LGWR network server could not communicate with the remote host.
Action: The alert log contains more problem-specific information.

An example query is:

SQL> alter session set nls_date_format = 'MON-DD-YYYY HH24:MI:SS';
Session altered.
SQL> column MESSAGE format a50
SQL> select timestamp, message from v$dataguard_status where error_code=16166;

TIMESTAMP            MESSAGE
-------------------- --------------------------------------------------
JUN-10-2004 13:14:04 LGWR: I/O error 16166 archiving log 2 to 'Primary'
JUN-10-2004 13:19:22 LGWR: I/O error 16166 archiving log 2 to 'Primary'

The above query shows there were two instances where LGWR ASYNC failed. More information on each of these events is available in the alert log of the primary database. For example, in the alert log detail for the second failure is:

Thu Jun 10 13:19:22 2004
Timing out on NetServer 0 prod=80,cons=8,threshold=64
ORA-16166: LGWR timed out on Network Server 0 due to buffer full condition.
No action is required since the log file transfer will be attempted via ARCH
Thu Jun 10 13:19:22 2004
Errors in file /private1/oracle/OraHome92/admin/Physical/bdump/physical_lgwr_3597.trc:
ORA-16166: LGWR network server failed to send remote message
LGWR: I/O error 16166 archiving log 2 to 'Primary'
Thu Jun 10 13:19:22 2004
Errors in file /private1/oracle/OraHome92/admin/Physical/bdump/physical_lgwr_3597.trc:
ORA-16166: LGWR network server failed to send remote message

This alert log message is typical of ORA-16166 errors for Data Guard configurations using LGWR ASYNC. Note from the information in the alert log that: LGWR timed out on Network Server 0 due to buffer full condition. The primary database generated redo faster than LGWR could ship until finally reaching the point where the ASYNC buffer became full and LGWR abandoned shipping redo to the standby. 

The default setting for the LGWR ASYNC buffer is 1MB. The buffer size can be increased to a maximum of 10MB (in Oracle 10g this the maximum value has been increased to 50MB). The increased buffer is only used if it is absolutely needed. This means that increasing the buffer size has no impact on how quickly LGWR will ship redo to the standby, it always ships as quickly as the network will allow. If there is a frequent occurrence of LGWR failing due to a buffer full condition, consider increasing the size of the ASYNC buffer by using the following command:

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=STANDBYTNS LGWR ASYNC=20480';

Also note that the primary server continues processing transactions in cases where LGWR abandons shipping to the standby server, therefore no immediate action is required by the DBA. Data Guard automatically resolves the resulting gap (a situation where there are archive logs on the primary contain redo that has not been shipped to the standby) using the ARCH process. This can be seen in the same alert log message above: No action is required since the log file transfer will be attempted via ARCH. At the next log switch, LGWR will attempt to reconnect to the standby server. When successful, LGWR begins shipping the then-current redo stream. When ARCH completes the process of resolving the gap, the primary and standby servers are back in sync.