Help
Did this article help solve your problem?  Select Yes No Does Not Apply   Would you recommend this document to others?  Select Yes No Not Sure   
            

  
TIP:  Click help for a detailed explanation of this page.
Bookmark Go to End

Subject:  Data Gathering for Troubleshooting RAC and CRS issues
  Doc ID:  Note:289690.1 Type:  TROUBLESHOOTING
  Last Revision Date:  19-DEC-2005 Status:  PUBLISHED





This note includes links to information that will be helpful in providing up from to Oracle Support when logging your TAR.   Depending on the type of problem you are having, Oracle Support may require different types of disagnostics information in order to resolve the issue.   The information below is not required for logging a TAR – however, gathering the information now, or being prepared to gather the information when requested, will help to speed the resolution of your TAR.





1. ALL RAC issues

2. CRS Installation Problems

3. Other CRS Problems/Errors

4. Node Reboots

5. RAC/RDBMS Installation Issues

6. Debugging Java Configuration Tools: DBCA, NETCA, VIPCA, DBUA and SRVCTL

7. Instance Evictions (ORA-29740)

8. Hangs, Spins and Deadlocks

9. RAC Performance Issues

10. ASM Errors/Problems

11. Instance Crashes, Process Crashes, ORA-600 or ORA-7445

12. Other





Generic Data Gathering for All RAC Issues
RDA:
Please provide a current RDA from each node in the cluster.  Older runs of RDA  (i.e. if RDA was run from a previous TAR that was logged a few days or weeks back) will not be able to provide current log information, or performance information, so it is a good idea to have current RDA output from all cluster nodes in a RAC environment.  



Information on how to download and run RDA can be found in the following Metalink Note:

                                    Note 175853.1 “Remote Diagnostics Agent (RDA)”

RACDIAG:
In addition to the RDA, the RACDIAG script. will gather information specific to the performance and configuration of your RAC cluster.  Uploading output from the RACDIAG script. can also assist support in the timely resolution of your issue.   The RACDIAG script. is available on Metalink under:

Note 135714.1 - Script. to Collect RAC Diagnostic Information (racdiag.sql)

Additional Info:
In addition to the information gathered by the RDA and RACDIAG scripts, the following information will also be helpful in troubleshooting your issue:

System messages files from all nodes – depending on your platform, these logs are found in different locations:

Windows: Save Application Log and System Log as txt files Using Event Viewer

Linux: /var/log/messages

Sun: /var/adm/messages

HP-UX: /var/adm/syslog/syslog.log

Tru64: /var/adm/messages

IBM: /bin/errpt -a  (redirect this to a file called messages.out or something similar)





End of Generic Section

=====================================









Data Gathering for Diagnosing CRS Installation Issues


For Diagnosing CRS Installation issues, appropriate data to provide to support includes



CLUVFY Output

Trace output from the Oracle Installer
All Installation Logs

Trace output of root.sh



CLUVFY Output.  Please refer to the following note for running the Cluster Verification Utility


Note 339939.1 - Running Cluster Verification Utility to Diagnose Install Problems



Trace Output from Oracle Installer:
On Unix/Linux, launch the installer with tracing turned on – i.e.:

./runInstaller -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2



On Windows launch the installer using ‘setup.exe’ with:

setup.exe -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2



Logs will be written to the ORACLE_BASE/oraInventory/logs on Unix/Linux

Logs will be written to the ‘C:\Program Files\oracle\Inventory\logs’ on Windows



See also the following Note:

                                    Note 269837.1 – Tracing the OUI from 9.2.0.5 to 10g



All Installation Logs
Logs will be written to the ORACLE_BASE/oraInventory/logs on Unix/Linux

Logs will be written to the ‘C:\Program Files\oracle\Inventory\logs’ on Windows



Please upload all recent/applicable logs from this drectory.

Trace output from root.sh
For tracing root.sh, please refer to the following note:

Note 240001.1: 10g RAC: Troubleshooting CRS Root.sh Problems





End of CRS Install Section

=====================================









Data Gathering for CRS Error/Problems


For troubleshooting CRS errors that are not related to installation, please collect the following information:



Note 272332.1    - CRS 10g Diagnostic Collection Guide

Note 330358.1    - CRS 10g R2 Diagnostic Collection Guide





Please also gather the system messages files from all nodes – depending on your platform, these logs are found in different locations:

Windows: Save Application Log and System Log as txt files Using Event Viewer

Linux: /var/log/messages

Sun: /var/adm/messages

HP-UX: /var/adm/syslog/syslog.log

Tru64: /var/adm/messages

IBM: /bin/errpt -a   (redirect this to a file called  messages.out)





End of CRS Error Section

=====================================









Data Gathering for Node Reboots


For troubleshooting of Node Reboots when running Oracle Clusterware, please collect the following information:



Note 272332.1    - CRS 10g R1 Diagnostic Collection Guide

Note 330358.1    - CRS 10g R2 Diagnostic Collection Guide







In addition to the CRS diagnostic collection guide for your particular version, please gather information from OS Watcher, if available on your platform.

OS Watcher is documented in:



                                    Note 301137.1 – OS Watcher User Guide



   Note that OS Watcher currently does not support the Windows Platform.  



Please also gather the system messages files from all nodes – depending on your platform, these logs are found in different locations:

Windows: Save Application Log and System Log as txt files Using Event Viewer

Linux: /var/log/messages

Sun: /var/adm/messages

HP-UX: /var/adm/syslog/syslog.log

Tru64: /var/adm/messages

IBM: /bin/errpt - a  (redirect this to a file called messages.out or something similar)





End of Node Reboot Section

=====================================









Data Gathering for RAC/RDBMS Installation Issues


For Diagnosing CRS Installation issues, appropriate data to provide to support includes:



CLUVFY Output

Trace output from the Oracle Installer

All Installation Logs

Output from lsnodes or olsnodes  



Note 339939.1 - Running Cluster Verification Utility to Diagnose Install Problems



Trace Output from Oracle Installer:
On Unix/Linux, launch the installer with tracing turned on – i.e.:

./runInstaller -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2



On Windows launch the installer using ‘setup.exe’ with:

setup.exe -J-DTRACING.ENABLED=true -J-DTRACING.LEVEL=2





See also the following Note:

Note 269837.1 – Tracing the OUI from 9205 to 10g





All Installation Logs


Logs will be written to the ORACLE_BASE/oraInventory/logs on Unix/Linux

Logs will be written to the ‘C:\Program Files\oracle\Inventory\logs’ on Windows



Output of lsnodes or olsnodes


Please Provide lsnodes or olsnodes output, as appropriate:

For Oracle version 9.2:

a).  Start the installer

b).  cd to /tmp directory after the product selection screen

c).  do a ‘find’  for ‘lsnodes’, and execute from that location.

d). upload output to support



For Oracle10g, run olsnodes from the ORA_CRS_HOME/bin directory:

a). $ORA_CRS_HOME/bin/olsnodes

b). upload output to support





End of RDBMS Install Section

=====================================









Data Gathering for Java Configuration Tools - i.e. DBCA, NETCA, VIPCA, DBUA and SRVCTL


For Troubleshooting problems with Java tools, please refer to the following notes for gathering additional trace information to upload to support:

Note 178683.1  - Tracing GSD, SRVCTL, GSDCTL and SRVCONFIG

Note 188134.1  - Tracing the Database Configuration Assistant (DBCA)





End of Java Config Tools Section

=====================================









Data Gathering for Instance Evictions (ORA-29740)


For diagnosing Instance Eviction Issues – i.e. ORA-29740 errors, please refer to the following notes for data to collect:



Note 219361.1    - Troubleshooting ORA-29740 in a RAC Environment   - At minimum, be sure to upload:

a). Each Instance’s LMON, LCK, LMD and LMS trace files

b). Each Instance’s Alert Log

c). Statspack or AWR Reports from each instance leading up to the eviction

Note 301137.1    - OS Watcher User Guide  - upload output of OS Watcher

Note 272332.1    - CRS 10g R1 Diagnostic Collection Guide

Note 330358.1    - CRS 10g R2 Diagnostic Collection Guide



For Windows Platforms, Performance Data should be Gathered as Discussed in the following Note:

Note 297498.1  - Resolving Instance Evictions on Windows Platforms



In addition, please provide System messages files from all nodes – depending on your platform, these logs are found in different locations:

Windows: Save Application Log and System Log as txt files Using Event Viewer

Linux: /var/log/messages

Sun: /var/adm/messages

HP-UX: /var/adm/syslog/syslog.log

Tru64: /var/adm/messages

IBM: /bin/errpt - a  (redirect this to a file called messages.out or something similar)





End of Instance Evictions Section

=====================================









Data Gathering for Hangs, Deadlocks, and Process Spins:


Data to Gather for Hanging Issues, Deadlocks or Process Spins include the following information:



Note 135714.1 - Script. to Collect RAC Diagnostic Information (racdiag.sql)

Note 301137.1 – OS Watcher User Guide



  Note that OS Watcher currently does not support the Windows Platform



Database Performance data for all nodes (If the database is not responsive, this may need to be taken after the fact.)

1.  9i: Statspack reports from all nodes from within 2 hours before the problem at 1 hour intervals (or less).  

2.  10g: AWR and ADDM reports from all nodes from within 2 hours before the problem at 1 hour intervals (or less).



               Depending on the version, please gather either AWR/ADDM reports, or Statspack Reports, as per the following notes:  

Note 94224.1    - FAQ- Statstyle="mso-spacerun: yes">     

in this note, please gather the information in Section 5 - What Oracle Support needs to diagnose a performance problem in Oracle 10g



In addition, please provide System messages files from all nodes – depending on your platform, these logs are found in different locations:

Windows: Save Application Log and System Log as txt files Using Event Viewer

Linux: /var/log/messages

Sun: /var/adm/messages

HP-UX: /var/adm/syslog/syslog.log

Tru64: /var/adm/messages

IBM: /bin/errpt a  (redirect this to a file called messages.out or something similar)





End of Hang/Deadlock/Spin Section

=====================================









Data Gathering for RAC Performance Issues


Data to Gather for RAC Performance issues includes the following information:



Note 135714.1 - Script. to Collect RAC Diagnostic Information (racdiag.sql)

Note 301137.1 – OS Watcher User Guide



  Note that OS Watcher currently does not support the Windows Platform



Database Performance data for all nodes (If the database is not responsive, this may need to be taken after the fact.)

1.      10g: AWR and ADDM reports from all nodes from within 2 hours before the problem at 1 hour intervals (or less).

2.      .  9i: Statspack reports from all nodes from within 2 hours before the problem at 1 hour intervals (or less).



Note 94224.1    - FAQ- Statspack Complete Reference

Note 276103.1  - Performance Tuning Using 10g Advisors and Manageability Features -

in this note, please gather the information in Section 5 - What Oracle Support needs to diagnose a performance problem in Oracle 10g





End of RAC Performance Section

=====================================









Data Gathering for ASM Errors/problems


For troubleshooting ASM Issues, please gather data per the following Metalink Note:



Note 340417.1 - Data Gathering for Troubleshooting ASM Issues





End of ASM Section

=====================================



Data Gathering for Instance Crashes, Process Crashes, ORA-600’s or ORA-7445’s


For troubleshooting ORA-600 errors or ORA-7445 errors, please refer to the following Metalink Notes for data gathering:



Note 146581.1  - How to deal with ORA-600 Internal Errors

Note 211909.1 - Customer Introduction to ORA-7445 Errors





End of ORA-600/ORA-7445 Section

=====================================









Other


RDA:
Please provide a current RDA from each node in the cluster.  Older runs of RDA  (i.e. if RDA was run from a previous TAR that was logged a few days or weeks back) will not be able to provide current log information, or performance information, so it is a good idea to have current RDA output from all cluster nodes in a RAC environment.  



Information on how to download and run RDA can be found in the following Metalink Note:

                                    Note 175853.1 “Remote Diagnostics Agent (RDA)”





The End

=====================================