RAC Troubleshoot

###############################################################

1.CRS log directory
 $ORA_CRS_HOME/crs/log   => CRS resource
 $ORA_CRS_HOME/crs/init
 $ORA_CRS_HOME/css/log
 $ORA_CRS_HOME/css/init
 $ORA_CRS_HOME/evm/log
 $ORA_CRS_HOME/evm/init
 $ORA_CRS_HOME/srvm/log
 $ORA_CRS_HOME/log

2.DB log directory
 $ORACLE_HOME/admin/udump
 $ORACLE_HOME/admin/cdump

3.Crsctl Usage: 
 crsctl check  crs          - checks the viability of the CRS stack
 crsctl check  cssd         - checks the viability of CSS
 crsctl check  crsd         - checks the viability of CRS
 crsctl check  evmd         - checks the viability of EVM
 crsctl set    css - sets a parameter override
 crsctl get    css - gets the value of a CSS parameter
 crsctl unset  css - sets CSS parameter to its default
 crsctl query  css votedisk    - lists the voting disks used by CSS
 crsctl add    css votedisk - adds a new voting disk
 crsctl delete css votedisk - removes a voting disk
 crsctl enable  crs    - enables startup for all CRS daemons
 crsctl disable crs    - disables startup for all CRS daemons
 crsctl start crs  - starts all CRS daemons.
 crsctl stop  crs  - stops all CRS daemons. Stops CRS resources in case of cluster.
 crsctl start resources  - starts CRS resources.
 crsctl stop resources  - stops  CRS resources.
 crsctl debug statedump evm  - dumps state info for evm objects
 crsctl debug statedump crs  - dumps state info for crs objects
 crsctl debug statedump css  - dumps state info for css objects
 crsctl debug log css [module:level]{,module:level} ...
                             - Turns on debugging for CSS
 crsctl debug trace css - dumps CSS in-memory tracing cache
 crsctl debug log crs [module:level]{,module:level} ...
                             - Turns on debugging for CRS
 crsctl debug trace crs - dumps CRS in-memory tracing cache
 crsctl debug log evm [module:level]{,module:level} ...
                             - Turns on debugging for EVM
 crsctl debug trace evm - dumps EVM in-memory tracing cache
 crsctl debug log res turns on debugging for resources
 crsctl query crs softwareversion [] - lists the version of CRS software installed
 crsctl query crs activeversion - lists the CRS software operating version
 crsctl lsmodules css - lists the CSS modules that can be used for debugging
 crsctl lsmodules crs - lists the CRS modules that can be used for debugging
 crsctl lsmodules evm - lists the EVM modules that can be used for debugging

   If necesary any of these commands can be run with additional tracing by adding a "trace" argument at the very front.
   Example: crsctl trace check css
 

4.ocrconfig
Name:
        ocrconfig - Configuration tool for Oracle Cluster Registry.
Synopsis:
        ocrconfig [option]
        option:
                -export [-s online]      - Export cluster register contents to a file
                -import                   - Import cluster registry contents from a file
                -upgrade [ []]
                                                    - Upgrade cluster registry from previous version
                -downgrade [-version ]
                                                    - Downgrade cluster registry to the specified version
                -backuploc                 - Configure periodic backup location
                -showbackup                         - Show backup information
                -restore                  - Restore from physical backup
                -replace ocr|ocrmirror [] - Add/replace/remove a OCR device/file
                -overwrite                          - Overwrite OCR configuration on disk
                -repair ocr|ocrmirror     - Repair local OCR configuration
                -help                               - Print out this help information
Note:
        A log file will be created in
        $ORACLE_HOME/log//client/ocrconfig_.log. Please ensure
        you have file creation privileges in the above directory before running this tool.

5.ocrdump

ocrdump -help
Name:
        ocrdump - Dump contents of Oracle Cluster Registry to a file.
Synopsis:
        ocrdump [|-stdout] [-backupfile ] [-keyname ] [-xml] [-noheader]
Description:
        Default filename is OCRDUMPFILE. Examples are:
        prompt> ocrdump
        writes cluster registry contents to OCRDUMPFILE in the current directory
        prompt> ocrdump MYFILE
        writes cluster registry contents to MYFILE in the current directory
        prompt> ocrdump -stdout -keyname SYSTEM
        writes the subtree of SYSTEM in the cluster registry to stdout
        prompt> ocrdump -stdout -xml
        writes cluster registry contents to stdout in xml format
Notes:
        The header information will be retrieved based on best effort basis.
        A log file will be created in 
        ORACLE_HOME/log//client/ocrdump_.log. 
        Make sure you have file creation privileges in the above directory before

6.srvctl
Usage: srvctl