今天在例行数据库检查时，发现 日志里有关于时间同步问题的日志信息。
      报错日志如下:
2014-07-04 13:40:33.301
[ctssd(15137)]CRS-2409:The clock on host rac-1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2014-07-04 14:10:34.009
[ctssd(15137)]CRS-2409:The clock on host rac-1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
1、报错信息问题的分析
1）操作系统的NTPD服务处于启动状态，ctssd就不合作，但是只要ctssd服务启动，正常情况下应该处于观察模式。
2）当前OS的NTPD的服务正在运行，并且ctssd不能处于观察模式运行


2、排除过程
1）检查两个节点时间是否存在差异
    ssh rac-2  date
    date
2)检查OS的NTPD服务
    # service ntpd status
    ntpd is stopped
3）检查ctssd的运行情况
    su  -  grid
  crsctl  stat  res  ora.ctssd  -init
4) 借助crs的cluvfy工具诊断ctss错误的原因
    cluvfy   comp  clocksync  -n  all  -verbose
   cd /oracle/app/11.2/grid/bin/
 ./cluvfy comp clocksync -n all -verbose




Verifying Clock Synchronization across the cluster nodes 




Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed




Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status                  
  ------------------------------------  ------------------------
  rac-2                                 passed                  
  rac-1                                 passed                  
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed


Check CTSS state started...
Check: CTSS state
  Node Name                             State                   
  ------------------------------------  ------------------------
  rac-2                                 Observer                
  rac-1                                 Observer                
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...


NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed


Checking daemon liveness...


Check: Liveness for "ntpd"
  Node Name                             Running?                
  ------------------------------------  ------------------------
  rac-2                                 no                      
  rac-1                                 no                      
Result: Liveness check failed for "ntpd"
PRVF-5494 : The NTP Daemon or Service was not alive on all nodes
PRVF-5415 : Check to see if NTP daemon or service is running failed
Result: Clock synchronization check using Network Time Protocol(NTP) failed


PRVF-9652 : Cluster Time Synchronization Services check failed


Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes. 


5)检查OS层ntpd的配置
   /etc/ntp.conf
   /etc/rc.tcpip


3、问题的处理方法（两个节点都执行）
1）停用ctssd （grid用户执行）
    crsctl  stop  resource ora.ctssd  -init
2) 停止ntpd (root用户执行）
    stopsrc  -s  xntpd
3) 启动ntpd,加入”-X“ 模式 （root用户执行，启动需几分钟）
   startsrc  -s xntpd  -a  "-X"
4) 启动ctssd （grid 用户执行）
    crsctl  start  resource  ora.ctssd  -init