#### Recovering from a failed takeover
#### 不用重起HEARTBEAT,可以对单个资源SCORE恢复

P.275

有两种情况:
## CASE1: NODE1.SCORE > 0 , NODE2.SCORE=-100000/-INFINITY
(处于这个值,是因为MON发现资源失败,SCORE-=200,重启(如果重启成功SCORE+=100),
结果重启失败,SCORE=-100000,crm_failcount=INFINITY. HB不再重复重启,直接切换)
## CASE2: 两NODE的SCORE=负数. 资源再也不会发生切换..

[@more@]

### CASE1的处理:
=================
## 查看目录资源情况
# crm_mon -V -1

## 查看SCORE
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"
ptest[15710]: 2009/11/19_13:17:05 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql3: -1000000
ptest[15710]: 2009/11/19_13:17:05 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql4: 150

## 查看失败的次数
# crm_failcount -G -U mysql3 -r mysql
name=fail-count-mysql value=INFINITY
# crm_failcount -G -U mysql4 -r mysql
name=fail-count-mysql value=3

##========== 下面开始恢复

## 让资源脱离管理,Place the resource in the unmanaged mode
# crm_resource -r mysql -p is_managed -v false
# crm_mon -V -1

Resource Group: group_mysql
mysql (lsb:mysql): Started mysql4 (unmanaged)

## 重置failcount
# crm_failcount -U mysql3 -r mysql -D
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"

ptest[15894]: 2009/11/19_13:19:39 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql3: -1000000
ptest[15894]: 2009/11/19_13:19:39 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql4: 1000000

## 查看
# crm_failcount -U mysql3 -r mysql -G

-- name=fail-count-mysql value=0

## 清除错误消息 Remove the error messages:
## set SCORE =0 For NODE mysql3
# crm_resource -H mysql3 -r mysql -C
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"

ptest[15989]: 2009/11/19_13:20:56 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql3: 0
ptest[15989]: 2009/11/19_13:20:56 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql4: 1000000

## 让资源置管理模式: Place the resource in managed mode
## 当crm_mon里找不到这个资源时,也可以用这句恢复.
# crm_resource -r mysql -d is_managed
# crm_mon -V -1
###这里查看,就没有:(unmanaged)

# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"

ptest[15710]: 2009/11/19_13:17:05 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql3: -1000000
ptest[15710]: 2009/11/19_13:17:05 debug: debug2: dump_node_scores: group_color: mysql allocation score on mysql4: 150

## 这时mysql3的score会归到初始值. 当然如果该值大于另一台的SCORE,则发生切换

# crm_resource -W -r mysql

### CASE2的处理:
=================
## 查看目录资源情况 (看不到资源MYSQL)
# crm_mon -V -1

## 查看SCORE
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"
ptest[16987]: 2009/11/19_13:32:48 debug: debug2: dump_node_scores: Pre-allloc: mysql allocation score on mysql3: -100
ptest[16987]: 2009/11/19_13:32:48 debug: debug2: dump_node_scores: Pre-allloc: mysql allocation score on mysql4: -150

## 查看失败的次数
# crm_failcount -G -U mysql3 -r mysql
name=fail-count-mysql value=INFINITY
# crm_failcount -G -U mysql4 -r mysql
name=fail-count-mysql value=3

##========== 下面开始恢复

## 重置failcount
# crm_failcount -U mysql3 -r mysql -D
# crm_failcount -U mysql4 -r mysql -D
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"
ptest[17012]: 2009/11/19_13:34:51 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql3: 0
ptest[17012]: 2009/11/19_13:34:51 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql4: 0

## 查看
# crm_failcount -U mysql3 -r mysql -G
# crm_failcount -U mysql4 -r mysql -G
name=fail-count-mysql value=0

## 清除错误消息 Remove the error messages:
## set SCORE =0 For NODE mysql3
# crm_resource -H mysql3 -r mysql -C
# crm_resource -H mysql4 -r mysql -C
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"
ptest[17042]: 2009/11/19_13:35:58 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql3: 0
ptest[17042]: 2009/11/19_13:35:58 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql4: 0

## 让资源置管理模式: Place the resource in managed mode
## 当crm_mon里找不到这个资源时,也可以用这句恢复.
# crm_resource -r mysql -d is_managed
# 2>&1 /usr/heartbeat/sbin/ptest -LVVVVVVVV|grep -E "dump_node_scores"

ptest[17048]: 2009/11/19_13:36:15 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql3: 400
ptest[17048]: 2009/11/19_13:36:15 debug: debug3: dump_node_scores: Pre-allloc: mysql allocation score on mysql4: 350

# crm_mon -V -1
###这里你又可以看到MYSQL资源了. 按SCORE看应该跑在MYSQL3上.

# crm_resource -W -r mysql
resource mysql is running on: mysql3

ITPub博客

Heartbeat Score的重置

最新文章