线上HBase集群突然一台regionserver的网卡流量打满了，堵住了所有的request，我们赶紧把这台regionserver下掉，但是不解决问题，接着另外一台regionserver的网卡也打满了，就这样搞了一个下午，HBase集群也不能稳定的提供服务，丢失了1/3的流量。

最后终于发现了，是因为一个超大的scan导致的网络流量突增。为此，我们需要监控这种慢操作，类似监控MySQL的慢查询一样。

开发了相应的nagios检测脚本：

#!/usr/bin/python
#
# Copyright (c) 2014/2015 Qunar and/or its affiliates. All rights reserved.
#

# -*- coding: utf-8 -*-

import sys
import os
import urllib2
import json
import time

E_OK=0
E_WARNING=1
E_CRITICAL=2
E_UNKNOWN=3

WARNING_TIME = 60
CRITICAL_TIME = 300

url = 'http://127.0.0.1:60030/rs-status?format=json&filter=rpc'

def getHBaseProcesslist():
response = urllib2.urlopen(url)
content = response.read()
return json.loads(content)

def checkProcesslist():
try:
pls = getHBaseProcesslist()

now = int(time.time())

for pl in pls:
if (pl['state'] == 'RUNNING'):
runtime = now - pl['rpcCall']['starttimems']/1000
if (runtime > CRITICAL_TIME):
print "CRIT. Thread running %d seconds, method:%s, client:%s" % (runtime, pl['rpcCall']['method'], pl['rpcCall']['clientaddress'])
return E_CRITICAL
elif (runtime > WARNING_TIME):
print "WARN. Thread running %d seconds, method:%s, client:%s" % (runtime, pl['rpcCall']['method'], pl['rpcCall']['clientaddress'])
return E_WARNING
except Exception, e:
print str(e)
return E_UNKNOWN

print "OK."
return E_OK

if __name__ == '__main__':
ret = checkProcesslist()
sys.exit(ret)

转自:Louis Hust's Blog

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/15498/viewspace-1969291/，如需转载，请注明出处，否则将追究法律责任。

ITPub博客

HBase大Scan监控

最新文章