一、背景介绍
JedisClusterPipeline使用
JedisClusterPipline jedisClusterPipline = redisService.clusterPipelined();List<Object> response;try { for (String key : keys) { jedisClusterPipline.hmget(key, VALUE1, VALUE2); } // 获取结果 response = jedisClusterPipline.syncAndReturnAll();} finally { jedisClusterPipline.close();}
二、故障现场记录
三、故障过程分析
-
线程一直等待连接而没有被中断。 -
线程获取不到连接。
3.1 线程一直等待连接而没有被中断原因分析
org.apache.commons.pool2.impl.GenericObjectPool#borrowObject(long)方法下。
public T borrowObject(long borrowMaxWaitMillis) throws Exception {...PooledObject<T> p = null;// 获取blockWhenExhausted配置项，该配置默认值为trueboolean blockWhenExhausted = getBlockWhenExhausted();boolean create;long waitTime = System.currentTimeMillis();while (p == null) {create = false;if (blockWhenExhausted) {// 从队列获取空闲的对象，该方法不会阻塞，没有空闲对象会返回nullp = idleObjects.pollFirst();// 没有空闲对象则创建if (p == null) {p = create();if (p != null) {create = true;}}if (p == null) {// borrowMaxWaitMillis默认值为-1if (borrowMaxWaitMillis < 0) {// 线程栈快照里所有的dubbo线程都卡在这里，这是个阻塞方法，如果队列里没有新的连接会一直等待下去p = idleObjects.takeFirst();} else {// 等待borrowMaxWaitMillis配置的时间还没有拿到连接的话就返回nullp = idleObjects.pollFirst(borrowMaxWaitMillis,TimeUnit.MILLISECONDS);}}if (p == null) {throw new NoSuchElementException("Timeout waiting for idle object");}if (!p.allocate()) {p = null;}}...}updateStatsBorrow(p, System.currentTimeMillis() - waitTime);return p.getObject();}
3.2 线程获取不到连接原因分析
获取不到连接无非两种情况：
连不上Redis，无法创建连接连接池里的所有连接都被占用了，无法获取到连接
创建连接
private PooledObject<T> create() throws Exception {int localMaxTotal = getMaxTotal();long newCreateCount = createCount.incrementAndGet();if (localMaxTotal > -1 && newCreateCount > localMaxTotal ||newCreateCount > Integer.MAX_VALUE) {createCount.decrementAndGet();return null;}final PooledObject<T> p;try {// 创建redis连接，如果发生超时会抛出异常// 默认的connectionTimeout和soTimeout都是2秒p = factory.makeObject();} catch (Exception e) {createCount.decrementAndGet();// 这里会把异常继续往上抛出throw e;}AbandonedConfig ac = this.abandonedConfig;if (ac != null && ac.getLogAbandoned()) {p.setLogAbandoned(true);}createdCount.incrementAndGet();allObjects.put(new IdentityWrapper<T>(p.getObject()), p);return p;}
猜想二：是不是业务代码没有归还Redis连接?
JedisClusterPipeline使用
JedisClusterPipline jedisClusterPipline = redisService.clusterPipelined();List<Object> response;try { for (String key : keys) { // 申请连接，内部会先调用JedisClusterPipeline.getClient(String key)方法获取连接 jedisClusterPipline.hmget(key, VALUE1, VALUE2); // 获取到了连接，缓存到poolToJedisMap } // 获取结果 response = jedisClusterPipline.syncAndReturnAll();} finally { // 归还所有连接 jedisClusterPipline.close();}
JedisClusterPipeline部分源码
public class JedisClusterPipline extends PipelineBase implements Closeable {private static final Logger log = LoggerFactory.getLogger(JedisClusterPipline.class);// 用于记录redis命令的执行顺序private final Queue<Client> orderedClients = new LinkedList<>();// redis连接缓存private final Map<JedisPool, Jedis> poolToJedisMap = new HashMap<>();private final JedisSlotBasedConnectionHandler connectionHandler;private final JedisClusterInfoCache clusterInfoCache;public JedisClusterPipline(JedisSlotBasedConnectionHandler connectionHandler, JedisClusterInfoCache clusterInfoCache) {this.connectionHandler = connectionHandler;this.clusterInfoCache = clusterInfoCache;}@Overrideprotected Client getClient(String key) {return getClient(SafeEncoder.encode(key));}@Overrideprotected Client getClient(byte[] key) {Client client;// 计算key所在的slotint slot = JedisClusterCRC16.getSlot(key);// 获取solt对应的连接池JedisPool pool = clusterInfoCache.getSlotPool(slot);// 从缓存获取连接Jedis borrowedJedis = poolToJedisMap.get(pool);// 缓存中没有连接则从连接池获取并缓存if (null == borrowedJedis) {borrowedJedis = pool.getResource();poolToJedisMap.put(pool, borrowedJedis);}client = borrowedJedis.getClient();orderedClients.add(client);return client;}@Overridepublic void close() {for (Jedis jedis : poolToJedisMap.values()) {// 清除连接内的残留数据,防止连接归还时有数据漏读的现象try {jedis.getClient().getAll();} catch (Throwable throwable) {log.warn("关闭jedis时遍历异常，遍历的目的是：清除连接内的残留数据,防止连接归还时有数据漏读的现象");}try {jedis.close();} catch (Throwable throwable) {log.warn("关闭jedis异常");}}// 归还连接clean();orderedClients.clear();poolToJedisMap.clear();}/*** go through all the responses and generate the right response type (warning :* usually it is a waste of time).** @return A list of all the responses in the order*/public List<Object> syncAndReturnAll() {List<Object> formatted = new ArrayList<>();List<Throwable> throwableList = new ArrayList<>();for (Client client : orderedClients) {try {Response response = generateResponse(client.getOne());if(response == null){continue;}formatted.add(response.get());} catch (Throwable e) {throwableList.add(e);}}slotCacheRefreshed(throwableList);return formatted;}}
猜想四：是不是发生了死锁？
举个例子：
四、死锁证明
以上只是猜想，为了证明确实发生了死锁，需要以下条件：
线程当前获取到了哪些连接池的连接线程当前在等待哪些连接池的连接每个连接池还剩多少连接
4.1 步骤一：获取线程在等待哪个连接池有空闲连接
引用关系：
4.2 步骤二：获取线程当前持有了哪些连接池的连接
第二步：第一步拿到JedisClusterPipeline持有哪个连接池的连接后，再查找持有此
4.3 死锁分析
0x6a578e0c80x6a578e0480x6a578ddc80x6a578e5380x6a578e838
五、总结
来源：本文转自公众号vivo互联网技术
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/70013542/viewspace-2998576/ ，如需转载，请注明出处，否则将追究法律责任。
下一篇：
医院运维 告警闪现后的故障排查
请登录后发表评论
登录
全部评论
