ITPub博客

首页 > 大数据 > 数据分析 > java.io.IOException: Connection reset by peer

java.io.IOException: Connection reset by peer

原创 数据分析 作者:hackeruncle 时间:2017-10-24 12:08:37 0 删除 编辑
一.场景:
    spark通过phoenix 读取hbase表,其实说白了先要去Zookeeper建立connection。

二.代码:

点击(此处)折叠或打开

  1. val zkUrl = "192.168.100.39,192.168.100.40,192.168.100.41:2181"
  2. val formatStr = "org.apache.phoenix.spark"
  3. val oms_orderinfoDF = spark.read.format(formatStr)
  4.   .options(Map("table" -> "oms_orderinfo", "zkUrl" -> zkUrl))
  5.   .load

三.查看SparkJob日志:

点击(此处)折叠或打开

  1. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop40/192.168.100.40:2181. Will not attempt to authenticate using SASL (unknown error)
  2. 17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.100.48:35952, server: hadoop40/192.168.100.40:2181
  3. 17/10/24 03:25:25 WARN zookeeper.ClientCnxn: Session 0x0 for server hadoop40/192.168.100.40:2181, unexpected error, closing socket connection and attempting reconnect
  4. java.io.IOException: Connection reset by peer
  5.     at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
  6.     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
  7.     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
  8.     at sun.nio.ch.IOUtil.read(IOUtil.java:192)
  9.     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
  10.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
  11.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
  12.     at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
  13. 17/10/24 03:25:25 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hdfs/.sparkStaging/application_1507703377455_4854
  14. 17/10/24 03:25:25 INFO util.ShutdownHookManager: Shutdown hook called
主要错误是“java.io.IOException: Connection reset by peer”,其实意思很明白了Zookeeper将connection closing.
至于为什么会被关闭connection?那么去检查一下hadoop40机器的zookeeper的日志就会发现一些有用的信息。

四.查看Zookeeper日志:

点击(此处)折叠或打开

  1. 2017-10-24 03:25:22,498 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500
  2. 2017-10-24 03:25:25,588 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500
  3. 2017-10-24 03:25:25,819 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500
  4. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
  5. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee09897d, likely client has closed socket
  6.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)
  7.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
  8.     at java.lang.Thread.run(Thread.java:745)
  9. 2017-10-24 03:25:26,092 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47084 which had sessionid 0x15ed091ee09897d
  10. 2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
  11. EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee098981, likely client has closed socket
  12.     at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)
  13.     at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
  14.     at java.lang.Thread.run(Thread.java:745)
  15. 2017-10-24 03:25:26,093 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47304 which had sessionid 0x15ed091ee098981
  16. 2017-10-24 03:25:26,093 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception
这个时候我们发现"Too many connections from /192.168.100.48 - max is 500",
那就说明这台hadoop40机器的zookeeper的当前connection数已经超过配置的500了,所以新的connection必然被closed,


五.解决方法:
那么我们只需去zookeeper的配置修改一下参数
maxClientCnxns调大到1000,重启zookeeper服务生效配置即可。

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/30089851/viewspace-2146302/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
主要从事大数据平台产品的技术工作; 负责基于Spark、Flink生产开发及性能调优; 负责对公司的集群的部署、监控、调优和运营; 主导我司的数据平台0到1及实时中间件项目开发; 此外,精通Java、Shell、Python编程; 且荣获2016年IBM的Spark大赛全球第四名,中国区域第一名。

注册时间:2015-01-03

  • 博文量
    187
  • 访问量
    352825