ITPub博客

首页 > 大数据 > Hadoop > hadoop之 node manager起不来, 执行mapreduce 程序hang住

hadoop之 node manager起不来, 执行mapreduce 程序hang住

原创 Hadoop 作者:张冲andy 时间:2018-01-29 10:16:49 0 删除 编辑

现象: node manager起不来, 执行mapreduce 程序hang住

namenode 进程状态查询
[root@hadp-master sbin]# jps
8608 ResourceManager
8257 NameNode
9268 Jps
8453 SecondaryNameNode

datanode 进程状态查询 , 发现 nodemanager 没有起来
[root@hadp-node1 hadoop]# jps
4179 DataNode
4940 Jps

namenode 单独起 nodemanager
[root@hadp-master sbin]# yarn nodemanager
18/01/26 03:53:54 INFO webapp.WebApps: Web app node started at 8042
18/01/26 03:53:54 INFO client.RMProxy: Connecting to ResourceManager at hadp-master/10.100.25.30:8031
18/01/26 03:53:54 INFO nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
18/01/26 03:53:54 INFO nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
18/01/26 03:53:54 ERROR nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdater
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadp-master doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. ## 提示 NodeManager 内存不足


解决过程:

1、原虚拟机755m内存,调大虚拟机内存至2G

2、修改 yarn-site.xml ,添加如下参数

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>200</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1600</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1600</value>
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>


参数 : yarn.scheduler.minimum-allocation-mb 
默认 : 1024 m 
官方解释: The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this will throw a InvalidResourceRequestException.

参数 : yarn.scheduler.maximum-allocation-mb
默认 : 8192 m
官方解释: The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException.

参数 : yarn.nodemanager.resource.memory-mb
默认 : 8192 m
官方解释: Amount of physical memory, in MB, that can be allocated for containers.

参数 : yarn.nodemanager.resource.cpu-vcores
默认 : 8
官方解释: Number of vcores that can be allocated for containers. This is used by the RM scheduler when allocating resources for containers. This is not used to limit the number of physical cores used by YARN containers.

来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/31383567/viewspace-2150593/ ,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
支持就关注我个人公众号:DBA烂笔头 。专注数据库领域,熟悉oracle、mysql、redis、mongodb,clickhouse等相关技术

注册时间:2016-08-11

  • 博文量
    626
  • 访问量
    286339