ITPub博客

首页 > 大数据 > 数据分析 > CCAH-CCA-500-6题:You want YARN to launch no more than 16 containers per node.

CCAH-CCA-500-6题:You want YARN to launch no more than 16 containers per node.

原创 数据分析 作者:hackeruncle 时间:2016-05-09 20:39:14 0 删除 编辑

6.What should you do?
Each node in your Hadoop cluster, running YARN, has 64GB memory and 24 cores.
Your yarn.site.xml has the following configuration:
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>32768</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>12</value>
</property>
You want YARN to launch no more than 16 containers per node. What should you do?

A.
Modify yarn-site.xml with the following property:
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
B.
Modify yarn-sites.xml with the following property:
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>4096</value>
C.
Modify yarn-site.xml with the following property:
<name>yarn.nodemanager.resource.cpu-vccores</name>
D.
No action is needed: YARN’s dynamic resource allocation automatically optimizes the node
memory and cores


问题:
      Hadoop集群的每台节点内存64G,CPU 24核。当前一个节点可分配的物理内存总量是32768M(32G),可分配的虚拟cpu总个数是12。
   那么yarn在每个节点上,发起不超过16个容器,我们应该怎样配置?

分析:A

   yarn.scheduler.minimum-allocation-mb:最小可申请内存量,默认是1024,我们设置2048M,见如下计算公式

       Maximum memory YARN can utilize on the node
    ————————————————————————————————————————————— = minimum memory per container
       Number of containers
    
配置文件                     配置项名称                                  配置项值
yarn-site.xml        yarn.nodemanager.resource.memory-mb        = Containers个数* 每个Container内存
yarn-site.xml        yarn.scheduler.minimum-allocation-mb        = 每个Container内存
yarn-site.xml        yarn.scheduler.maximum-allocation-mb        = Containers个数* 每个Container内存
mapred-site.xml        mapreduce.map.memory.mb        = 每个Container内存
mapred-site.xml        mapreduce.reduce.memory.mb        = 2 * 每个Container内存
mapred-site.xml        mapreduce.map.java.opts        = 0.8 * 每个Container内存
mapred-site.xml        mapreduce.reduce.java.opts        = 0.8 * 2 * 每个Container内存
yarn-site.xml (check)        yarn.app.mapreduce.am.resource.mb        = 2 * 每个Container内存
yarn-site.xml (check)        yarn.app.mapreduce.am.command-opts        = 0.8 * 2 * 每个Container内存

http://developer.51cto.com/art/201401/426610.htm

来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/30089851/viewspace-2096676/ ,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
主要从事大数据平台产品的技术工作; 负责基于Spark、Flink生产开发及性能调优; 负责对公司的集群的部署、监控、调优和运营; 主导我司的数据平台0到1及实时中间件项目开发; 此外,精通Java、Shell、Python编程; 且荣获2016年IBM的Spark大赛全球第四名,中国区域第一名。

注册时间:2015-01-03

  • 博文量
    187
  • 访问量
    395981