在YARN中,如何控制和监控map/reduce的并发数
在YARN中,如何控制和监控map/reduce的并发数
cloud-coder 发表于3年前
在YARN中,如何控制和监控map/reduce的并发数
  • 发表于 3年前
  • 阅读 1299
  • 收藏 0
  • 点赞 1
  • 评论 0

腾讯云 技术升级10大核心产品年终让利>>>   

摘要: 在YARN架构中,如何控制和监控map/reduce的并发数,来使机器的获得较好的利用率?

配置建议:

1.
    In MR1, the mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum properties dictated how many map and reduce slots each TaskTracker had.

    These properties no longer exist in YARN. Instead, YARN uses yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores, which control the amount of memory and CPU on each node, both available to both maps and reduces

    Essentially:
YARN has no TaskTrackers, but just generic NodeManagers. Hence, there's no more Map slots and Reduce slots separation. Everything depends on the amount of memory in use/demanded

2.

Using the web UI you can get lot of monitoring/admin kind of info:

NameNode - http://:50070/ 
Resource Manager - http://:8088/

其他配置参考:

  1. There is a good guide on YARN configuration from Hortonworks
  2. You may analyze your job in Job History server. It usually may be found on port 19888. Ambari andGanglia are also very good for cluster utilization measurement.
标签: yarn map reduce
共有 人打赏支持
粉丝 234
博文 187
码字总数 134554
×
cloud-coder
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: