ITPub博客

首页 > IT职业 > IT图书 > 本地Eclipse提交MR程序到Yarn

本地Eclipse提交MR程序到Yarn

IT图书 作者:蚂蚁没问题 时间:2021-09-09 10:07:09 0 删除 编辑

1、一般地,从Windows本地的Eclipse提交程序到yarn,会报如下错误:

Diagnostics: Exception from container-launch.
Container id: container_1526537597068_0006_02_000001Exit code: 1Exception message: /bin/bash: line 0: fg: no job control

Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
    at org.apache.hadoop.util.Shell.run(Shell.java:478)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


Container exited with a non-zero exit code 1

2、原因:
客户端提交任务时候,在YARNRunner.java中调用了  createApplicationSubmissionContext方法,该方法负责初始化启动MRAppMaster的脚本,该脚本的生成依赖于本地系统,而目前由于是在Windows系统下,所以生成的脚本提交到Linux集群上运行必然出错。

图片描述

createApplicationSubmissionContext

3、手工修改YARNRunner适配Linux系统,

// Construct necessary information to start the MR AM
        ApplicationSubmissionContext appContext = createApplicationSubmissionContext(conf, jobSubmitDir, ts);

4、修改完再次运行debug看到appContext的值为:

application_id { id: 7 cluster_timestamp: 1526537597068 } application_name: "wc-fat.jar" queue: "default" am_container_spec { localResources { key: "job.jar" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.jar" } size: 73165925 timestamp: 1526810933633 type: PATTERN visibility: APPLICATION pattern: "(?:classes/|lib/).*" } } localResources { key: "jobSubmitDir/job.splitmetainfo" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.splitmetainfo" } size: 15 timestamp: 1526810935006 type: FILE visibility: APPLICATION } } localResources { key: "jobSubmitDir/job.split" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.split" } size: 355 timestamp: 1526810934874 type: FILE visibility: APPLICATION } } localResources { key: "job.xml" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.xml" } size: 93151 timestamp: 1526810944696 type: FILE visibility: APPLICATION } } tokens: "HDTS

来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/2568/viewspace-2816752/ ,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论

注册时间:2019-08-16

  • 博文量
    107
  • 访问量
    675