首页 > IT职业 > IT图书 > 本地Eclipse提交MR程序到Yarn
1、一般地,从Windows本地的Eclipse提交程序到yarn,会报如下错误:
Diagnostics: Exception from container-launch. Container id: container_1526537597068_0006_02_000001Exit code: 1Exception message: /bin/bash: line 0: fg: no job control Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:561) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Container exited with a non-zero exit code 1
2、原因:
客户端提交任务时候,在YARNRunner.java中调用了 createApplicationSubmissionContext方法,该方法负责初始化启动MRAppMaster的脚本,该脚本的生成依赖于本地系统,而目前由于是在Windows系统下,所以生成的脚本提交到Linux集群上运行必然出错。

createApplicationSubmissionContext
3、手工修改YARNRunner适配Linux系统,
// Construct necessary information to start the MR AM ApplicationSubmissionContext appContext = createApplicationSubmissionContext(conf, jobSubmitDir, ts);
4、修改完再次运行debug看到appContext的值为:
application_id { id: 7 cluster_timestamp: 1526537597068 } application_name: "wc-fat.jar" queue: "default" am_container_spec { localResources { key: "job.jar" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.jar" } size: 73165925 timestamp: 1526810933633 type: PATTERN visibility: APPLICATION pattern: "(?:classes/|lib/).*" } } localResources { key: "jobSubmitDir/job.splitmetainfo" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.splitmetainfo" } size: 15 timestamp: 1526810935006 type: FILE visibility: APPLICATION } } localResources { key: "jobSubmitDir/job.split" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.split" } size: 355 timestamp: 1526810934874 type: FILE visibility: APPLICATION } } localResources { key: "job.xml" value { resource { scheme: "hdfs" host: "192.168.92.150" port: 8020 file: "/tmp/hadoop-yarn/staging/hadoop/.staging/job_1526537597068_0007/job.xml" } size: 93151 timestamp: 1526810944696 type: FILE visibility: APPLICATION } } tokens: "HDTS
来自 “ ITPUB博客 ” ,链接:https://blog.itpub.net/2568/viewspace-2816752/ ,如需转载,请注明出处,否则将追究法律责任。