YARN：容器启动退出代码 127 的异常答案

【问题标题】：YARN: Exception from container-launch exit code 127YARN：容器启动退出代码 127 的异常
【发布时间】：2021-12-19 04:01:50
【问题描述】：

我正在尝试像这样向 YARN 提交 teragen 作业：

yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.3.1.jar teragen 1000 /teragen

一切顺利，直到出错：

2021-11-04 23:45:20,540 INFO mapreduce.Job: Running job: job_1636069364859_0003
2021-11-04 23:45:25,629 INFO mapreduce.Job: Job job_1636069364859_0003 running in uber mode : false
2021-11-04 23:45:25,630 INFO mapreduce.Job:  map 0% reduce 0%
2021-11-04 23:45:27,658 INFO mapreduce.Job: Task Id : attempt_1636069364859_0003_m_000000_0, Status : FAILED
[2021-11-04 23:45:26.200]Exception from container-launch.
Container id: container_1636069364859_0003_01_000002
Exit code: 127

[2021-11-04 23:45:26.201]Container exited with a non-zero exit code 127. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
/bin/bash: line 1: m: command not found

我不知道问题是什么。我试过查看日志，尤其是prelaunch.err 文件，但它是空的。 stderr 文件有：

/bin/bash: line 1: m: command not found

查看节点管理器日志，我发现：

2021-11-04 23:44:05,765 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system started
2021-11-04 23:44:06,423 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1636069364859_0001_01_000002 transitioned from LOCALIZING to SCHEDULED
2021-11-04 23:44:06,423 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler: Starting container [container_1636069364859_0001_01_000002]
2021-11-04 23:44:06,453 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1636069364859_0001_01_000002 transitioned from SCHEDULED to RUNNING
2021-11-04 23:44:06,453 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1636069364859_0001_01_000002
2021-11-04 23:44:06,457 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /opt/yarn/local/usercache/vagrant/appcache/application_1636069364859_0001/container_1636069364859_0001_01_000002/default_container_executor.sh]
2021-11-04 23:44:06,477 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1636069364859_0001_01_000002 is : 127
2021-11-04 23:44:06,478 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1636069364859_0001_01_000002 and exit code: 127
ExitCodeException exitCode=127: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
    at org.apache.hadoop.util.Shell.run(Shell.java:901)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:309)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:585)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:373)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2021-11-04 23:44:06,479 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2021-11-04 23:44:06,479 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1636069364859_0001_01_000002
2021-11-04 23:44:06,479 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 127
2021-11-04 23:44:06,479 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container launch failed : Container exited with a non-zero exit code 127. 
2021-11-04 23:44:06,501 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1636069364859_0001_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE
2021-11-04 23:44:06,503 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerCleanup: Cleaning up container container_1636069364859_0001_01_000002
2021-11-04 23:44:06,515 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2021-11-04 23:44:06,515 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
2021-11-04 23:44:06,515 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
2021-11-04 23:44:06,525 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /opt/yarn/local/usercache/vagrant/appcache/application_1636069364859_0001/container_1636069364859_0001_01_000002
2021-11-04 23:44:06,526 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1636069364859_0001_01_000002 transitioned from EXITED_WITH_FAILURE to DONE
2021-11-04 23:44:06,526 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1636069364859_0001_01_000002 from application application_1636069364859_0001
2021-11-04 23:44:06,526 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1636069364859_0001_01_000002

我已阅读其他回复，当他们提到 Java 丢失或未设置 JAVA_HOME 时。那不是我的情况，我的JAVA_HOME 设置为/usr/lib/jvm/java-8-openjdk-amd64。

知道这里会发生什么吗？谢谢:)

【问题讨论】：

脚本有错误...line 1: m: command not found来自bash，与Hadoop无关
嘿@OneCricketeer，实际上是内存配置错误。出于某种原因，某些东西“存活”的时间不够长，无法实际发布堆大小问题。在让几个容器失败并重新尝试执行作业后，我最终解决了这个问题。我打算马上发布解决方案。

标签： java hadoop hadoop-yarn

【解决方案1】：

问题在于为每个容器分配的内存。一些容器的寿命不够长，显然无法实际记录错误。

但经过几次尝试，我实际上得到了一个看起来像这样的错误：

Error occurred during initialization of VM
Too small initial heap

由于某种原因，我使用的 YARN 和 MapReduce 作业的内存配置不正确。我最终使用 Ambari 的 HDP yarn-util.py 来获取适合我的设置的值。

【讨论】：