InvalidResourceRequestException：执行 Hadoop Streaming MR 作业时出现奇怪错误

Question

我在执行以下命令时遇到了这个奇怪的错误：

hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar  -file ./methratio.py -file './genome.fa' -mapper './methratio.py -r -g ' -input  /TextLab/sravisha_test/SamFiles/test_bam  -output ./outfile

错误：

    15/01/28 18:56:29 ERROR streaming.StreamJob: Error Launching job : org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=3072maxMemory=2583 at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtil s.validateResourceRequest(SchedulerUtils.java:196)  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateResourceRequest(RMAppManager.java:385) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:345)  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:271)
 at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitA pplication(ClientRMService.java:538)
            at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolP                                                                                                             BServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:188)
            at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:323)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
            at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:415)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)

谁能告诉我这里出了什么问题？

Answer 1

错误指向YARN环境下节点内存不足。可能是节点机器/VM 没有足够的内存，或者 yarn-site.xml 中的 yarn.nodemanager.resource.memory-mb 值设置为低于 3072 的值，但您的流应用程序需要更多内存。