运行子错误：java.lang.OutOfMemoryError: Java heap space

【问题标题】：Error running child : java.lang.OutOfMemoryError: Java heap space运行子错误：java.lang.OutOfMemoryError: Java heap space
【发布时间】：2020-04-03 13:00:45
【问题描述】：

我在互联网上阅读了很多内容，但没有找到解决问题的方法。 我使用 Hadoop 2.6.0。

MapReduce 的主要目标是运行一个 SequenceFile 并对键/值对进行一些分析。

2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling map output
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 23342; bufvoid = 104857600
2015-01-29 10:09:50,554 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396(104857584); kvend = 26213840(104855360); length = 557/6553600
2015-01-29 10:09:50,570 INFO [main] org.apache.hadoop.mapred.MapTask: Finished spill 0
2015-01-29 10:09:50,577 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:66)
    at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

这里是 STDOUT 的输出

15/01/29 10:09:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
15/01/29 10:09:35 INFO compress.CodecPool: Got brand-new compressor [.gz]

15/01/29 10:09:36 INFO client.RMProxy: Connecting to ResourceManager at xxxxxxxxxxxxxxxxxxxxx:8040
15/01/29 10:09:37 INFO input.FileInputFormat: Total input paths to process : 1
15/01/29 10:09:37 INFO mapreduce.JobSubmitter: number of splits:1
15/01/29 10:09:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422374835659_0059
15/01/29 10:09:37 INFO impl.YarnClientImpl: Submitted application application_1422374835659_0059
15/01/29 10:09:37 INFO mapreduce.Job: The url to track the job: http://xxxxxxxxxxxxxxxxxxxxx:8088/proxy/application_1422374835659_0059/
15/01/29 10:09:37 INFO mapreduce.Job: Running job: job_1422374835659_0059
15/01/29 10:09:44 INFO mapreduce.Job: Job job_1422374835659_0059 running in uber mode : false
15/01/29 10:09:44 INFO mapreduce.Job:  map 0% reduce 0%
15/01/29 10:09:50 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_0, Status : FAILED
Error: Java heap space
15/01/29 10:09:58 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_1, Status : FAILED
Error: Java heap space
15/01/29 10:10:04 INFO mapreduce.Job: Task Id : attempt_1422374835659_0059_m_000000_2, Status : FAILED
Error: Java heap space
15/01/29 10:10:10 INFO mapreduce.Job:  map 100% reduce 100%
15/01/29 10:10:11 INFO mapreduce.Job: Job job_1422374835659_0059 failed with state FAILED due to: Task failed task_1422374835659_0059_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/01/29 10:10:11 INFO mapreduce.Job: Counters: 12
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=37910
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=18955
        Total vcore-seconds taken by all map tasks=18955
        Total megabyte-seconds taken by all map tasks=38819840
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0

我的配置几乎是默认的，与 Java 堆大小无关。

这个我也试过了，没什么区别。

<property>
        <name>mapred.child.java.opts</name>
        <value>-Xmx1024m</value>
</property>

master 上的主程序以 -Xmx512m 开头，而 node 上的容器按预期以 -Xmx1024m 开头。

还将hadoop-env.sh 编辑为以下内容，但无济于事：

export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

我的 MapReduce 应用程序中的配置：

conf.setInt("mapreduce.map.memory.mb", 2048);
conf.setInt("mapreduce.reduce.memory.mb", 1024);

编辑 1 29.01：

-Xmx2048m 我收到了同样的错误。

-Xmx3072m 出现以下错误：

Error: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
    at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
    at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

-Xmx4096m 我遇到了一个完全不同的错误，我不明白为什么他现在要使用 5GB 的虚拟内存：

Container [pid=61687,containerID=container_1422374835659_0064_01_000002] is running beyond virtual memory limits. Current usage: 866.8 MB of 2 GB physical memory used; 5.0 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1422374835659_0064_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 61687 61685 61687 61687 (bash) 0 0 12640256 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx4096m -Djava.io.tmpdir=/home/hduser/tmp/nm-local-dir/usercache/hduser/appcache/application_1422374835659_0064/container_1422374835659_0064_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.97.83.13 33802 attempt_1422374835659_0064_m_000000_0 2 1>/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002/stdout 2>/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002/stderr  
    |- 61692 61687 61687 61687 (java) 629 149 5384613888 221601 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx4096m -Djava.io.tmpdir=/home/hduser/tmp/nm-local-dir/usercache/hduser/appcache/application_1422374835659_0064/container_1422374835659_0064_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop-2.6.0/logs/userlogs/application_1422374835659_0064/container_1422374835659_0064_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.97.83.13 33802 attempt_1422374835659_0064_m_000000_0 2 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

编辑 2 29.01

即使在 map() 函数中注释掉所有内容，也会出现错误。

SequenceFile (132.93 KB) 中只有 10 个键/值对，一切正常。

编辑 3 30.01

这里是最小化的 Source，它会产生相同的错误。

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class Dummy implements Tool {

    private Configuration conf;

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        int res = ToolRunner.run(conf, new Dummy(), args);
        System.exit(res);
    }

    @Override
    public void setConf(Configuration conf) {
        // Set some Job options
        conf.set("dfs.blocksize", "16m");

        // set heap size
        // conf.set("yarn.app.mapreduce.am.command-opts", "-Xmx1024m");
        // conf.set("mapred.child.java.opts", "-Xmx200m");

        // request more memory be the ressourcemanager
        conf.setInt("mapreduce.map.memory.mb", 2048);
        conf.setInt("mapreduce.reduce.memory.mb", 1024);

        // IO space
        // conf.setInt("mapreduce.task.io.sort.mb", 256);

        // Since we have lots of small tasks we should reduce overhead
        // conf.setInt("mapreduce.job.jvm.numtasks", -1);

        this.conf = conf;
    }

    /**
     * configuration getter
     */
    @Override
    public Configuration getConf() {
        return conf;
    }

    @Override
    public int run(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        // Configure the job
        Job job = Job.getInstance(conf, "Dummy");

        job.setJarByClass(Dummy.class);

        job.setInputFormatClass(SequenceFileInputFormat.class);
        job.setMapperClass(Map.class);

        // Set number of Reducers to number of actions + 1 for error log
        // job.setNumReduceTasks(action_count+2);
        job.setReducerClass(Reduce.class); // Global Aggregation

        // Set output
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        // Enable record skipping for failed Maps
        // SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);

        // only create a output file it there is content
        // LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class);

        // set input and output for job
        // FileInputFormat.addInputPath(job, repo.getRepository());
        FileInputFormat.setInputPaths(job, new Path("/test/test.seq"));
        FileOutputFormat.setOutputPath(job, new Path("/test/out"));

        // Execute Job
        int res = 0;
        // job.submit();
        res = job.waitForCompletion(true) ? 0 : 1;

        return res;

    }

    public static class Map extends Mapper<Text, Text, Text, Text> {
        @Override
        protected void map(Text key, Text value, Mapper<Text, Text, Text, Text>.Context context) {
            // TODO Auto-generated method stub
        }
    }

    public static class Reduce extends Reducer<Text, Text, Text, Text> {

        @Override
        protected void reduce(Text key, Iterable<Text> value, Reducer<Text, Text, Text, Text>.Context context) {
            // TODO Auto-generated method stub

        }
    }
}

【问题讨论】：

  • 我会使用 jvisualvm 监控进程，以确认堆是您认为应该的大小，如果仍然出现错误，请给它更多内存。最大内存应该是你宁愿死也不愿使用更多的大小。
  • 在我看来，您似乎是在告诉 Hadoop，它应该可以随意使用 2048 MB 来进行 Map 操作，但是您是在告诉 JVM 它应该使用不超过 1024 MB 的总内存。您的倒数第二个代码块将其增加到 2048 MB，但您仍然不允许任何非 Hadoop 内容的空间。你能把-Xmx 至少增加到 3072 MB 吗？
  • 我试过 -Xmx2048m、-Xmx3072m 和 -Xmx4096m。见主帖。
  • 我使用 visualvm 分析了 mapreduce 进程，发现堆大小最大为 380MB，仅使用了 150MB。
  • 为了让我们了解您的作业内存不足的原因，我们可能需要查看您的代码。也许您正在为每个 map() 创建内存密集型对象，而这些对象可以在 setup() 中创建一次并重复使用每个 map()？

标签： java hadoop


【解决方案1】：

我最近遇到了同样的问题。 我正在使用 Oracle VM 来学习 hadoop。分配的基本内存为 512 MB，我遇到了同样的错误：

java.lang.Exception: java.lang.OutOfMemoryError: Java 堆空间

然后我将它增加到 1024MB，然后我就可以成功运行 MR 程序了。

【讨论】：

  • 这可能应该是一个评论，但它可以回答这个问题。我编辑了它，我会让其他人决定：p
猜你喜欢
  • 2013-12-20
  • 2010-09-07
  • 2012-08-03
  • 2016-04-15
  • 2013-06-20
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode