CDH 4.1：运行子错误：java.lang.OutOfMemoryError：Java 堆空间

【问题标题】：CDH 4.1: Error running child : java.lang.OutOfMemoryError: Java heap spaceCDH 4.1：运行子错误：java.lang.OutOfMemoryError：Java 堆空间
【发布时间】：2012-11-20 09:12:57
【问题描述】：

我正在努力克服突然出现的问题。在那个问题之前，我使用了旧的虚拟机。我已经下载了新的虚拟机，但仍然无法运行我的工作。我收到 Java 堆空间错误。 我已经读过这篇文章：out of Memory Error in Hadoop

这是我来自 /etc/hadoop/conf 的配置：

sudo vi hadoop-env.sh 

# Extra Java runtime options.  Empty by default.
#export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"
export HADOOP_CLIENT_OPTS="-Xmx256m $HADOOP_CLIENT_OPTS"

这是我的 mapred-site.xml

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx256m</value>
  </property>
  <property>
    <name>io.sort.mb</name>
    <value>128</value>
  </property>

没有任何帮助:(

这是控制台输出：

Deleted /var/log/hadoop-yarn/apps/cloudera/logs
12/12/02 16:31:45 INFO input.FileInputFormat: Total input paths to process : 1
12/12/02 16:31:45 INFO mapreduce.JobSubmitter: number of splits:13
12/12/02 16:31:45 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
12/12/02 16:31:45 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
12/12/02 16:31:45 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
12/12/02 16:31:45 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
12/12/02 16:31:45 WARN conf.Configuration: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
12/12/02 16:31:45 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
12/12/02 16:31:45 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
12/12/02 16:31:46 INFO mapred.ResourceMgrDelegate: Submitted application application_1354455034384_0007 to ResourceManager at /0.0.0.0:8032
12/12/02 16:31:46 INFO mapreduce.Job: The url to track the job: http://localhost.localdomain:8088/proxy/application_1354455034384_0007/
12/12/02 16:31:46 INFO mapreduce.Job: Running job: job_1354455034384_0007
12/12/02 16:31:51 INFO mapreduce.Job: Job job_1354455034384_0007 running in uber mode : false
12/12/02 16:31:51 INFO mapreduce.Job:  map 0% reduce 0%
12/12/02 16:32:02 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_0, Status : FAILED

Killed by external signal

12/12/02 16:32:15 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_0, Status : FAILED
Error: Java heap space
12/12/02 16:32:19 INFO mapreduce.Job:  map 1% reduce 0%
12/12/02 16:32:29 INFO mapreduce.Job:  map 2% reduce 0%
12/12/02 16:32:29 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_1, Status : FAILED
Error: Java heap space
12/12/02 16:32:36 INFO mapreduce.Job:  map 3% reduce 0%
12/12/02 16:32:40 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_1, Status : FAILED
Error: Java heap space
12/12/02 16:32:43 INFO mapreduce.Job:  map 4% reduce 0%
12/12/02 16:32:51 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000005_2, Status : FAILED
Error: Java heap space
12/12/02 16:32:53 INFO mapreduce.Job:  map 5% reduce 0%
12/12/02 16:33:00 INFO mapreduce.Job:  map 6% reduce 0%
12/12/02 16:33:03 INFO mapreduce.Job: Task Id : attempt_1354455034384_0007_m_000006_2, Status : FAILED
Error: Java heap space
12/12/02 16:33:07 INFO mapreduce.Job:  map 7% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job:  map 8% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job:  map 15% reduce 0%
12/12/02 16:33:15 INFO mapreduce.Job: Job job_1354455034384_0007 failed with state FAILED due to: 
12/12/02 16:33:15 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=600
        FILE: Number of bytes written=349925
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=105310577
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=15
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Job Counters 
        Failed map tasks=7
        Launched map tasks=12
        Other local map tasks=5
        Data-local map tasks=7
        Total time spent by all maps in occupied slots (ms)=597080
        Total time spent by all reduces in occupied slots (ms)=0
    Map-Reduce Framework
        Map input records=252675
        Map output records=15
        Map output bytes=992
        Map output materialized bytes=0
        Input split bytes=590
        Combine input records=0
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=272051
        CPU time spent (ms)=112790
        Physical memory (bytes) snapshot=1094082560
        Virtual memory (bytes) snapshot=3678527488
        Total committed heap usage (bytes)=780533760
    File Input Format Counters 
        Bytes Read=105938372

这是我的日志：

2012-12-02 16:36:13,072 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2012-12-02 16:36:14,405 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2012-12-02 16:36:15,552 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1354455034384_0008
2012-12-02 16:36:15,916 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2012-12-02 16:36:15,919 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
2012-12-02 16:36:15,920 WARN [main] org.apache.hadoop.conf.Configuration: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
2012-12-02 16:36:15,920 WARN [main] org.apache.hadoop.conf.Configuration: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2012-12-02 16:36:15,921 WARN [main] org.apache.hadoop.conf.Configuration: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: job.local.dir is deprecated. Instead, use mapreduce.job.local.dir
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files
2012-12-02 16:36:15,922 WARN [main] org.apache.hadoop.conf.Configuration: mapred.job.id is deprecated. Instead, use mapreduce.job.id
2012-12-02 16:36:16,575 WARN [main] org.apache.hadoop.conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
2012-12-02 16:36:18,332 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@3582c132
2012-12-02 16:36:21,168 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:912)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:638)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:709)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2012-12-02 16:36:21,448 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
(&container_1354455034384_0008_01_000005�}stderr156WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
stdout0syslog35152012-12-02 16:35:31,838 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2012-12-02 16:35:32,283 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-12-02 16:35:32,283 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2012-12-02 16:35:33,314 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2012-12-02 16:35:34,339 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1354455034384_0008
2

接下来我可以尝试什么？ 请帮忙。谢谢。

【问题讨论】：

  • -XX:-HeapDumpOnOutOfMemoryError
  • 我设置了这个：export HADOOP_CLIENT_OPTS="-Xmx256m -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/cloudera $HADOOP_CLIENT_OPTS" 并且我在指定文件夹中没有看到任何磁头转储。我做错了什么？
  • 一些猜测：那个目录不存在； Java 进程对该目录没有写权限； Hadoop 进程没有启动$HADOOP_CLIENT_OPTS； 256m 根本不够，所以尝试更大的数字。
  • 256 JVM 到 128 排序的组合可能是您的问题。尝试 io.sort.mb 大小为 64mb
  • 没什么帮助。我不知道如何调试它。似乎问题出在其他地方。所有这些调试参数都被忽略。在指定文件夹中看不到堆转储。

标签： hadoop mapreduce cloudera hadoop-yarn


【解决方案1】：

也可以尝试将mapred.job.shuffle.input.buffer.percent 设置为 20%。默认情况下，这设置为 70%，如果您正在处理大量数据，这可能会很多。

尝试在 mapred-site.xml 中设置此属性：

<property>
   <name>mapred.job.shuffle.input.buffer.percent</name>
   <value>0.20</value>
</property>

有关更多性能调整，请参阅此文档： http://developer.amd.com.php53-23.ord1-1.websitetestlink.com/wordpress/media/2012/10/Hadoop_Tuning_Guide-Version5.pdf

希望有人觉得这很有用:)

【讨论】：

    【解决方案2】：

    好的，看来问题已经解决了。这是我的 mapred-site.xml

    <configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>0.0.0.0:8021</value>
  </property>

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

  <property>
    <description>To set the value of tmp directory for map and reduce tasks.</description>
    <name>mapreduce.task.tmp.dir</name>
    <value>/var/lib/hadoop-mapreduce/cache/${user.name}/tasks</value>
  </property>

  <property>
    <name>jobtracker.thrift.address</name>
    <value>0.0.0.0:9290</value>
  </property>
  <property>
    <name>mapred.jobtracker.plugins</name>
    <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
    <description>Comma-separated list of jobtracker plug-ins to be activated.</description>
  </property>
  <property>
      <name>mapred.child.java.opts</name>
      <value>-Xmx512m</value>
  </property>
  <property>
      <name>io.sort.mb</name>
      <value>64</value>
  </property>
</configuration>

    这是我的 hadoop-env.sh：

    export HADOOP_CLIENT_OPTS="-Xmx512m -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/cloudera/heapdump $HADOOP_CLIENT_OPTS"

    我想问题在于应用的更改。只有重新启动对我有帮助。我已尝试重新启动服务 - 没有效果。

    【讨论】：

    • 是的。问题在于应用更改。重新启动对我有用。谢谢！
    猜你喜欢
    • 2016-01-19
    • 2016-04-10
    • 2020-09-25
    • 2013-03-31
    • 1970-01-01
    • 2010-12-08
    • 2015-05-14
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode