日志文件中的 Map 和 reduce 任务计数不正确答案

【问题标题】：Map and reduce task counts are not correct in log file日志文件中的 Map 和 reduce 任务计数不正确
【发布时间】：2014-02-20 12:32:14
【问题描述】：

我正在运行一个正确运行的 mapreduce 作业。但是我对正在生成的日志文件有一定的困惑。

运行map-red的命令

hadoop jar mapred-0.0.1-SNAPSHOT.jar tcs.hadoop.org.mapreduce.MaxTemperatureDriver /priya/sample.txt /output

14/02/20 17:35:10 INFO input.FileInputFormat: Total input paths to process : 1
14/02/20 17:35:10 WARN snappy.LoadSnappy: Snappy native library is available
14/02/20 17:35:10 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/02/20 17:35:10 INFO snappy.LoadSnappy: Snappy native library loaded
14/02/20 17:35:10 INFO mapred.JobClient: Running job: job_201402111203_0034
14/02/20 17:35:11 INFO mapred.JobClient:  map 0% reduce 0%
14/02/20 17:35:22 INFO mapred.JobClient:  map 100% reduce 0%
14/02/20 17:35:36 INFO mapred.JobClient:  map 100% reduce 100%
14/02/20 17:35:39 INFO mapred.JobClient: Job complete: job_201402111203_0034
14/02/20 17:35:40 INFO mapred.JobClient: Counters: 26
14/02/20 17:35:40 INFO mapred.JobClient:   Job Counters 
14/02/20 17:35:40 INFO mapred.JobClient:     Launched reduce tasks=2
14/02/20 17:35:40 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=11900
14/02/20 17:35:40 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/02/20 17:35:40 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/02/20 17:35:40 INFO mapred.JobClient:     Launched map tasks=1
14/02/20 17:35:40 INFO mapred.JobClient:     Data-local map tasks=1
14/02/20 17:35:40 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=23142
14/02/20 17:35:40 INFO mapred.JobClient:   FileSystemCounters
14/02/20 17:35:40 INFO mapred.JobClient:     FILE_BYTES_READ=34
14/02/20 17:35:40 INFO mapred.JobClient:     HDFS_BYTES_READ=633
14/02/20 17:35:40 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=154973
14/02/20 17:35:40 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=17
14/02/20 17:35:40 INFO mapred.JobClient:   Map-Reduce Framework
14/02/20 17:35:40 INFO mapred.JobClient:     Map input records=5
14/02/20 17:35:40 INFO mapred.JobClient:     Reduce shuffle bytes=34
14/02/20 17:35:40 INFO mapred.JobClient:     Spilled Records=4
14/02/20 17:35:40 INFO mapred.JobClient:     Map output bytes=45
14/02/20 17:35:40 INFO mapred.JobClient:     CPU time spent (ms)=4420
14/02/20 17:35:40 INFO mapred.JobClient:     Total committed heap usage (bytes)=172822528
14/02/20 17:35:40 INFO mapred.JobClient:     Combine input records=5
14/02/20 17:35:40 INFO mapred.JobClient:     SPLIT_RAW_BYTES=103
14/02/20 17:35:40 INFO mapred.JobClient:     Reduce input records=2
14/02/20 17:35:40 INFO mapred.JobClient:     Reduce input groups=2
14/02/20 17:35:40 INFO mapred.JobClient:     Combine output records=2
14/02/20 17:35:40 INFO mapred.JobClient:     Physical memory (bytes) snapshot=300945408
14/02/20 14/02/20 17:35:40 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=7375564800
14/02/20 17:35:40 INFO mapred.JobClient:     Map output records=517:35:40 INFO mapred.JobClient:     Reduce output records=2

所以从这里我可以看到我正在创建一个 map 任务和两个 reduce 任务。

但是，当我查看位于 $HADOOP_HOME/logs/history 目录中的作业历史日志时，我得到 5 个任务已由作业跟踪器触发，如下所示（仅提供日志行）。 我无法理解为什么 5 个任务而不是 3 个。

MapAttempt TASK_TYPE="SETUP" TASKID="task_201402111203_0034_m_000002" TASK_ATTEMPT_ID="attempt_201402111203_0034_m_000002_0" START_TIME="1392897911096" TRACKER_NAME="tracker_IMBDBOX1:IMBDBOX
1/157\.227\.44\.207:40925" HTTP_PORT="50060" .

MapAttempt TASK_TYPE="MAP" TASKID="task_201402111203_0034_m_000000" TASK_ATTEMPT_ID="attempt_201402111203_0034_m_000000_0" TASK_STATUS="SUCCESS" FINISH_TIME="1392897989806" HOSTNAME="/defaul

ReduceAttempt TASK_TYPE="REDUCE" TASKID="task_201402111203_0034_r_000001" TASK_ATTEMPT_ID="attempt_201402111203_0034_r_000001_0" START_TIME="1392897947754" TRACKER_NAME="tracker_IMBDBOX3:loc
alhost/127\.0\.0\.1:34625" HTTP_PORT="50060" 

ReduceAttempt TASK_TYPE="REDUCE" TASKID="task_201402111203_0034_r_000000" TASK_ATTEMPT_ID="attempt_201402111203_0034_r_000000_0" START_TIME="1392897992388" TRACKER_NAME="tracker_IMBDBOX4:loc
alhost/127\.0\.0\.1:59439" HTTP_PORT="50060" .

MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201402111203_0034_m_000001" TASK_ATTEMPT_ID="attempt_201402111203_0034_m_000001_0" START_TIME="1392898004324" TRACKER_NAME="tracker_IMBDBOX4:local
host/127\.0\.0\.1:59439" HTTP_PORT="50060"

再次，当我进入位于 $HADOOP_HOME/logs/userlogs 中的用户日志时，我只能看到已生成日志的一个地图任务。 为什么没有生成其他 map 和 reduce 任务的日志？

请帮忙。谢谢！

用户日志目录

total 8
-rw-r----- 1 hadoop hdusers 497 2014-02-20 17:35 job-acls.xml
lrwxrwxrwx 1 hadoop hdusers  96 2014-02-20 17:35 attempt_201402111203_0034_m_000002_0 -> /app/hadoop/tmp/mapred/local/userlogs/job_201402111203_0034/attempt_201402111203_0034_m_000002_0

【问题讨论】：

标签： hadoop mapreduce

【解决方案1】：

请注意这个字符串：

TASK_TYPE="SETUP"
TASK_TYPE="CLEANUP"

Hadoop 需要在 Map Reduce 作业生命周期中运行多个辅助作业。

【讨论】：

谢谢罗迪！我有其他疑问，在 userlog 中，仅用于任务 attempt_201402111203_0034_m_000002_0 ，这是作业的设置任务已被记录。是不是我也应该为我的地图和减少任务获取日志？我也给了我的用户日志目录内容。
您真的在寻找名称节点上的作业日志吗？你检查你的数据节点了吗？ Hadoop 将日志本地存储到作业中。您的地图和归约运行在数据节点上，您应该在那里寻找它们。