无法访问 YARN 作业历史记录

Question

我正在使用从源代码构建的最新 hadoop 版本 3.0.0。我已经启动并运行了我的时间线服务，并已将 hadoop 配置为也将其用于作业历史记录。但是，当我在资源管理器 UI 中单击历史记录时，出现以下错误：-

HTTP ERROR 404

Problem accessing /jobhistory/job/job_1444395439959_0001. Reason:

    NOT_FOUND

有人可以指出我在这里缺少什么。以下是我的 yarn-site.xml：-

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <description>The hostname of the Timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>0.0.0.0</value>
</property>
<property>
  <description>Address for the Timeline server to start the RPC server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10200</value>
</property>

<property>
  <description>The http address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>

<property>
  <description>The https address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:8190</value>
</property>

<property>
  <description>Handler thread count to serve the client RPC requests.</description>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>10</value>
</property>
<property>
  <description>Indicate to ResourceManager as well as clients whether
  history-service is enabled or not. If enabled, ResourceManager starts
  recording historical data that Timelien service can consume. Similarly,
  clients can redirect to the history service when applications
  finish if this is enabled.</description>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Store class name for history store, defaulting to file system
  store</description>
  <name>yarn.timeline-service.generic-application-history.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>
<property>
     <description>URI pointing to the location of the FileSystem path where the history will be persisted.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.uri</name>
     <value>/tmp/yarn/system/history</value>
</property>
<property>
     <description>T-file compression types used to compress history data.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.compression-type</name>
     <value>none</value>
</property>



 <property>
     <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

还有我的 mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>mapreduce.job.emit-timeline-data</name>
<value>true</value>
</property>
</configuration>

JPS 输出：

6022 NameNode
27976 NodeManager
27859 ResourceManager
6139 DataNode
6310 SecondaryNameNode
28482 ApplicationHistoryServer
29230 Jps

Answer 1

如果您想通过 YARN RM Web UI 查看日志，则需要启用日志聚合。为此，您需要在 yarn-site.xml 中设置以下参数：

  <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
  </property>
  <property>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
  </property>
  <property>
      <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
      <value>logs</value>
  </property>

如果您不启用日志聚合，则 NM 会将日志存储在本地。通过上述设置，日志在 HDFS 中聚合在“/app-logs/{username}/logs/”。在此文件夹下，您可以找到迄今为止运行的所有应用程序的日志。同样，日志保留时间由配置参数“yarn.log-aggregation.retain-seconds”（聚合日志保留多长时间）确定。

当 MapReduce 应用程序运行时，您可以从 YARN 的 Web UI 访问日志。申请完成后，日志将通过 Job History Server 提供。

另外，在 yarn-site.xml 中设置以下配置参数：

<property>
  <name>yarn.log.server.url</name>
  <value>http://{job-history-hostname}:8188/jobhistory/logs</value>
</property>