使用 Yarn-client 在 Google Cloud 上的 Hadoop 中运行 JAR

Question

我想使用 Yarn-client 在 Google Cloud 上的 Hadoop 中运行 JAR。

我在hadoop的主节点使用这个命令

spark-submit --class find --master yarn-client find.jar

但它返回此错误

    15/06/17 10:11:06 INFO client.RMProxy: Connecting to ResourceManager at hadoop-m-on8g/10.240.180.15:8032
15/06/17 10:11:07 INFO ipc.Client: Retrying connect to server: hadoop-m-on8g/10.240.180.15:8032. Already tried 0 
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

有什么问题？如果有用，这是我的 yarn-site.xml

<?xml version="1.0" ?>
<!--
     <configuration>
      <!-- Site specific YARN configuration properties -->
      <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/yarn-logs/</value>
        <description>
          The remote path, on the default FS, to store logs.
        </description>
      </property>
      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>
      <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-m-on8g</value>
      </property>
      <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>5999</value>
        <description>

Answer 1

在您的情况下，YARN ResourceManager 可能由于未知原因而运行不正常；您可以尝试使用以下方法修复纱线：

sudo sudo -u hadoop /home/hadoop/hadoop-install/sbin/stop-yarn.sh
sudo sudo -u hadoop /home/hadoop/hadoop-install/sbin/start-yarn.sh

但是，您似乎正在使用 Click-to-Deploy 解决方案； Click-to-Deploy 的 Spark + Hadoop 2 部署目前实际上不支持 YARN 上的 Spark，因为存在一些错误和缺少内存配置。如果您尝试使用开箱即用的--master yarn-client 运行它，通常会遇到这样的情况：

15/06/17 17:21:08 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: -1
   appStartTime: 1434561664937
   yarnAppState: ACCEPTED

15/06/17 17:21:09 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: -1
   appStartTime: 1434561664937
   yarnAppState: ACCEPTED

15/06/17 17:21:10 INFO cluster.YarnClientSchedulerBackend: Application report from ASM: 
   appMasterRpcPort: 0
   appStartTime: 1434561664937
   yarnAppState: RUNNING

15/06/17 17:21:15 ERROR cluster.YarnClientSchedulerBackend: Yarn application already ended: FAILED
15/06/17 17:21:15 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/metrics/json,null}
15/06/17 17:21:15 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler{/stages/stage/kill,null}

得到很好支持的部署方式是在 Google Compute Engine 上使用 Hadoop 2 和配置为能够在 YARN 上运行的 Spark 的集群是使用 bdutil。你会运行类似的东西：

./bdutil -P <instance prefix> -p <project id> -b <bucket> -z <zone> -d  \
    -e extensions/spark/spark_on_yarn_env.sh generate_config my_custom_env.sh
./bdutil -e my_custom_env.sh deploy

# Shorthand for logging in to the master
./bdutil -e my_custom_env.sh shell

# Handy way to run a socks proxy to make it easy to access the web UIs
./bdutil -e my_custom_env.sh socksproxy

# When done, delete your cluster
./bdutil -e my_custom_env.sh delete

使用spark_on_yarn_env.sh，Spark 应默认为yarn-client，但您可以随时重新指定--master yarn-client。您可以在bdutil 和./bdutil --help 中看到对可用标志的更详细说明。以下是我上面包含的标志的帮助条目：

-b, --bucket
  Google Cloud Storage bucket used in deployment and by the cluster.

-d, --use_attached_pds
  If true, uses additional non-boot volumes, optionally creating them on
  deploy if they don't exist already and deleting them on cluster delete.

-e, --env_var_files
  Comma-separated list of bash files that are sourced to configure the cluster
  and installed software. Files are sourced in order with later files being
  sourced last. bdutil_env.sh is always sourced first. Flag arguments are
  set after all sourced files, but before the evaluate_late_variable_bindings
  method of bdutil_env.sh. see bdutil_env.sh for more information.

-P, --prefix
  Common prefix for cluster nodes.

-p, --project
  The Google Cloud Platform project to use to create the cluster.

-z, --zone
  Specify the Google Compute Engine zone to use.