spark-submit 上的 Amazon EMR 异常

【问题标题】：Amazon EMR Exception on spark-submitspark-submit 上的 Amazon EMR 异常
【发布时间】：2020-10-19 16:06:37
【问题描述】：

当我在 Amazon AWS EMR 中运行我的 spark 应用程序时，我遇到了以下错误，但我不明白是什么问题。我创建了一个由两个工作节点和一个驱动节点组成的集群

17/05/30 19:38:34 INFO RMProxy: Connecting to ResourceManager at ip-172-31-10-147.eu-central-1.compute.internal/172.31.10.147:8032
17/05/30 19:38:34 INFO Client: Requesting a new application from cluster with 2 NodeManagers
17/05/30 19:38:34 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (11520 MB per container)
17/05/30 19:38:34 INFO Client: Will allocate AM container, with 11520 MB memory including 1047 MB overhead
17/05/30 19:38:34 INFO Client: Setting up container launch context for our AM
17/05/30 19:38:34 INFO Client: Setting up the launch environment for our AM container
17/05/30 19:38:35 INFO Client: Preparing resources for our AM container
17/05/30 19:38:36 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/05/30 19:38:39 INFO Client: Uploading resource file:/mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69/__spark_libs__9005882140690733981.zip -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/__spark_libs__9005882140690733981.zip
17/05/30 19:38:41 INFO Client: Uploading resource file:/home/hadoop/Main.jar -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/Main.jar
17/05/30 19:38:41 INFO Client: Uploading resource file:/mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69/__spark_conf__5018873114441326186.zip -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/__spark_conf__.zip
17/05/30 19:38:41 INFO SecurityManager: Changing view acls to: hadoop
17/05/30 19:38:41 INFO SecurityManager: Changing modify acls to: hadoop
17/05/30 19:38:41 INFO SecurityManager: Changing view acls groups to: 
17/05/30 19:38:41 INFO SecurityManager: Changing modify acls groups to: 
17/05/30 19:38:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
17/05/30 19:38:41 INFO Client: Submitting application application_1496172789995_0001 to ResourceManager
17/05/30 19:38:41 INFO YarnClientImpl: Submitted application application_1496172789995_0001
17/05/30 19:38:42 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:42 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1496173121751
     final status: UNDEFINED
     tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
     user: hadoop
17/05/30 19:38:43 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:44 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:45 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:46 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:47 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:48 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:49 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:49 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 172.31.1.113
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1496173121751
     final status: UNDEFINED
     tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
     user: hadoop
17/05/30 19:38:50 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:51 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:52 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:53 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:54 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:55 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:56 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:57 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:58 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:59 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:00 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:01 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:02 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:03 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:04 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:05 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:06 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:07 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:08 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:09 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:10 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:11 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:12 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:13 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:14 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:15 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:16 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:17 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:18 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:19 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:20 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:21 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:22 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:23 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:24 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:24 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1496173121751
     final status: UNDEFINED
     tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
     user: hadoop
17/05/30 19:39:25 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:26 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:27 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:28 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:29 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:30 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:30 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 172.31.0.226
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1496173121751
     final status: UNDEFINED
     tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
     user: hadoop
17/05/30 19:39:31 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:32 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:33 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:34 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:35 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:36 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:37 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:38 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:39 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:40 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:41 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:42 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:43 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:44 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:45 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:46 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:47 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:48 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:49 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:50 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:51 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:52 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:53 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:54 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:55 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:56 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:57 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:58 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:59 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:00 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:01 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:02 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:03 INFO Client: Application report for application_1496172789995_0001 (state: FINISHED)
17/05/30 19:40:03 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 172.31.0.226
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1496173121751
     final status: FAILED
     tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
     user: hadoop
17/05/30 19:40:04 INFO Client: Deleted staging directory hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001
Exception in thread "main" org.apache.spark.SparkException: Application application_1496172789995_0001 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/05/30 19:40:04 INFO ShutdownHookManager: Shutdown hook called
17/05/30 19:40:04 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69

在运行我的 spark 应用程序时，我使用了以下命令：

spark-submit --class Main.Main --deploy-mode cluster --master yarn --num-executors 2 Main.jar 1.txt 2 10

【问题讨论】：

标签： apache-spark amazon-emr


【解决方案1】：

这是 EMR 上的错误： spark-submit --deploy-mode cluster --master yarn

请改用： spark-submit --master yarn-client

【讨论】：

    猜你喜欢
    • 2016-12-19
    • 2021-09-26
    • 2019-09-08
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-10-18
    • 2015-09-03
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode