如何在本地运行胶水作业？答案

【问题标题】：How run glue job locally?如何在本地运行胶水作业？
【发布时间】：2019-11-21 08:45:47
【问题描述】：

我已经按照here 的描述设置了项目。但是代码：

import com.amazonaws.services.glue.{AWSGlueClientBuilder, GlueContext}
import org.apache.spark.SparkContext
import org.slf4j.LoggerFactory

object MyGlueJob {
  private val logger = LoggerFactory.getLogger(getClass)
  def main(sysArgs: Array[String]) {

    val spark: SparkContext = SparkContext.getOrCreate()
    val glueContext: GlueContext = new GlueContext(spark)
    val awsGlueClient = AWSGlueClientBuilder.defaultClient
  }
}

因错误而失败：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/11/21 15:40:32 INFO SparkContext: Running Spark version 2.4.3
19/11/21 15:40:33 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2544)
    at MyGlueJob$.main(MyGlueJob.scala:13)
    at MyGlueJob.main(MyGlueJob.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
19/11/21 15:40:33 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
    at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$postApplicationEnd(SparkContext.scala:2416)
    at org.apache.spark.SparkContext$$anonfun$stop$1.apply$mcV$sp(SparkContext.scala:1931)
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1340)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1930)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:585)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2544)
    at MyGlueJob$.main(MyGlueJob.scala:13)
    at MyGlueJob.main(MyGlueJob.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
19/11/21 15:40:33 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2544)
    at MyGlueJob$.main(MyGlueJob.scala:13)
    at MyGlueJob.main(MyGlueJob.scala)
    ... 5 more

很明显应该设置主 url，但是如何从命令行或系统变量中设置呢？（例如不接触代码）

我还 [阅读] --master 参数可以解决问题，但将其添加到 args 没有任何作用（这里是 Intellij Idea 运行配置）：

关键问题是在本地运行胶水作业，并且能够在不接触代码的情况下在aws中运行它，有可能吗？

【问题讨论】：

标签： scala amazon-web-services apache-spark aws-glue

【解决方案1】：

您可以显式创建 Spark 会话并设置所需的任何参数。但我不能说这最终会在 Glue 中起作用。以下是我用来在本地测试 Spark 作业的本地会话，即使我最终在 Glue 中运行它们。我只测试纯火花代码。

  lazy val spark: SparkSession = {
    UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("hduser"))
    SparkSession
      .builder()
      .master("local")
      .appName("spark unit test")
      .getOrCreate()
  }

关键问题是在本地运行胶水作业，并且能够在不接触代码的情况下在aws中运行它，有可能吗？

可以使用开发端点和 Zeppelin 运行任何代码。见aws docs。

【讨论】：