错误 SparkContext：初始化 SparkContext 时出错。错误实用程序：线程主线程中未捕获的异常答案

【问题标题】：ERROR SparkContext: Error initializing SparkContext.ERROR Utils: Uncaught exception in thread main错误 SparkContext：初始化 SparkContext 时出错。错误实用程序：线程主线程中未捕获的异常
【发布时间】：2020-09-26 13:32:33
【问题描述】：

我以为它可以工作，但我实际上失败了

import math._
import org.apache.spark.sql.SparkSession

object Position {
  def main(args: Array[String]): Unit = {
    // create Spark DataFrame with Spark configuration
    val spark= SparkSession.builder().getOrCreate()
    // Read csv with DataFrame
    val file1 = spark.read.csv("file:///home/aaron/Downloads/taxi_gps.txt")
    val file2 = spark.read.csv("file:///home/aaron/Downloads/district.txt")
    //change the name
    val new_file1= file1.withColumnRenamed("_c4","lat")
                        .withColumnRenamed("_c5","lon")
    val new_file2= file2.withColumnRenamed("_c0","dis")
                        .withColumnRenamed("_1","lat")
                        .withColumnRenamed("_2","lon")
                        .withColumnRenamed("_c3","r")
    //geo code
    def haversine(lat1:Double, lon1:Double, lat2:Double, lon2:Double): Double ={
      val R = 6372.8  //radius in km
      val dLat=(lat2 - lat1).toRadians
      val dLon=(lon2 - lon1).toRadians
      val a = pow(sin(dLat/2),2) + pow(sin(dLon/2),2) * cos(lat1.toRadians) * cos(lat2.toRadians)
      val c = 2 * asin(sqrt(a))
      R * c
    }
    //count
    new_file2.foreach(row => {
      val district = row.getAs[Float]("dis")
      val lon = row.getAs[Float]("lon")
      val lat = row.getAs[Float]("lat")
      val distance = row.getAs[Float]("r")
      var temp = 0
      new_file1.foreach(taxi => {
        val taxiLon = taxi.getAs[Float]("lon")
        val taxiLat = taxi.getAs[Float]("lat")
        if(haversine(lat,lon,taxiLat,taxiLon) <= distance) {
          temp+=1
        }
      })
      println(s"district:$district temp=$temp")
    })
   }
  }

这是结果

  20/06/07 23:04:11 ERROR SparkContext: Error initializing SparkContext.
  org.apache.spark.SparkException: A master URL must be set in your configuration
  ......
  20/06/07 23:04:11 ERROR Utils: Uncaught exception in thread main
  java.lang.NullPointerException
  ......
  20/06/07 23:04:11 INFO SparkContext: Successfully stopped SparkContext
  Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration

我不确定因为这似乎是 Spark，所以在 DF 中使用 DF 是该程序的唯一错误。我不熟悉 scala 和 spark，这对我来说是一个相当棘手的问题。希望大家能帮帮我，谢谢！

【问题讨论】：

重复stackoverflow.com/questions/42032169/…
如果以下解决方案有帮助，请接受或支持.. :)

标签： scala dataframe apache-spark foreach

【解决方案1】：

你的例外是org.apache.spark.SparkException: A master URL must be set in your configuration set master url in master function。

我希望您在某些 IDE 中运行代码。如果是，请将您的代码中的 val spark= SparkSession.builder().getOrCreate() 替换为 val spark= SparkSession.builder().master("local[*]").getOrCreate()。

或者如果您使用spark-submit 执行此代码，请尝试添加此--master yarn。

【讨论】：