sbt 运行后 SparkContext 未初始化答案

【问题标题】：SparkContext Not Being Initialized after sbt runsbt 运行后 SparkContext 未初始化
【发布时间】：2016-10-23 20:48:46
【问题描述】：

下面有我的build.sbt 文件：

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)

我在src/main/scala/example.scala 中也有example.scala：

import org.apache.spark._
import org.apache.spark.SparkContext._

object WordCount {
    def main(args: Array[String]) {
      val conf = new SparkConf().setAppName("wordCount").setMaster("local")
      val sc = new SparkContext(conf)
      val input =  sc.textFile("food.txt")
      val words = input.flatMap(line => line.split(" "))
      val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}
      counts.saveAsTextFile("output.txt")
    }
}

由于某种原因，当我在我的根目录（而不是 src/main/scala）中执行 sbt run 时，我得到了错误：

[info] Running WordCount 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/06/21 22:05:08 INFO SparkContext: Running Spark version 1.6.1
16/06/21 22:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/21 22:05:09 ERROR SparkContext: Error initializing SparkContext.
java.net.UnknownHostException: LM-SFA-11002982: LM-SFA-11002982: nodename nor servname provided, or not known
    at java.net.InetAddress.getLocalHost(InetAddress.java:1475)
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:788)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:781)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:781)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:838)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:838)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:838)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:420)
    at WordCount$.main(exam.scala:8)
    at WordCount.main(exam.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sbt.Run.invokeMain(Run.scala:67)
    at sbt.Run.run0(Run.scala:61)
    at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
    at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
    at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
    at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
    at sbt.Logger$$anon$4.apply(Logger.scala:84)
    at sbt.TrapExit$App.run(TrapExit.scala:248)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: LM-SFA-11002982: nodename nor servname provided, or not known
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1471)
    ... 23 more
16/06/21 22:05:09 INFO SparkContext: Successfully stopped SparkContext

有人可以向我解释此错误中所述的问题吗？这是因为我的依赖项没有正确安装还是因为其他原因？

【问题讨论】：

标签： scala sbt

【解决方案1】：

您的系统的主机名似乎无法解析为 IP 地址。

作为一种 [相当蹩脚] 的解决方法，您可以尝试：

echo "127.0.0.1 LM-SFA-11002982" | sudo tee -a /etc/hosts

【讨论】：

您静态使用的主机名可以更改。这可能会发生，例如当您将机器连接到另一个网络时。根据您的操作系统，有更好的方法来设置主机名解析，例如通过hostname 命令。
在 Mac 上，sudo hostname LM-SFA-11002982.local 完成这项工作