【发布时间】：2016-07-13 00:01:42
【问题描述】：
我在新安装的 Ubuntu 服务器上设置了一个新的独立 Apache Spark 服务器。我尝试在那里发送我的第一份工作，但我不是很成功。
这是我在本地所做的：
SparkConf conf = new SparkConf().setAppName("myFirstJob").setMaster("local[*]");
JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
javaSparkContext.setLogLevel("WARN");
SQLContext sqlContext = new SQLContext(javaSparkContext);
System.out.println("Hello, Remote Spark v." + javaSparkContext.version());
DataFrame df;
df = sqlContext.read().option("dateFormat", "yyyy-mm-dd")
.json("./src/main/resources/north-carolina-school-performance-data.json");
df = df.withColumn("district", df.col("fields.district"));
df = df.groupBy("district").count().orderBy(df.col("district"));
df.show(150);
有效：它显示北卡罗来纳州的学区名称以及该地区的学校数量：
Hello, Remote Spark v.1.6.1
+--------------------+-----+
| district|count|
+--------------------+-----+
|Alamance-Burlingt...| 34|
|Alexander County ...| 10|
|Alleghany County ...| 4|
|Anson County Schools| 10|
| Ashe County Schools| 5|
|Asheboro City Sch...| 8|
...
现在，如果我将第一行更改为：
SparkConf conf = new SparkConf().setAppName("myFirstJob").setMaster("spark://10.0.100.120:7077");
很顺利：
Hello, Remote Spark v.1.6.1
16/07/12 10:58:34 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/07/12 10:58:49 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
第一件奇怪的事（对我来说）是服务器有 Spark 1.6.2。我有点期待看到 1.6.2 作为版本号。
然后在 UI 上，我去那里看看：
如果我点击 app-20160712105816-0011，我会得到：
任何点击其他链接都会将我带到我的本地 Apache Spark 实例。
点击后，我可以看到类似：
如果我查看服务器上的日志，我会看到：
16/07/12 10:37:00 INFO Master: Registered app myFirstJob with ID app-20160712103700-0009
16/07/12 10:37:03 INFO Master: Received unregister request from application app-20160712103700-0009
16/07/12 10:37:03 INFO Master: Removing app app-20160712103700-0009
16/07/12 10:37:03 INFO Master: 10.0.100.100:54396 got disassociated, removing it.
16/07/12 10:37:03 INFO Master: 10.0.100.100:54392 got disassociated, removing it.
16/07/12 10:50:44 INFO Master: Registering app myFirstJob
16/07/12 10:50:44 INFO Master: Registered app myFirstJob with ID app-20160712105044-0010
16/07/12 10:51:20 INFO Master: Received unregister request from application app-20160712105044-0010
16/07/12 10:51:20 INFO Master: Removing app app-20160712105044-0010
16/07/12 10:51:20 INFO Master: 10.0.100.100:54682 got disassociated, removing it.
16/07/12 10:51:20 INFO Master: 10.0.100.100:54680 got disassociated, removing it.
16/07/12 10:58:16 INFO Master: Registering app myFirstJob
16/07/12 10:58:16 INFO Master: Registered app myFirstJob with ID app-20160712105816-0011
在我看来一切都很好......
我在Apache Spark Server installation requires Hadoop? Not automatically installed? 有一个先前的问题（未解决），使用相同的环境，但这是一个完全不同的 - 小得多的 - 应用程序。
有什么线索吗？
【问题讨论】：
标签： java apache-spark