hive spark子进程在连接之前退出

Question

当我运行像

这样的简单查询时，我遇到了 Hive on spark 问题

select * from table_name

在 hive 控制台上一切正常，但是当我执行时

select count(*) from table_name 

查询以以下内容终止 错误：

Query ID = ab_20160515134700_795fc14c-e89b-4172-bcc6-0cfcffadcd88
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Spark Job = d5e1856e-de67-4e2d-a914-ca1aae324b7f
Status: SENT
Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

---

版本：

hadoop-2.7.2
apache-hive-2.0.0
spark-1.6.0-bin-hadoop2
scala: 2.11.8

我已经设置： hive-site.xml 中的 spark.master 现在我得到：java.util.concurrent.ExecutionException：java.lang.RuntimeException：取消客户端'8ffe7ea3-aaf4-456c-ae18-23c572a766c5'。错误：子进程在 io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) ~[netty-all-4.0.23.Final.jar:4.0.23.Final] 在 org. apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:101) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java :80) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:98) [hive-exec-2.0。 0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:94) [hive-exec-2.0.0.jar:2.0.0] 在 org. org.apache.hadoop.hive.ql.exec 中的 apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:63) [hive-exec-2.0.0.jar:2.0.0]。 spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSessio n(SparkSessionManagerImpl.java:114) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131) [hive -exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:106) [hive-exec-2.0.0.jar:2.0 .0] 在 org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive。 ql.exec.TaskRunner.runSequential(TaskRunner.java:101) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1840) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1584) [hive-exec-2.0.0.jar:2.0.0 ] 在 org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1361) [hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.Driver。运行（Driver.java:1184）[hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.ql.Driver.run（Driver.java:1172）[hive-exec-2.0 .0.jar:2.0.0] 在 org.apache.hadoop.hive.cli.CliDriver.processLocalCmd (CliDriver.java:233) [hive-cli-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) [hive-cli-2.0. 0.jar:2.0.0] 在 org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400) [hive-cli-2.0.0.jar:2.0.0] 在 org.apache.hadoop .hive.cli.CliDriver.executeDriver(CliDriver.java:778) [hive-cli-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717 ) [hive-cli-2.0.0.jar:2.0.0] 在 org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645) [hive-cli-2.0.0.jar:2.0. 0] 在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77] 在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77] 在 sun.reflect。 DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77] at org.apache.hadoop。 org.apache.hadoop.util.RunJa 上的 util.RunJar.run(RunJar.java:221) [spark-assembly-1.6.0-hadoop2.6.0.jar:1.6.0] r.main(RunJar.java:136) [spark-assembly-1.6.0-hadoop2.6.0.jar:1.6.0] 引起：java.lang.RuntimeException: Cancel client '8ffe7ea3-aaf4-456c-ae18-23c572a766c5 '。错误：在 org.apache.hive.spark.client.rpc.RpcServer.cancelClient(RpcServer.java:180) ~[hive-exec-2.0.0.jar:2.0.0] 在 org.apache.hive.spark.client.rpc.RpcServer.cancelClient(RpcServer.java:180) 连接之前退出子进程。 apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:450) ~[hive-exec-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_77] 16/05/16 18:00:33 [Driver]: WARN client.SparkClientImpl: Child process exited with code 1

---

我已经构建了 Spark 1.6.1 和 hive 2.0.0 所以错误已更改为：

Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Iterable
    at org.apache.hadoop.hive.ql.parse.spark.GenSparkProcContext.<init>(GenSparkProcContext.java:163)
    at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.generateTaskTree(SparkCompiler.java:195)
    at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:258)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10861)
    at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:239)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:437)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:329)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1158)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1253)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: scala.collection.Iterable
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Answer 1

我在 Hive 2.0.0 和 Spark 1.6.1 上遇到了与您相同的问题。如前所述，它已在issues.apache.org/jira/browse/HIVE-9970 进行了讨论。

话虽如此，对于 Hive：

1. 下载 Hive 源码包
2. 在 pom.xml 中设置正确的 Hadoop/Spark/Tez 版本
3. 扩展 maven 的内存限制。我用export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
4. 构建 Hive 使用 Maven：mvn clean package -Pdist -DskipTests
5. 结果在：packaging/target/apache-hive-2.x.y-bin。配置 hive-site.xml。

对于 Spark：

1. 下载 Spark 源码包
2. 在 pom.xml 中设置正确的 Hadoop 版本
3. 使用 ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.6,parquet-provided" 在没有 Hive 的情况下构建 Spark
4. dist/ 的结果。配置 spark-defaults.conf。

由于您在没有 Hadoop 的情况下构建了 Spark，因此您需要将 Hadoop 包 jar 路径包含到 $SPARK_DIST_CLASSPATH。见this documentation page。此外，您可以阅读Hive on Spark guide 作为参考。