从 Hive 表创建数据框时找不到表答案

【问题标题】：Table not found while creating dataframe from Hive Table从 Hive 表创建数据框时找不到表
【发布时间】：2017-01-11 22:40:54
【问题描述】：

Spark 命令：

spark-submit \
--class com.dev.SparkHiveToHdfs \
--jars /home/dev/dbJars/datanucleus-api-jdo-3.2.6.jar,/home/dev/dbJars/datanucleus-rdbms-3.2.9.jar,/home/dev/dbJars/datanucleus-core-3.2.10.jar \
--master yarn-cluster \
--name DCA_SPARK_JOB \
/home/dev/dbJars/data-connector-spark.jar dev.emp

data-connector-spark.jar 包含以下代码：

public class SparkHiveToHdfs {

  public static void main(String[] args) throws Exception {

    String hiveTableNameWithSchema = args[0];

    SparkConf conf = new SparkConf(true).setMaster("yarn-cluster").setAppName("DCA_HIVE_HDFS");
    SparkContext sc = new SparkContext(conf);
    HiveContext hc = new HiveContext(sc);
    DataFrame df = hc.sql("select * from "+hiveTableNameWithSchema);
    df.printSchema();
 }
}

hive-site.xml 中的属性$SPARK_HOME/conf：

  <property>
      <name>hive.metastore.client.connect.retry.delay</name>
      <value>5</value>
    </property>

    <property>
      <name>hive.metastore.client.socket.timeout</name>
      <value>1800</value>
    </property>

    <property>
      <name>hive.metastore.connect.retries</name>
      <value>24</value>
    </property>

    <property>
      <name>hive.metastore.uris</name>
      <value>thrift://xxxx:9083</value>
    </property>

    <property>
      <name>hive.server2.enable.doAs</name>
      <value>false</value>
    </property>

    <property>
      <name>hive.server2.thrift.port</name>
      <value>10000</value>
    </property>

    <property>
      <name>hive.server2.transport.mode</name>
      <value>binary</value>
    </property>

错误日志：

ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Table not found: `dev`.`emp`; line 1 pos 18
org.apache.spark.sql.AnalysisException: Table not found: `dev`.`emp`; line 1 pos 18
    at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:54)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:50)
    at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:121)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
    at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:120)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:120)
    at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:50)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
    at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
    at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817)
    at com.impetus.idw.data.connector.SparkHiveToHdfs.main(SparkHiveToHdfs.java:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:559)

【问题讨论】：

请使用 --files 选项在您的 spark-submit 中传递 hive-site.xml 文件
请在 SQL 中尝试：show databases。然后使用use <database> 选择数据库，然后尝试show tables 看看你能看到哪些表。
@RajatMishra 是对的。以yarn-cluster 提交作业时，您必须提供hive-site.xml。
@RajatMishra 我不能在我的情况下添加--files。如果它在$SPARK_HOME/conf 下，它应该可以工作。如果我的理解有误，请告诉我？
@devツ no.. 实际上在集群模式下，我们需要指定 hive-site.xml 或者您需要手动将 hive-site.xml 文件放在 $SPARK_HOME/conf 文件夹中节点。

标签： apache-spark hive

【解决方案1】：

请尝试在 spark 提交命令中传递 hive-site.xml。

spark-submit \
--class com.dev.SparkHiveToHdfs \
--jars /home/dev/dbJars/datanucleus-api-jdo-3.2.6.jar,/home/dev/dbJars/datanucleus-rdbms-3.2.9.jar,/home/dev/dbJars/datanucleus-core-3.2.10.jar \
--master yarn-cluster \
--name DCA_SPARK_JOB \
--files hive-site.xml
/home/dev/dbJars/data-connector-spark.jar dev.emp

【讨论】：