在 spark 工作空间中创建 Derby Metastore 目录

Question

我安装了 spark 2.1.0 并与安装了 eclipse 和 hive2 集成，并且在 Mysql 中配置的 Metastore 还将 hive-site.xml 文件放在 spark >> conf 文件夹中。我正在尝试从 Eclipse 访问 hive 中已经存在的表。 当我执行程序 metastore 文件夹并在 spark 工作区中创建 derby.log 文件时，eclipse 控制台显示以下信息：

Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
17/06/13 18:26:43 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/06/13 18:26:43 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/06/13 18:26:43 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
17/06/13 18:26:43 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
17/06/13 18:26:43 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
17/06/13 18:26:43 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL

spark 找不到配置的 mysql metastore 数据库

同样报错

Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':

代码：

import org.apache.spark.SparkContext, org.apache.spark.SparkConf
import com.typesafe.config._
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession

object hivecore {

  def main(args: Array[String]) {

val warehouseLocation = "hdfs://HADOOPMASTER:54310/user/hive/warehouse"

val spark = SparkSession
  .builder().master("local[*]")
  .appName("hivecore")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

import spark.implicits._
import spark.sql

sql("SELECT * FROM sample.source").show()

}
}

构建.sbt

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "com.typesafe" % "config" % "1.3.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.42"

注意：我可以从 Spark-shell 访问配置单元表

谢谢

Answer 1

当您输入context.setMaster(local) 时，它可能不会查找您在集群中设置的火花配置；特别是当你从 ECLIPSE 触发它时。

用它做一个罐子；并从 cmd 触发为spark-submit --class <main class package> --master spark://207.184.161.138:7077 --deploy-mode client

主 ip:spark://207.184.161.138:7077 应替换为集群的 ip 和 spark 端口。

并且，记得初始化 HiveContext 以触发对底层 HIVE 的查询。

val hc = new HiveContext(sc)
hc.sql("SELECT * FROM ...")