使用Greenplum的data direct jdbc驱动（greenplum.jar）读取greenplum表时如何在spark.read.format中指定datasource？

Question

我正在尝试使用 spark 从 Greenplum 上的表中读取数据。我写的代码如下：

  val yearDF = spark.read.format("io.pivotal.greenplum.spark.GreenplumRelationProvider").option("url", connectionUrl)
    .option("server.port","5432")
    .option("dbtable", "tablename")
    .option("dbschema","schemaname")
    .option("user", devUserName)
    .option("password", devPassword)
    .option("partitionColumn","employeeLoc")
    .option("partitions",450)
    .load()
    .where("period_year=2017 and period_num=12")
    .select(gpColSeq map col:_*)
    .withColumn(flagCol, lit(0))

我正在使用greenplum.jar，它提供数据直接 jdbc 驱动程序以使用 Spark 从 greenplum 表中读取数据。 我正在使用以下 spark-submit 命令：

SPARK_MAJOR_VERSION=2 spark-submit --class com.partition.source.YearPartition --master=yarn --conf spark.ui.port=4090 --driver-class-path /home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar --conf spark.jars=/home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar --executor-cores 3 --executor-memory 13G --keytab /home/hdpuser/hdpuser.keytab --principal hdpuser@devuser.COM --files /usr/hdp/current/spark2-client/conf/hive-site.xml,testconnection.properties --name Splinter --conf spark.executor.extraClassPath=/home/hdpuser/jars/greenplum.jar,/home/hdpuser/jars/postgresql-42.1.4.jar splinter_2.11-0.1.jar

当我提交 jar 时，我看到了异常：

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: io.pivotal.greenplum.spark.GreenplumRelationProvider. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:553)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:89)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:304)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
    at com.partition.source.YearPartition$.prepareFinalDF$1(YearPartition.scala:154)
    at com.partition.source.YearPartition$.main(YearPartition.scala:181)
    at com.partition.source.YearPartition.main(YearPartition.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: io.pivotal.greenplum.spark.GreenplumRelationProvider.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22$$anonfun$apply$14.apply(DataSource.scala:537)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22$$anonfun$apply$14.apply(DataSource.scala:537)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22.apply(DataSource.scala:537)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$22.apply(DataSource.scala:537)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:537)

我知道这是由于在数据源语句中使用了 io.pivotal.greenplum.spark.GreenplumRelationProvider，即

spark.read.format("io.pivotal.greenplum.spark.GreenplumRelationProvider")

我尝试了“io.pivotal.greenplum.spark.GreenplumRelationProvider”和“greenplum”，但都导致相同的异常：

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source:

io.pivotal.greenplum.spark.GreenplumRelationProvider。请找到 包裹http://spark.apache.org/third-party-projects.html

在使用数据直接 jdbc jar:greenplum.jar 时，我想不出我应该在语句 spark.read.format 中提供什么作为我的数据源

谁能告诉我如何解决这个问题？

Answer 1

您使用的是什么版本的 greenplum-spark 连接器？

您应该能够在“驱动程序”选项中指定自定义 jdbc 驱动程序。参考http://greenplum-spark.docs.pivotal.io/160/using_the_connector.html#use_custom_jdbcdriver。

您可以按如下方式指定数据源：

spark.read.format("greenplum")