SQL Server 的 JDBC 驱动程序（java.sql.SQLException：没有合适的驱动程序）

Question

在将我的 python 代码推送到粘合作业之前，我试图在本地对其进行测试，但遇到了以下问题：

py4j.protocol.Py4JJavaError: An error occurred while calling o35.load.
: java.sql.SQLException: No suitable driver
        at java.sql.DriverManager.getDriver(Unknown Source)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:108)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:108)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:38)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
        at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Unknown Source)

我的代码如下： 我尝试使用config 参数，但也没有用。 jdbc驱动的jar文件与python脚本位于同一位置。

.config("spark.driver.extraClassPath", "C:/python_code/mssql-jdbc-9.2.1.jre15.jar") \

脚本触发：C:\pyth_code>spark-submit code.py

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()


jdbc_df = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlserver://mssql-server.xxxxxxx.us-east-1.rds.amazonaws.com:xxxx") \
    .option("query", "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, IS_NULLABLE FROM db_name.information_schema.columns WHERE TABLE_SCHEMA = 'RDM'") \
    .option("user", user) \
    .option("password", pass) \
    .load()

jdbc_df.show()

环境变量：

MS 的 jdbc 驱动位置：

更新 - 使用gluecontext修复了驱动问题，但提供了新的：

glue_context = GlueContext(SparkContext.getOrCreate())
jdbc_df_t = glue_context.spark_session.read.format("org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider") \
    .option("url", "jdbc:sqlserver://mssql-rds-development.xxxxx.us-east-1.rds.amazonaws.com:xxxx") \
    .option("query", "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, IS_NULLABLE FROM information_schema.columns") \
    .option("user", user) \
    .option("password", pass) \
    .load()

使用gluecontext后出错：

  File "C:\Program Files\Anaconda3\lib\site-packages\awsglue\context.py", line 45, in __init__
    self._glue_scala_context = self._get_glue_scala_context(**options)
  File "C:\Program Files\Anaconda3\lib\site-packages\awsglue\context.py", line 66, in _get_glue_scala_context
    return self._jvm.GlueContext(self._jsc.sc())
TypeError: 'JavaPackage' object is not callable

Answer 1

我不希望您需要使用自定义 JDBC 驱动程序，请在没有它​​的情况下尝试以下操作（如我最近在此处的回答： Create dynamic frame from options (from rds - mysql) providing a custom query with where clause)

这个块应该可以解决问题：

from awsglue.context import GlueContext
from pyspark.context import SparkContext

glue_context = GlueContext(SparkContext.getOrCreate())

jdbc_df = glue_context.spark_session.read.format("org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider") \
    .option("url", "jdbc:sqlserver://mssql-server.xxxxxxx.us-east-1.rds.amazonaws.com:xxxx") \
    .option("dbtable", "(SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, IS_NULLABLE FROM db_name.information_schema.columns WHERE TABLE_SCHEMA = 'RDM')") \
    .option("user", user) \
    .option("password", pass) \
    .load()