在 PYSPARK 配置中绕过 SSL 验证

Question

当我在代码下面运行时，它会给我错误。

from delta import configure_spark_with_delta_pip
from pyspark.sql import SparkSession

builder = SparkSession.builder.appName("MyApp") \
        .master("local[*]")\
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
        .config("spark.jars.packages", "io.delta:delta-core_2.12:1.1.0") \
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    
spark = configure_spark_with_delta_pip(builder).getOrCreate()

spark.sql("""CREATE IF NOT EXISTS TABLE s (
                              sd int)
                              USING DELTA""")

错误如下：

:::: 错误 url https://repo1.maven.org/maven2/io/delta/delta-core_2.12/1.1.0/delta-core_2.12-1.1.0.pom 处的服务器访问错误（javax.net.ssl.SSLHandshakeException：PKIX 路径构建失败：sun.security.provider.certpath.SunCertPathBuilderException：无法找到请求目标的有效证书路径）

    Server access error at url https://repo1.maven.org/maven2/io/delta/delta-core_2.12/1.1.0/delta-core_2.12-1.1.0.jar (javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

    Server access error at url https://repos.spark-packages.org/io/delta/delta-core_2.12/1.1.0/delta-core_2.12-1.1.0.pom (javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

    Server access error at url https://repos.spark-packages.org/io/delta/delta-core_2.12/1.1.0/delta-core_2.12-1.1.0.jar (javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

:: 使用详细或调试消息级别了解更多详细信息 线程“主”java.lang.RuntimeException 中的异常：[未解决的依赖项：io.delta#delta-core_2.12;1.1.0：未找到] 在 org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1447) 在 org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185) 在 org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308) 在 org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898) 在 org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) 在 org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) 在 org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) 在 org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) 在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) 在 org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 回溯（最近一次通话最后）： 文件“/home/abhi/projects/dal/modules/data-management/test/job_tests.py”，第 17 行，在 spark = configure_spark_with_delta_pip(builder).getOrCreate() 文件“/home/abhi/.local/lib/python3.8/site-packages/pyspark/sql/session.py”，第 228 行，在 getOrCreate sc = SparkContext.getOrCreate(sparkConf) 文件“/home/abhi/.local/lib/python3.8/site-packages/pyspark/context.py”，第 392 行，在 getOrCreate SparkContext(conf=conf 或 SparkConf()) 文件“/home/abhi/.local/lib/python3.8/site-packages/pyspark/context.py”，第 144 行，在在里面SparkContext._ensure_initialized(self, gateway=gateway, conf=conf) _ensure_initialized 中的文件“/home/abhi/.local/lib/python3.8/site-packages/pyspark/context.py”，第 339 行 SparkContext._gateway = 网关或launch_gateway(conf) 文件“/home/abhi/.local/lib/python3.8/site-packages/pyspark/java_gateway.py”，第 108 行，在 launch_gateway raise RuntimeError("Java 网关进程在发送其端口号之前已退出") RuntimeError：Java 网关进程在发送其端口号之前已退出

++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++

我试图绕过 SSL 并使用“--no-check-certificate”运行 WGET，它起作用了。

wget --ca-certificate=/etc/ssl/certs/ca-certificates.crt  https://repo1.maven.org/maven2/io/delta/delta-core_2.12/1.0.0/delta-core_2.12-1.0.0.pom  --no-check-certificate

我需要帮助以在 PYSPARK 中以相同的方式绕过 SSL。请帮忙。

Answer 1

我正在努力解决同样的问题。您找到解决方案了吗？