【发布时间】：2020-10-09 19:03:34
【问题描述】：
我一直在逐字母跟踪Databricks Official guide，将初始化脚本包含到 Databricks 集群（一个作业集群，不是实时集群）中，以安装 Azure CosmosDB Jar Library(PySpark 2.4)。
这是我的初始化脚本：
dbutils.fs.put("/databricks/scripts/cosmosdb-install.sh","""
#!/bin/bash
wget --quiet -O /mnt/driver-daemon/jars/azure-cosmosdb-spark_2.4.0_2.11-2.1.2-uber.jar https://search.maven.org/remotecontent?filepath=com/microsoft/azure/azure-cosmosdb-spark_2.4.0_2.11/2.1.2/azure-cosmosdb-spark_2.4.0_2.11-2.1.2-uber.jar""", True)
我验证它是否存在：display(dbutils.fs.ls("dbfs:/databricks/scripts/cosmosdb-install.sh"))。
然后我通过 Azure 数据工厂启动我的作业集群，当它到达 CosmosDB 代码时，它一直失败：
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:143)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:122)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:378)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:328)
at com.databricks.logging.UsageLogging$.<init>(UsageLogging.scala:39)
at com.databricks.logging.UsageLogging$.<clinit>(UsageLogging.scala)
at com.databricks.logging.UsageLogging$class.recordUsage(UsageLogging.scala:597)
at com.databricks.DatabricksMain.recordUsage(DatabricksMain.scala:63)
at com.databricks.logging.UsageLogging$class.recordEvent(UsageLogging.scala:699)
at com.databricks.DatabricksMain.recordEvent(DatabricksMain.scala:63)
at com.databricks.DatabricksMain.main(DatabricksMain.scala:83)
at com.databricks.backend.daemon.driver.DriverDaemon.main(DriverDaemon.scala)
Caused by: java.lang.SecurityException: class "org.slf4j.spi.LocationAwareLogger"'s signer information does not match signer information of other classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:891)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:661)
at java.lang.ClassLoader.defineClass(ClassLoader.java:754)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
... 12 more
在交互式集群中测试并通过 UI 按钮安装库时，相同的代码可以 100% 正常运行。
我测试此代码的交互式集群与我通过 Azure 数据工厂配置的作业集群具有相同的运行时版本：6.6 (includes Apache Spark 2.4.5, Scala 2.11)。
这也是 Databricks Web UI 中作业集群的屏幕截图：
这是 PySpark 代码：
df = spark.read.format(
"com.microsoft.azure.cosmosdb.spark").options(**readConfig).load()
有人可以复制这个吗，或者有人可以解释一下吗？
【问题讨论】：
标签： azure-cosmosdb databricks azure-databricks