火花异常：java.lang.NoSuchMethodError：org.apache.http.conn.ssl.SSLConnectionSocketFactory

Question

当我在本地代码中运行我的代码时，它运行良好。但是，当我在集群中运行它时，我的 Jar 文件中似乎遗漏了一些依赖项：

18/05/23 10:46:02 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifier;)V
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
        at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)

这是 SBT 文件：

scalaVersion := "2.11.8"
val sparkVersion = "2.2.0"
val mahoutVersion = "0.13.1"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided",
  "org.sedis" %% "sedis" % "1.2.2",
  "org.scalactic" %% "scalactic" % "3.0.0",
  "org.scalatest" %% "scalatest" % "3.0.0" % "test",
  "com.github.nscala-time" %% "nscala-time" % "2.14.0",
  "com.github.scopt" %% "scopt" % "3.3.0"
)
unmanagedSourceDirectories in Compile += baseDirectory.value / "lib"
resolvers += "typesafe repo" at " http://repo.typesafe.com/typesafe/releases/"
resolvers += Resolver.mavenLocal
assemblyMergeStrategy in assembly := {
  case "plugin.properties" => MergeStrategy.discard
  //case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case PathList("org", "joda", "time", "base", "BaseDateTime.class") => MergeStrategy.first
  case PathList(ps @ _*) if ps.last endsWith "package-info.class" => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "WrapDynaClass.class") => MergeStrategy.first
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "aop", xs @ _*) => MergeStrategy.first
  case PathList("org", "aopalliance", "intercept", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
  case PathList("org", "apache", "commons", "collections", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "locale", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", "converters", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "beanutils", xs @ _*) => MergeStrategy.first
  case PathList("assets", "org", "apache", "commons", "math3", "exception", "util", "LocalizedFormats_fr.properties") => MergeStrategy.first
  case PathList("com", "google", "common", "annotations", "Beta.class") => MergeStrategy.first
  case PathList("com", "google", "common", "base", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "cache", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "collect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "eventbus", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "hash", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "io", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "math", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "net", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "primitives", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "reflect", xs @ _*) => MergeStrategy.first
  case PathList("com", "google", "common", "util", "concurrent", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "differentiation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "function", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "integration", "gauss", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "interpolation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "polynomials", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "analysis", "solvers", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "complex", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "dfp", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "distribution", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "exception", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "filter", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "fraction", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "genetics", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "oned", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "threed", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "euclidean", "twod", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "geometry", "partitioning", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ml", "distance", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "events", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "nonstiff", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "ode", "sampling", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "gradient", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "scalar", "noderiv", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "nonlinear", "vector", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optim", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "direct", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "fitting", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "general", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "linear", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "optimization", "univariate", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "primes", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "random", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "special", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "clustering", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "correlation", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "moment", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "descriptive", "summary", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "inference", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "ranking", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "stat", "regression", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "transform", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "math3", "util", xs @ _*) => MergeStrategy.first
  case PathList("org", "apache", "commons", "logging", xs @ _*) => MergeStrategy.first
  case PathList("overview.html") => MergeStrategy.first
  case PathList("mime.types") => MergeStrategy.first
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

我的罐子里有什么问题？在 IntelliJ IDEA 中，我选择了类检查选项并粘贴了 SSLConnectionSocketFactory。这个类被发现并引用了httpclient-4.3.6.jar

Answer 1

Spark 带有许多库，您可能与其中一个有冲突。

我想我对 io.netty.netty-all 包也有类似的问题。我们最终将服务器上的那个包升级到稍微更新的次要版本，但这是因为我们正在为部署在节点上的 spark 构建集成。

您可以尝试使用这些参数部署您的 Spark 应用

--conf 'spark.driver.extraJavaOptions=-verbose:class'
--conf 'spark.executor.extraJavaOptions=-verbose:class'

这些允许您查看类路径。我很确定你会在那里看到冲突的包。我会开始检查加载了哪个版本的 netty，因为我很确定它取决于 httpclient。