Hbase 0.96 与 Spark v 1.0+答案

【问题标题】：Hbase 0.96 with Spark v 1.0+Hbase 0.96 与 Spark v 1.0+
【发布时间】：2014-08-19 05:37:21
【问题描述】：

这种 Hbase / Spark 版本的组合似乎非常有害。我花了几个小时试图找到各种可行但无济于事的 MergeStrategy。

这里是当前build.sbt的核心：

val sparkVersion = "1.0.0"
// val sparkVersion = "1.1.0-SNAPSHOT"

val hbaseVersion = "0.96.1.1-cdh5.0.2"

libraryDependencies ++= Seq(
    "org.apache.hbase" % "hbase-client" % hbaseVersion,
    "org.apache.hbase" % "hbase-common" % hbaseVersion,
    "org.apache.hbase" % "hbase-server" % hbaseVersion,
    "org.apache.hbase" % "hbase-protocol" % hbaseVersion,
    "org.apache.hbase" % "hbase-examples" % hbaseVersion,
  ("org.apache.spark" % "spark-core_2.10" % sparkVersion  withSources()).excludeAll(ExclusionRule("org.mortbay.jetty")),
  "org.apache.spark" % "spark-sql_2.10" % sparkVersion  withSources()
)

以下是不可避免地重新出现的错误消息：

14/06/27 19:49:24 INFO HttpServer: Starting HTTP Server
[error] (run-main-0) java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
        at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
        at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:136)
        at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:129)
        at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:98)
        at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:98)
        at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:89)
        at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:65)
        at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:58)
        at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:58)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:58)
        at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:66)
        at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:60)
        at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:42)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:222)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
        at com.huawei.swlab.sparkpoc.hbase.HBasePop$.main(HBasePop.scala:31)
        at com.huawei.swlab.sparkpoc.hbase.HBasePop.main(HBasePop.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last *:runMain for the full output.
14/06/27 19:49:44 INFO ConnectionManager: Selector thread was interrupted!
java.lang.RuntimeException: Nonzero exit code: 1

【问题讨论】：

我真的很讨厌依赖问题！ 特别是使用 hbase - hadoop - spark，因为 ST 非常神秘。无论如何，我唯一的猜测是尝试将 spark deps 放在 hbase deps 之上。如果这不起作用，请尝试为 spark 人员创建 JIRA，他们通常会很有帮助：issues.apache.org/jira/browse/SPARK
@samthebest 感谢收看！ Sean Owen 对 spark 邮件列表发表了评论。但是是的，我可能需要听从您的建议并提交 JIRA。

标签： scala apache-spark sbt hbase

【解决方案1】：

我的 Spark/HBase 应用程序遇到了完全相同的异常。我通过将 org.mortbay.jetty 排除规则移动到我的 hbase-server 依赖项来修复它：

libraryDependencies += "org.apache.hbase" % "hbase-server" % "0.98.6-cdh5.2.0" excludeAll ExclusionRule(organization = "org.mortbay.jetty")

如果您将 hadoop-common 作为直接依赖项之一，那么我还发现有必要为 javax.servlet 依赖项创建排除规则：

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.5.0-cdh5.2.0" excludeAll ExclusionRule(organization = "javax.servlet")

我没有改变我的 Spark 依赖项：

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0-cdh5.2.0"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.0-cdh5.2.0"

libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % "1.1.0-cdh5.2.0"

【讨论】：