无法在我的本地机器上运行 data-validator_2.11-0.12.1.jar

Question

我正在尝试使用目标数据验证器，但是当我在 cmd 中运行它时，它会显示如下错误：

D:\Spark\DemoTime>spark-submit --master local data-validator_2.11-0.12.1.jar --help
    21/12/28 15:05:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class.
            at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:972)
            at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:492)
            at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
            at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
            at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
            at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
            at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

但如果我给出主类名称，它会显示为：

D:\Spark\DemoTime>spark-submit --class com.target.data_validator.Main data-validator_2.11-0.13.0.jar --help
Exception in thread "main" java.lang.NoClassDefFoundError: com/typesafe/scalalogging/Logger
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.typesafe.scalalogging.Logger
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 13 more
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

我正在关注目标中的数据验证器工具： https://github.com/target/data-validator

但我无法解决这个问题。

您可以在以下位置找到 jar 文件： https://github.com/target/data-validator/releases

因此，我请求您帮助运行此工具。 如果您还需要什么，请告诉我。

谢谢。

Answer 1

来自 GitHub 的自述文件：

组装肥罐：sbt clean assembly

spark-submit --master local data-validator-assembly-0.13.0.jar --help

您需要一个 fat jar 的数据验证器 + 其依赖项才能运行它。您只提供了 data-validator 简单的 jar。