火花发射器。 java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.<init>答案

【问题标题】：SparkLauncher. java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.<init>火花发射器。 java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml.<init>
【发布时间】：2019-01-13 14:20:48
【问题描述】：

同事们好。我开发了一个基于 sparkLauncher 的应用程序，它运行一个可执行的 jar，其中有 5 个操作。每个操作取决于具体的变量。我有一个主要的 hadoop 集群 spark2.3.0-hadoop2.6.5。它很好用。我的部分工作代码：

 private void runSparkJob(String pathToJar, final LocalDate startDate, final LocalDate endDate) {
        if (executionInProgress.get()) {
            LOGGER.warn("Execution already in progress");
            return;
        }
        Process sparkProcess = null;
        try {
            LOGGER.info("Create SparkLauncher. SparkHome: [{}]. JarPath: [{}].", sparkHome, vmJarPath);
            executionInProgress.set(true);
            sparkProcess = new SparkLauncher()
                    .setAppName(activeOperationProfile)
                    .setSparkHome(sparkHome) //sparkHome folder on main cluster
                    .setAppResource(pathToJar) // jar with 5 operation
                    .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS,
                            String.format("-Drunner.operation-profile=%1$s -Doperation.startDate=%2$s -Doperation.endDate=%3$s", activeOperationProfile, startDate,endDate))
                    .setConf(SparkLauncher.DRIVER_MEMORY, "12G")
                    .redirectToLog(LOGGER.getName())
                    .setMaster("yarn")
                    .launch();

            sparkProcess.waitFor();
            int exitCode = sparkProcess.exitValue();
            if (exitCode != 0) {
                throw new RuntimeException("Illegal exit code. Expected: [0]. Actual: [" + exitCode + "]");
            }

        } catch (IOException | InterruptedException e) {
            LOGGER.error("Error occurred while running SparkApplication.", e);
            throw new RuntimeException(e);
        } finally {
            if (sparkProcess != null && sparkProcess.isAlive()) {
                LOGGER.warn("Process still alive. Try to kill");
                sparkProcess.destroy();
            }
            executionInProgress.set(false);
        }
    }

我已经启动了一个 docker 容器，下载的 spark 2.3.0-hadoop6 在哪里。测试人员需要这个容器。 我将 master 更改为 .setMaster("local")，将带有路径的新配置文件放入 sparkHome、jarsWithOperations 并在没有阴影的情况下打包 jar（尝试使用阴影，但它对我不起作用）。当我尝试运行我的 sparkLaunch 应用程序时，我现在遇到了一个异常：

2018-08-06 14:47:53,150 信息 [n.m.m.b.r.SparkBaseOperationsRunner.runSparkJob] 105：创建火花发射器。火花之家： [/opt/bigtv/spark/spark-2.3.0-bin-hadoop2.6]。罐子路径： [/opt/bigtv/bin/multirating-bigdata-operations-MASTER-SNAPSHOT.jar]。 2018-08-06 14:47:54,905 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63 : 2018-08-06 14:47:54 WARN NativeCodeLoader:62 - 无法加载 native-hadoop 适合您平台的库...使用内置 java 类，其中适用 2018-08-06 14:47:57,042 INFO [o.a.spark.launcher.OutputRedirector.redirect] 63 : 2018-08-06 14:47:57 错误 SpringApplication:842 - 应用程序运行失败 2018-08-06 14:47:57,043 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63： java.lang.NoSuchMethodError： org.yaml.snakeyaml.Yaml.(Lorg/yaml/snakeyaml/constructor/BaseConstructor;Lorg/yaml/snakeyaml/representer/Representer;Lorg/yaml/snakeyaml/DumperOptions;Lorg/yaml/snakeyaml/LoaderOptions;Lorg/yaml/snakeyaml /resolver/Resolver;)V 2018-08-06 14:47:57,043 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.env.OriginTrackedYamlLoader.createYaml(OriginTrackedYamlLoader.java:70) 2018-08-06 14:47:57,043 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.beans.factory.config.YamlProcessor.process(YamlProcessor.java:139) 2018-08-06 14:47:57,044 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.env.OriginTrackedYamlLoader.load(OriginTrackedYamlLoader.java:75) 2018-08-06 14:47:57,044 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.env.YamlPropertySourceLoader.load(YamlPropertySourceLoader.java:50) 2018-08-06 14:47:57,044 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.loadDocuments(ConfigFileApplicationListener.java:547) 2018-08-06 14:47:57,044 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:517) 2018-08-06 14:47:57,045 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.loadForFileExtension(ConfigFileApplicationListener.java:496) 2018-08-06 14:47:57,045 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:464) 2018-08-06 14:47:57,045 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.lambda$null$6(ConfigFileApplicationListener.java:446) 2018-08-06 14:47:57,046 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 java.lang.Iterable.forEach(Iterable.java:75) 2018-08-06 14:47:57,046 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.lambda$load$7(ConfigFileApplicationListener.java:445) 2018-08-06 14:47:57,046 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 java.lang.Iterable.forEach(Iterable.java:75) 2018-08-06 14:47:57,046 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:442) 2018-08-06 14:47:57,046 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener$Loader.load(ConfigFileApplicationListener.java:330) 2018-08-06 14:47:57,047 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.addPropertySources(ConfigFileApplicationListener.java:212) 2018-08-06 14:47:57,047 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.postProcessEnvironment(ConfigFileApplicationListener.java:195) 2018-08-06 14:47:57,047 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:182) 2018-08-06 14:47:57,047 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:168) 2018-08-06 14:47:57,048 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) 2018-08-06 14:47:57,048 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) 2018-08-06 14:47:57,048 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) 2018-08-06 14:47:57,048 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) 2018-08-06 14:47:57,049 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74) 2018-08-06 14:47:57,049 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) 2018-08-06 14:47:57,049 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:358) 2018-08-06 14:47:57,049 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.run(SpringApplication.java:317) 2018-08-06 14:47:57,050 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) 2018-08-06 14:47:57,050 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) 2018-08-06 14:47:57,050 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 net.mediascope.multirating.bigdata.operations.OperationRunner.main(OperationRunner.java:21) 2018-08-06 14:47:57,050 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-08-06 14:47:57,050 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2018-08-06 14:47:57,051 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-08-06 14:47:57,051 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 java.lang.reflect.Method.invoke(Method.java:498) 2018-08-06 14:47:57,051 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) 2018-08-06 14:47:57,051 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.Launcher.launch(Launcher.java:87) 2018-08-06 14:47:57,052 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.Launcher.launch(Launcher.java:50) 2018-08-06 14:47:57,052 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) 2018-08-06 14:47:57,052 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2018-08-06 14:47:57,052 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2018-08-06 14:47:57,053 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2018-08-06 14:47:57,053 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 java.lang.reflect.Method.invoke(Method.java:498) 2018-08-06 14:47:57,053 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) 2018-08-06 14:47:57,053 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879) 2018-08-06 14:47:57,054 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) 2018-08-06 14:47:57,054 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) 2018-08-06 14:47:57,054 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) 2018-08-06 14:47:57,054 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63：在 org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2018-08-06 14:47:57,058 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63： 2018-08-06 14:47:57 信息 ShutdownHookManager:54 - 关机挂钩致电 2018-08-06 14:47:57,060 信息 [o.a.spark.launcher.OutputRedirector.redirect] 63 : 2018-08-06 14:47:57 信息 ShutdownHookManager:54 - 删除目录 /tmp/spark-55b54924-e628-43fe-9e43-ed34d7f35a8b 2018-08-06 14:47:57,151 信息 [o.s.b.a.l.ConditionEvaluationReportLoggingListener.logAutoConfigurationReport] 101：

启动 ApplicationContext 时出错。显示条件报告在启用“调试”的情况下重新运行您的应用程序。

在我的项目中，我有来自 spring 5.0 的蛇 yaml 1.19，没有其他依赖项。我不明白问题是什么，也许当我将它放入 docker 容器手册时，除了 spark 之外还需要安装其他东西。

来自带有操作的模块的 Pom：

<dependencies>
        <dependency>
            <groupId>net.mediascope</groupId>
            <artifactId>multirating-bigdata-core</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
        <!-- Data Base -->
        <dependency>
            <groupId>org.jdbi</groupId>
            <artifactId>jdbi</artifactId>
            <version>2.71</version>
        </dependency>

        <dependency>
            <groupId>com.microsoft.sqlserver</groupId>
            <artifactId>sqljdbc42</artifactId>
            <version>4.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>commons-compiler</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
        </dependency>
        <dependency>
            <groupId>net.sourceforge.jtds</groupId>
            <artifactId>jtds</artifactId>
            <version>1.3.1</version>
        </dependency>
    </dependencies>

    <profiles>
        <profile>
            <id>local</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <configuration>
                            <profiles>
                                <profile>${profile.active}</profile>
                            </profiles>
                            <executable>true</executable>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>
        <profile>
            <id>hadoop</id>
            <build>
                <!--Необходимо для адаптации Spring-Boot приложения под запуск через Spark-->
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-shade-plugin</artifactId>
                        <version>2.3</version>
                        <executions>
                            <execution>
                                <phase>package</phase>
                                <goals>
                                    <goal>shade</goal>
                                </goals>
                                <configuration>
                                    <transformers>
                                        <transformer
                                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                            <resource>META-INF/spring.handlers</resource>
                                        </transformer>
                                        <transformer
                                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                            <resource>META-INF/spring.schemas</resource>
                                        </transformer>
                                        <transformer
                                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                            <resource>META-INF/spring.provides</resource>
                                        </transformer>
                                        <transformer
                                                implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                                            <resource>META-INF/spring.factories</resource>
                                        </transformer>
                                        <transformer
                                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                            <mainClass>${start-class}</mainClass>
                                        </transformer>
                                    </transformers>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>I

【问题讨论】：

标签： java docker apache-spark spring-boot yaml

【解决方案1】：

我找到了解决方案。 Origin spark包有snakeyml 1.15的文件夹jar，我把它改成了1.19，现在一切正常。

【讨论】：

就我而言，版本不同。我检查了我的 uber jar 上的版本，并在 jars 文件夹中更新了相同的版本。
我也遇到了类似的问题，你是怎么把版本改成1.19的，能不能分享一下修改后的pom文件。
要找到选择哪个版本，查看 maven 依赖树。使用命令mvn dependency:tree -Dverbose 这个命令帮助我解决了类似的 NoSuchMethodError，尽管它是针对不同的 jar 的。