使用 Spark Submit 执行 Spring Boot 应用程序时出现 ClassNotFoundException

Question

我正在尝试使用 spark-submit 执行我的 Spring Boot 示例应用程序，但它正在抛出 ClassNotFoundException。

java.lang.NoClassDefFoundError: org/springframework/boot/SpringApplication

Caused by: java.lang.ClassNotFoundException: org.springframework.boot.SpringApplication 

我还在 pom.xml 中添加了 Spring 依赖项

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>
<dependencies>
    <dependency> <!-- Spark -->
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.6.1</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
    </dependency>
</dependencies>

我的示例代码是：

@SpringBootApplication
public class dummy {    
public static void main (String[] args){
SpringApplication.run(dummy.class, args);
    SparkConf conf = new SparkConf().setAppName("sparkSample").set("spark.executor.heartbeatInterval", "700000")
            .set("spark.network.timeout", "1000000").set("spark.files.fetchTimeout", "800000");

    JavaSparkContext sc = new JavaSparkContext(conf); 
    System.out.println("normal code...........................");
    sc.stop();
}`

我的 spark 提交命令是：

./spark-submit  \
  --class sample.dummy \
  --master yarn-cluster \
  --driver-memory 5G \
  --deploy-mode cluster \
  --executor-memory 7G \
  --executor-cores 1 \
  --num-executors 1 \
  --conf spark.yarn.executor.memoryOverhead=1024  \
  /hadoop/app/abc/bootup/sample-0.0.1-SNAPSHOT.jar \
  --runner=SparkRunner 

可能是什么问题？

Answer 1

我通过使用 SHADE maven 插件解决了这个问题。

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <dependencies>
                    <dependency>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <version>1.5.2.RELEASE</version>
                    </dependency>
                </dependencies>
                <configuration>
                    <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <transformers>
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.handlers</resource>
                        </transformer>
                        <transformer
                            implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                            <resource>META-INF/spring.factories</resource>
                        </transformer>
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.schemas</resource>
                        </transformer>
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>com.yourpackage.MainClass</mainClass>
                        </transformer>
                    </transformers>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>