Spark 通过 oozie 工作流程提交抛出 NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(object,object,object)

Question

我有一个 oozie 工作流程，我在其中调用我的 spark 操作，同一个文件夹中有一个额外的 jar，可以说 xyz.jar，我的 spark jar 名称是 abc.jar。每当我从该文件夹中删除 xyz.jar 时，我的 abc.jar 都会正确提交，但是一旦我将该 jar 添加到同一文件夹中，它就会通过给出以下异常而失败。 我无法删除 xyz.jar，因为进一步的操作取决于 xyz。 在 xyz.jar 我们有 guava-27.1-jre.jar。 如果我要删除 xyz.jar，则不需要 Spark-OPTS。这是我强制使用 guava 16.0.1.jar 所做的额外操作

行动：

<action name="data_load">
            <spark xmlns="uri:oozie:spark-action:0.1">
                    <job-tracker>${jobTracker}</job-tracker>
                    <name-node>${nameNode}</name-node>
                    <master>yarn</master>
                    <mode>cluster</mode>
                    <name>Movement</name>
                    <class>com.package.Job</class>
                    <jar>path/lib/abc.jar
                    </jar>
                    <spark-opts>--jars path/lib/guava-16.0.1.jar --driver-class-path path/lib/guava-16.0.1.jar --conf spark.executor.extraClassPath=path/lib/guava-16.0.1.jar --conf spark.executor.userClassPathFirst=true --conf spark.driver.userClassPathFirst=true</spark-opts>       
            </spark>
            <ok to="notifyJobSuccess" />
            <error to="notifyJobFailure" />

错误：

2020-06-18 16:06:13,481 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
    java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
            at com.google.inject.TypeLiteral.getParameterTypes(TypeLiteral.java:269)
            at com.google.inject.spi.InjectionPoint.forMember(InjectionPoint.java:113)
            at com.google.inject.spi.InjectionPoint.<init>(InjectionPoint.java:72)
            at com.google.inject.spi.InjectionPoint.forMethod(InjectionPoint.java:316)
            at com.google.inject.internal.ProviderMethodsModule.createProviderMethod(ProviderMethodsModule.java:293)
            at com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:135)
            at com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:105)
            at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:347)
            at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:356)
            at com.google.inject.AbstractModule.install(AbstractModule.java:103)
            at com.google.inject.servlet.ServletModule.configure(ServletModule.java:50)
            at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
            at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:347)
            at com.google.inject.spi.Elements.getElements(Elements.java:104)
            at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:137)
            at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:105)
            at com.google.inject.Guice.createInjector(Guice.java:87)
            at com.google.inject.Guice.createInjector(Guice.java:69)
            at com.google.inject.Guice.createInjector(Guice.java:59)
            at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:280)
            at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:303)
            at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:144)
            at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
            at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1148)
            at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
            at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1553)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
            at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1549)
            at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1482)
    2020-06-18 16:06:13,484 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

Answer 1

是的，当 Oozie 和 Spark 等多个大型项目在运行时，类路径管理很困难。你试过shading your use of guava吗？