NoClassDefFoundError 与 gradle、giraph 和 hadoop

Question

所以，我一直在四处寻找，但我的问题没有找到一个好的答案，这让我发疯了，所以我想我会在这里问，希望我能得到帮助。我正在尝试使用 gradle 在 Giraph 项目中进行自动化测试。我是 gradle 的初学者。为了开始，我将 SimpleShortestPathComputation 类的测试代码复制到我的项目中，以确保我可以启动并运行测试。但是，当我执行gradle test 时，出现以下错误：

$ gradle test --info
<skipping some output here...>

Successfully started process 'Gradle Test Executor 1'
Gradle Test Executor 1 started executing tests.
WCCTest > testToyData FAILED
    java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:142)
        at WCCTest.testToyData(WCCTest.java:180)

    Caused by:
    java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        ... 2 more

Gradle Test Executor 1 finished executing tests.
WCCTest > testOnShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at     org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnShorterPathFound(WCCTest.java:64)

WCCTest > testToyDataJson FAILED
    java.lang.NoClassDefFoundError: Could not initialize class org.apache.giraph.conf.GiraphConfiguration
        at WCCTest.testToyDataJson(WCCTest.java:127)

WCCTest > testOnNoShorterPathFound FAILED
    java.lang.NoClassDefFoundError: org.apache.hadoop.conf.Configuration
        at sun.reflect.GeneratedSerializationConstructorAccessor33.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator.newInstance(SunReflectionFactoryInstantiator.java:40)
        at org.objenesis.ObjenesisBase.newInstance(ObjenesisBase.java:59)
        at org.mockito.internal.creation.jmock.ClassImposterizer.createProxy(ClassImposterizer.java:128)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:63)
        at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:56)
        at org.mockito.internal.creation.CglibMockMaker.createMock(CglibMockMaker.java:23)
        at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:26)
        at org.mockito.internal.MockitoCore.mock(MockitoCore.java:51)
        at org.mockito.Mockito.mock(Mockito.java:1243)
        at org.mockito.Mockito.mock(Mockito.java:1120)
        at org.apache.giraph.utils.MockUtils$MockedEnvironment.<init>(MockUtils.java:68)
        at org.apache.giraph.utils.MockUtils.prepareVertexAndComputation(MockUtils.java:132)
        at WCCTest.testOnNoShorterPathFound(WCCTest.java:95)

4 tests completed, 4 failed
<more output...>
:test FAILED
:test (Thread[main,5,main]) completed. Took 1.663 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///...build/reports/tests/index.html

BUILD FAILED

我使用的是完全标准的项目目录结构，这是我的 build.gradle 文件：

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    compile files('$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar')
    compile files('$HADOOP_HOME/hadoop-core-0.20.203.0.jar')

    testCompile group: 'junit', name: 'junit', version: '4.+'
    testCompile group: 'org.mockito', name: 'mockito-all', version: '1.9.5'
}

它编译没有问题，我作为依赖包含的 jar 文件包括它说 NoClassDefFoundError 的类（根据jar tf）。任何想法我做错了什么？提前致谢。

Answer 1

尝试将以下内容添加到您的 build.gradle 中

    println("HADOOP_HOME=$HADOOP_HOME")
    compile files("$HADOOP_HOME/hadoop-core-0.20.203.0.jar")
    println("System.env.HADOOP_HOME=$System.env.HADOOP_HOME")
    compile files("$System.env.HADOOP_HOME/hadoop-core-0.20.203.0.jar")

Answer 2

事实证明，我只需将 $HADOOP_HOME/lib 目录中的一些 jar 添加到依赖项中，它就可以工作了。一旦我添加了 org/apache/commons/logging/LogFactory 的依赖项，它就清除了 hadoop 配置类的错误，然后我只需要稍后添加其他所需的 jar。