尝试使用 CDH-5.2.0 写入 HDFS 时出现奇怪的错误

Question

我正在尝试写入由五个 spark 节点组成的集群，我们使用的是 CDH-5.2.0，刚刚从以前的版本升级，问题就来了。现在我无法将数据写入集群。

这是来自 pom.xml 的依赖项：

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.5.0-cdh5.2.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.5.0-cdh5.2.0</version>
    </dependency>

这是我用来写入 HDFS 的实际代码：

public class MyHadoopUtils {

final static Logger LOGGER = Logger.getLogger(MyHadoopUtils.class);
private static FileSystem fs;
static{
    Configuration myConf = new Configuration();     
    myConf.set("fs.defaultFS", MyWatchService.getDEFAULTFS());      
    myConf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
    myConf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
    try {
        fs = FileSystem.get(myConf);

    } 
    catch (IOException e) {
        e.printStackTrace();
        LOGGER.error("ATTENTION! an error occurred when trying to instantiate a hadoop file system handler!. " + e.getLocalizedMessage());
    } 
    catch (Exception e) {
        e.printStackTrace();
        LOGGER.error("ATTENTION! an error occurred when trying to instantiate a hadoop file system handler!. " + e.getLocalizedMessage());
    } catch (Throwable th) {
        th.printStackTrace();
        LOGGER.error("ATTENTION! an error occurred when trying to instantiate a hadoop file system handler!. " + th.getLocalizedMessage());
    }
}

public static boolean saveWithinHDFS(String path,StringBuffer sb) throws IOException{
    boolean isOk=false;
    Path pt = new Path(path);
    try {
        BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fs.create(pt, true)));
        br.write(sb.toString());
        br.close();
        isOk=true;
    } catch (IOException e) {
        // TODO Auto-generated catch block
        LOGGER.info("ATTENTION! it is been imposible to save a file within HDFS!. path: " + path);
        e.printStackTrace();
        throw e;
    }
    return isOk;
}

这是堆栈跟踪：

[WARN][FeedAdapter][2014-12-15 09:15:45] NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetServerDefaultsRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
    at java.lang.Class.privateGetPublicMethods(Class.java:2690)
    at java.lang.Class.privateGetPublicMethods(Class.java:2700)
    at java.lang.Class.getMethods(Class.java:1467)
    at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:426)
    at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:323)
    at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:672)
    at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:592)
    at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:244)
    at java.lang.reflect.WeakCache.get(WeakCache.java:141)
    at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:455)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:738)
    at org.apache.hadoop.ipc.ProtobufRpcEngine.getProxy(ProtobufRpcEngine.java:92)
    at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:537)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:365)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:262)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:153)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:632)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:570)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:147)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
    at com.tekcomms.c2d.utils.MyHadoopUtils.<clinit>(MyHadoopUtils.java:27)

警告并没有打扰我，因为我知道当机器是 x64 时，它是由某个库的 32 位编译版本引起的，我猜 cloudera 的家伙会为我们提供 64 位版本，但是这个错误真的很奇怪.

请帮忙

更新

mvn 依赖：树看起来像这样：

http://pastebin.com/s8Ti0HG3

对我来说看起来不错，也许这个列表中的一些 jar 是由 cloudera 临时提供的？

Answer 1

它与 64 位或 CDH 无关。该错误表明您在应用程序中包含了两个不同的不兼容版本的 Hadoop。也许您的一个依赖项引入了其他版本，或者您在运行时意外添加了另一个