Hadoop - Avro：java.lang.Exception：java.lang.NoClassDefFoundError：org/apache/avro/hadoop/io/AvroKeyComparator

Question

我正在开发一个 Map Reduce 程序来读取固定长度的记录并写入 avro 文件。这是草稿形式。执行时出现以下错误：

java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/avro/hadoop/io/AvroKeyComparator
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:406)
Caused by: java.lang.NoClassDefFoundError: org/apache/avro/hadoop/io/AvroKeyComparator
    at org.apache.avro.mapreduce.AvroJob.setMapOutputKeySchema(AvroJob.java:93)
    at org.apache.avro.mapreduce.AvroMultipleOutputs.setSchema(AvroMultipleOutputs.java:511)
    at org.apache.avro.mapreduce.AvroMultipleOutputs.getContext(AvroMultipleOutputs.java:547)
    at org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:399)
    at org.apache.avro.mapreduce.AvroMultipleOutputs.write(AvroMultipleOutputs.java:378)
    at com.visa.edp.common.vssParser.CustomTestMapper.map(CustomTestMapper.java:98)
    at com.visa.edp.common.vssParser.CustomTestMapper.map(CustomTestMapper.java:32)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.hadoop.io.AvroKeyComparator
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 16 more
17/11/10 18:13:28 INFO mapred.JobClient:  map 0% reduce 0%

我的项目中有以下依赖项：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.6.0-mr1-cdh5.8.3</version>
    <scope>provided</scope>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.6.0-mr1-cdh5.8.3</version>
</dependency>

<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro</artifactId>
    <version>1.7.6-cdh5.8.3</version>
</dependency>

<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-mapred</artifactId>
    <version>1.7.6-cdh5.8.3</version>
</dependency>

并且在项目依赖下找不到org.apache.avro.hadoop包。我是否缺少任何依赖项？

谢谢。

Answer 1

将依赖项添加到模块并正常工作。