【发布时间】：2015-06-17 08:59:40
【问题描述】：
当我手动运行 mapred 作业时，它会生成一个有效的 avro 文件，文件名为 . avro 扩展。但是当我在 oozie 工作流程中编写它时，它会生成一个文本文件，这是一个损坏的 avro 文件。这是我的工作流程：
<workflow-app name='sample-wf' xmlns="uri:oozie:workflow:0.2">
<start to='start_here'/>
<action name='start_here'>
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="${nameNode}/user/hadoop/${workFlowRoot}/final-output-data"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.reducer.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.mapper.new-api</name>
<value>true</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/user/hadoop/${workFlowRoot}/input-data</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>/user/hadoop/${workFlowRoot}/final-output-data</value>
</property>
<property>
<name>mapreduce.mapper.class</name>
<value>org.apache.avro.mapred.HadoopMapper</value>
</property>
<property>
<name>mapreduce.reducer.class</name>
<value>org.apache.avro.mapred.HadoopReducer</value>
</property>
<property>
<name>avro.mapper</name>
<value>com.flipkart.flap.data.batch.mapred.TestAvro$CFDetectionMapper</value>
</property>
<property>
<name>avro.reducer</name>
<value>com.flipkart.flap.data.batch.mapred.TestAvro$CFDetectionReducer</value>
</property>
<property>
<name>mapreduce.input.format.class</name>
<value>org.apache.avro.mapreduce.AvroKeyInputFormat</value>
</property>
<property>
<name>avro.schema.input.key</name>
<value>{... schema ...}</value>
</property>
<property>
<name>mapreduce.mapoutput.key.class</name>
<value>org.apache.hadoop.io.AvroKey</value>
</property>
<property>
<name>avro.map.output.schema.key</name>
<value>{... schema ...}</value>
</property>
<property>
<name>mapreduce.mapoutput.value.class</name>
<value>org.apache.hadoop.io.Text</value>
</property>
<property>
<name>mapreduce.output.format.class</name>
<value>org.apache.avro.mapred.AvroKeyValueOutputFormat</value>
</property>
<property>
<name>mapreduce.output.key.class</name>
<value>org.apache.avro.mapred.AvroKey</value>
</property>
<property>
<name>mapreduce.output.value.class</name>
<value>org.apache.avro.mapred.AvroValue</value>
</property>
<property>
<name>avro.schema.output.key</name>
<value>{ .... schema .... }</value>
</property>
<property>
<name>avro.schema.output.value</name>
<value>"string"</value>
</property>
<property>
<name>mapreduce.output.key.comparator.class</name>
<value>org.apache.avro.mapred.AvroKeyComparator</value>
</property>
<property>
<name>io.serializations</name>
<value>org.apache.hadoop.io.serializer.WritableSerialization,org.apache.avro.mapred.AvroSerialization
</value>
</property>
</configuration>
</map-reduce>
<ok to='end'/>
<error to='fail'/>
</action>
<kill name='fail'>
<message>MapReduce failed, error message[$sf:errorMessage(sf:lastErrorNode())}]</message>
</kill>
<end name='end'/>
</workflow-app>我的 mapper 和 reducer 是这样定义的：
public static class CFDetectionMapper extends
Mapper<AvroKey<AdClickFraudSignalsEntity>, NullWritable, AvroKey<AdClickFraudSignalsEntity>, Text> {
}
public static class CFDetectionReducer extends
Reducer<AvroKey<AdClickFraudSignalsEntity>, Text, AvroKey<AdClickFraudSignalsEntity>, AvroValue<CharSequence>>
你能告诉我这里有什么问题吗？
【问题讨论】：