1、读取文件是LzoTextInputFormat.class
2、设置
JavaSparkContext jsc = new JavaSparkContext(conf);
Configuration hconf = jsc.hadoopConfiguration();
hconf.set("textinputformat.record.delimiter", "[*&^%]");
hconf.set("io.compression.codecs", "org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzopCodec");
hconf.set("io.compression.codec.lzo.class", "com.hadoop.compression.lzo.LzoCodec");
JavaPairRDD<LongWritable, Text> newAPIHadoopFile = jsc.newAPIHadoopFile(args[4], LzoTextInputFormat.class,LongWritable.class,Text.class, hconf);
由于LzoTextInputFormat默认是按行读取，而我需要按照日志记录起始标识符读取行。
故修改源码：
com.hadoop.mapreduce.LzoTextInputFormat
com.hadoop.mapreduce.LzoLineRecordReader
实现功能！