【发布时间】：2019-05-03 17:14:20
【问题描述】：
来自 twitter 的推文存储在 hadoop 的 hdfs 中。
需要对推文进行处理以进行情绪分析。 hdfs 中的推文是 avro 格式，因此需要使用 Json 加载程序处理它们但是在猪脚本中，来自 hdfs 的推文没有被读取。更改 jar 文件后，猪脚本显示失败消息
通过pig脚本使用以下这些jar文件失败了。
注册'/home/cloudera/Desktop/elephant-bird-hadoop-compat-4.17.jar';
注册 '/home/cloudera/Desktop/elephant-bird-pig-4.17.jar';
注册 '/home/cloudera/Desktop/json-simple-3.1.0.jar';
这些是另一组 jar 文件，它没有失败，但也没有读取数据。
注册'/home/cloudera/Desktop/elephant-bird-hadoop-compat-4.17.jar';
注册 '/home/cloudera/Desktop/elephant-bird-pig-4.17.jar';
注册 '/home/cloudera/Desktop/json-simple-1.1.jar';
这是我用过的所有猪脚本命令：
tweets = LOAD '/user/cloudera/OutputData/tweets' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS myMap;
B = FOREACH tweets GENERATE myMap#'id' as id ,myMap#'tweets' as tweets;
tokens = foreach B generate id, tweets, FLATTEN(TOKENIZE(tweets)) As word;
dictionary = load ' /user/cloudera/OutputData/AFINN.txt' using PigStorage('\t') AS(word:chararray,rating:int);
word_rating = join tokens by word left outer, dictionary by word using 'replicated';
describe word_rating;
rating = foreach word_rating generate tokens::id as id,tokens::tweets as tweets, dictionary::rating as rate;
word_group = group rating by (id,tweets);
avg_rate = foreach word_group generate group, AVG(rating.rate) as tweet_rating;
positive_tweets = filter avg_rate by tweet_rating>=0;
DUMP positive_tweets;
negative_tweets = filter avg_rate by tweet_rating<=0;
DUMP negative_tweets;
为第一组 jar 文件转储上述 tweets 命令时出错：
输入： 从“/user/cloudera/OutputData/tweets”读取数据失败
输出： 无法在“hdfs://quickstart.cloudera:8020/tmp/temp-1614543351/tmp37889715”中产生结果
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1556902124324_0001
2019-05-03 09:59:09,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2019-05-03 09:59:09,427 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias tweets. Backend error : org.json.simple.parser.ParseException
Details at logfile: /home/cloudera/pig_1556902594207.log
为第二组 jar 文件转储上述 tweets 命令时出错：
输入： 成功从：“/user/cloudera/OutputData/tweets”读取0条记录（5178477字节）
输出： 成功存储0条记录在：“hdfs://quickstart.cloudera:8020/tmp/temp-1614543351/tmp479037703”
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1556902124324_0002
2019-05-03 10:01:05,417 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2019-05-03 10:01:05,418 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2019-05-03 10:01:05,418 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2019-05-03 10:01:05,428 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2019-05-03 10:01:05,428 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
预期的输出被排序为正面和整洁的推文，但出现错误。 请帮忙。谢谢。
【问题讨论】：
标签： hadoop apache-pig sentiment-analysis