用于过滤特定字符串文件的 Pig 脚本/命令

Question

我正在尝试编写 Hadoop Pig 脚本，该脚本将采用 2 个文件并根据字符串进行过滤，即

words.txt

google 
facebook 
twitter 
linkedin

tweets.json

{"created_time": "18:47:31 ", "text": "RT @Joey7Barton: ..give a facebook about whether the americans wins a Ryder cup. I mean surely he has slightly more important matters. #fami ...", "user_id": 450990391, "id": 252479809098223616, "created_date": "Sun Sep 30 2012"}

脚本

twitter  = LOAD 'Twitter.json' USING JsonLoader('created_time:chararray, text:chararray, user_id:chararray, id:chararray, created_date:chararray');
    filtered = FILTER twitter BY (text MATCHES '.*facebook.*');
    extracted = FOREACH filtered GENERATE 'facebook' AS pattern,id, user_id, created_time, created_date, text;
    final = GROUP extracted BY pattern;
    dump final;

输出

(facebook,{(facebook,252545104890449921,291041644,23:06:59 ,Sun Sep 30 2012,RT @Joey7Barton: ..give a facebook about whether the americans wins a Ryder cup. I mean surely he has slightly more important matters. #fami ...)})

我得到的输出是，不加载 words.txt 文件，即直接过滤推文。

我需要得到输出

(facebook)(complete tweet of that facebook word contained)

即它应该读取 words.txt 并且根据它读取的单词应该从 tweets.json 文件中获取所有推文

任何帮助

莫汉.V

Answer 1

您可以考虑在 FOREACH 语句中运行多个语句的方向。像这样的——

final = FOREACH words  {
            a = CONCAT(CONCAT('.*',words.$0),'.*') as aaa;
            filtered = FILTER twitter BY (text MATCHES aaa);
        generate a, flatten(filtered) as output; }

请注意，这只是提供一个想法，我没有测试过。我将在访问 Pig 环境后立即尝试测试，但这应该可以帮助您入门。