从 PIG 中的 SUM 中获取 MAX答案

【问题标题】：get MAX from SUM in PIG从 PIG 中的 SUM 中获取 MAX
【发布时间】：2015-09-30 06:46:02
【问题描述】：

    player = LOAD 'ass2_player' USING org.apache.hive.hcatalog.pig.HCatLoader();
player = FOREACH player GENERATE
    (chararray)$3 AS tmID,
    (int)$1 AS year,
    (int)$8 AS points;
group_data = GROUP player BY (year, tmID);
sum_data = FOREACH group_data GENERATE group, SUM(player.points) AS tot_points;
max_data = FOREACH sum_data GENERATE FLATTEN(group), MAX(sum_data.tot_points);
DUMP max_data;

我只想选择每年得分最高的团队的 tmID。

如何获取整行或部分字段或具有最大值的行。就像，按年分组后，组只包含“年”，元组将采用“tmID，tot_points”。我怎么会这样： (year, tmID, tot_points) 每一年。

【问题讨论】：

您可以为您的用例添加示例输入和预期输出吗？

标签： hadoop apache-pig bigdata

【解决方案1】：

你快到了。这是sum_data 的架构：

((year, tmID), tot_points)

从这里开始，你需要group on year 和max on tot_points。如果您只在sum_data 步骤中进行flatten 分组会更容易，例如：

sum_data = FOREACH group_data GENERATE flatten(group) as (year, tmID), SUM(player.points) AS tot_points;

sum_data_grouped = GROUP sum_data BY year;
max_data = FOREACH sum_data_grouped GENERATE group AS year, MAX(sum_data.tot_points) AS max_points, sum_data.tmpID AS tmID;

您的最终脚本应如下所示：

player = LOAD 'ass2_player' USING org.apache.hive.hcatalog.pig.HCatLoader();
player = FOREACH player GENERATE (chararray)$3 AS tmID, (int)$1 AS year, (int)$8 AS points;
group_data = GROUP player BY (year, tmID);
sum_data = FOREACH group_data GENERATE flatten(group) AS (year, tmID), SUM(player.points) AS tot_points;
sum_data_grouped = GROUP sum_data BY year;
max_data = FOREACH sum_data_grouped GENERATE group AS year, MAX(sum_data.tot_points) AS max_points, sum_data.tmpID AS tmID;

PS：我是在手机上写的，没有测试。告诉我你得到了什么。

【讨论】：