在 pig 中创建自定义代理键

Question

有没有办法在 Pig 中创建自定义代理键？

例如：我们有如下数据

Salary City Name

20000 newyork john   
30000 sydney joseph   
60000 delhi mike   
30000 sydney joseph

对于这些数据，我们需要创建如下的代理键，结果应该如下。

     Salary City Name

SCN1 20000 newyork john    
SCN2 30000 sydney joseph   
SCN3 60000 delhi mike  
SCN2 30000 sydney joseph

而不是创建随机唯一键？

提前致谢！！。

Answer 1

首先区分数据，使用RANK和CONCAT获取每个不同行的自定义键。然后与原始数据集连接不同。最后生成所需的列。

A = LOAD 'data.txt' USING PigStorage('\t');
B = DISTINCT A;
C = RANK B;
D = FOREACH C GENERATE CONCAT('SCN',$0),$1,$2,$3;
E = JOIN A BY ($0,$1,$2),D BY ($1,$2,$3);
F = FOREACH E GENERATE E::$3,E::$0,E::$1,E::$2;
DUMP F;

这就是它如何处理样本数据

一个

20000 newyork john   
30000 sydney joseph   
60000 delhi mike   
30000 sydney joseph

B

20000 newyork john   
30000 sydney joseph   
60000 delhi mike

C

1 20000 newyork john   
2 30000 sydney joseph   
3 60000 delhi mike

D

SCN1 20000 newyork john   
SCN2 30000 sydney joseph   
SCN3 60000 delhi mike

E

20000 newyork john SCN1 20000 newyork john     
30000 sydney joseph SCN2 30000 sydney joseph   
60000 delhi mike SCN3 60000 delhi mike 
30000 sydney joseph SCN2 30000 sydney joseph 

F

SCN1 20000 newyork john    
SCN2 30000 sydney joseph   
SCN3 60000 delhi mike  
SCN2 30000 sydney joseph

Answer 2

感谢 Inquistive Mind 帮助我生成唯一代理键。这是我测试并运行良好的 pig 脚本。

 A = LOAD '/user/root5/data3.txt' USING PigStorage(',');
 B = DISTINCT A;
 C = RANK B;
 D = FOREACH C GENERATE CONCAT('SCN',$0),$1,$2,$3;
 E = JOIN A BY ($0,$1,$2),D BY ($1,$2,$3);
 F = FOREACH E GENERATE $3, $0, $1, $2;
 DUMP F;

每一步的输出如下：

DUMP A;
(20000,newyork,john)
(30000,sydney,joseph)
(60000,delhi,mike)
(20000,newyork,john)
(30000,sydney,mike)
(60000,delhi,mike)  

DUMP B;
(20000,newyork,john)
(30000,sydney,mike)
(30000,sydney,joseph)
(60000,delhi,mike)

DUMP C;
(1,20000,newyork,john)
(2,30000,sydney,mike)
(3,30000,sydney,joseph)
(4,60000,delhi,mike)

DUMP D;
(SCN1,20000,newyork,john)
(SCN2,30000,sydney,mike)
(SCN3,30000,sydney,joseph)
(SCN4,60000,delhi,mike)

DUMP E;
(20000,newyork,john,SCN1,20000,newyork,john)
(20000,newyork,john,SCN1,20000,newyork,john)
(30000,sydney,mike,SCN2,30000,sydney,mike)
(30000,sydney,joseph,SCN3,30000,sydney,joseph)
(60000,delhi,mike,SCN4,60000,delhi,mike)
(60000,delhi,mike,SCN4,60000,delhi,mike)

DUMP F;
(SCN1,20000,newyork,john)
(SCN1,20000,newyork,john)
(SCN2,30000,sydney,mike)
(SCN3,30000,sydney,joseph)
(SCN4,60000,delhi,mike)
(SCN4,60000,delhi,mike)'