使用 awk 分析 csv 文件答案

【问题标题】：Analyze csv file with awk使用 awk 分析 csv 文件
【发布时间】：2019-04-02 14:52:24
【问题描述】：

我需要从另一个 csv 文件中分析一个 csv 文件，我们可以在其中找到一些“关键词”

我需要分析的 csv 文件如下所示，4 列 X 行：

NO1,NF_B000014,81920,23290,28,20480,22211,108,-16,0,100000000,none,online,fds,45501,none,Alo1,
NO1,N_000000,81920,63,0,20480,68,0,0,0,2464966,none,online,fds,131,none,Alo1,
NO1,NO_VM31_GRERW0I2_B000002,203162,87,0,50790,142,0,0,0,100000000,none,online,fds,229,none,Alo1,
NO1,NF_B000014,81920,23290,28,20480,22211,108,-16,0,100000000,none,online,fds,45501,none,Alo1,
NO1,NG_VM31_B000001,2347,54,0,432,69,0,0,0,4397642,none,online,fds,98,none,Alo2,
NO1,NG_VM31_B000001,2342,61,0,64532,69,0,0,0,2346,none,online,fds,90,none,Alo2,
NO1,NG_VM31_B000001,78692,61,0,432,69,0,0,0,23498765,none,online,fds,23,none,Alo2,
NO1,NG_VM31_B000001,98725,61,0,2080357,69,0,0,0,98643,none,online,fds,4330,none,Alo2,
NO1,NG_VM31_B000001,2351,61,0,3424,69,0,0,0,5673,none,online,fds,43,none,Alo2,
NO1,NL_098JD,51551,7,0,234,31,1,0,0,100000001,none,online,fds,99,none,Alo3,
NO1,NL_098JD,5145622,7,0,542,31,1,0,0,100000002,none,online,fds,99,none,Alo3,
NO1,NL_098JD,5123453,7,0,2714,31,1,0,0,100000003,none,online,fds,99,none,Alo3,
NO1,NL_098JD,51454,7,0,8567,31,1,0,0,100000004,none,online,fds,38,none,Alo3,
NO1,NL_098JD,515,7,0,532,31,1,0,0,100000005,none,online,fds,31,none,Alo3,
NO1,NL_098JD,51554,7,0,9347,31,1,0,0,100000006,none,online,fds,3812,none,Alo3,
NO1,NV_IUDS,19873,234,0,543,14,3,0,0,9869324,none,online,fds,54,none,Alo4,
NO1,NV_IUDS,32981,654,0,543,14,3,0,0,2346,none,online,fds,57,none,Alo4,
NO1,NV_IUDS,123554,634,0,543,14,3,0,0,2347642,none,online,fds,86,none,Alo4,
NO1,NV_IUDS,124432,846,0,543,14,3,0,0,1434326,none,online,fds,12,none,Alo4,
NO1,NV_IUDS,234531,402,0,543,14,3,0,0,234645234,none,online,fds,62,none,Alo4,
NO1,NJ_000004,305562,57467,19,76390,102,0,0,0,100000000,none,online,oiu,57569,none,NA,
NO1,NK_O09AE8,421888,221682,53,105472,200,0,0,0,100000000,none,online,oiu,12345,none,NA,
NO1,NK_O09AE8,42188,221682,53,105472,200,0,0,0,100000000,none,online,oiu,221882,none,NA,
NO1,NK_O09AE8,421488,221682,53,105472,200,0,0,0,100000000,none,online,oiu,4325,none,NA,
NO1,NK_O09AE8,421845,221682,53,105472,200,0,0,0,100000000,none,online,oiu,9877634,none,NA,
NO1,NK_O09AE8,421234,221682,53,105472,200,0,0,0,100000000,none,online,oiu,22324882,none,NA,
NO1,NK_O09AE8,421643,221682,53,105472,200,0,0,0,100000000,none,online,oiu,234,none,NA,
NO1,NK_O09AE8,421231,221682,53,105472,200,0,0,0,100000000,none,online,oiu,9834,none,NA,
NO1,NK_O09AE8,421324,221682,53,105472,200,0,0,0,100000000,none,online,oiu,234,none,NA,
NO1,NK_O09AE8,421987,221682,53,105472,200,0,0,0,100000000,none,online,oiu,2345,none,NA,
NO1,NK_O09AE8,42134,221682,53,105472,200,0,0,0,100000000,none,online,oiu,6542,none,NA,
NO1,NF_B000014,81920,23290,28,20480,22211,108,-16,0,100000000,none,online,fds,45501,none,Alo5,
NO1,N_000000,81920,63,0,20480,68,0,0,0,2464966,none,online,fds,131,none,Alo5,
NO1,NO_VM31_GRERW0I2_B000002,203162,87,0,50790,142,0,0,0,100000000,none,online,fds,229,none,Alo5,
NO1,NF_B000014,81920,23290,28,20480,22211,108,-16,0,100000000,none,online,fds,45501,none,Alo5,
NO1,NG_VM31_B000001,2347,54,0,432,69,0,0,0,4397642,none,online,fds,98,none,Alo6,
NO1,NG_VM31_B000001,2342,61,0,64532,69,0,0,0,2346,none,online,fds,90,none,Alo6,
NO1,NG_VM31_B000001,78692,61,0,432,69,0,0,0,23498765,none,online,fds,23,none,Alo6,
NO1,NG_VM31_B000001,98725,61,0,2080357,69,0,0,0,98643,none,online,fds,4330,none,Alo6,
NO1,NG_VM31_B000001,2351,61,0,3424,69,0,0,0,5673,none,online,fds,43,none,Alo6,
NO1,NL_098JD,51551,7,0,234,31,1,0,0,100000001,none,online,fds,99,none,Alo7,
NO1,NL_098JD,5145622,7,0,542,31,1,0,0,100000002,none,online,fds,99,none,Alo7,
NO1,NL_098JD,5123453,7,0,2714,31,1,0,0,100000003,none,online,fds,99,none,Alo7,
NO1,NL_098JD,51454,7,0,8567,31,1,0,0,100000004,none,online,fds,38,none,Alo7,
NO1,NL_098JD,515,7,0,532,31,1,0,0,100000005,none,online,fds,31,none,Alo7,
NO1,NL_098JD,51554,7,0,9347,31,1,0,0,100000006,none,online,fds,3812,none,Alo7,
NO1,NV_IUDS,19873,234,0,543,14,3,0,0,9869324,none,online,fds,54,none,Alo8,
NO1,NV_IUDS,32981,654,0,543,14,3,0,0,2346,none,online,fds,57,none,Alo8,
NO1,NV_IUDS,123554,634,0,543,14,3,0,0,2347642,none,online,fds,86,none,Alo8,
NO1,NV_IUDS,124432,846,0,543,14,3,0,0,1434326,none,online,fds,12,none,Alo8,
NO1,NV_IUDS,234531,402,0,543,14,3,0,0,234645234,none,online,fds,62,none,Alo8,
NO1,NJ_000004,305562,57467,19,76390,102,0,0,0,100000000,none,online,oiu,57569,none,NA,
NO1,NK_O09AE8,421888,221682,53,105472,200,0,0,0,100000000,none,online,oiu,12345,none,NA,
NO1,NK_O09AE8,42188,221682,53,105472,200,0,0,0,100000000,none,online,oiu,221882,none,NA,
NO1,NK_O09AE8,421488,221682,53,105472,200,0,0,0,100000000,none,online,oiu,4325,none,NA,
NO1,NK_O09AE8,421845,221682,53,105472,200,0,0,0,100000000,none,online,oiu,9877634,none,NA,
NO1,NK_O09AE8,421234,221682,53,105472,200,0,0,0,100000000,none,online,oiu,22324882,none,NA,
NO1,NK_O09AE8,421643,221682,53,105472,200,0,0,0,100000000,none,online,oiu,234,none,NA,
NO1,NK_O09AE8,421231,221682,53,105472,200,0,0,0,100000000,none,online,oiu,9834,none,NA,
NO1,NK_O09AE8,421324,221682,53,105472,200,0,0,0,100000000,none,online,oiu,234,none,NA,
NO1,NK_O09AE8,421987,221682,53,105472,200,0,0,0,100000000,none,online,oiu,2345,none,NA,
NO1,NK_O09AE8,42134,221682,53,105472,200,0,0,0,100000000,none,online,oiu,6542,none,NA,

带有我的“关键词”的 csv 文件如下所示：

Alo1
Alo2
Alo3
Alo4

我需要开发一个脚本来提取我的 csv 文件的所有行，我们可以在其中找到包含我的关键字的 csv 文件中存在的单词。

我可以用这个脚本做到这一点：

while read jour
do
        grep -wf "$1" "$2" | awk -F',' '{if(f!=$1)print"\n"; f=$1; print $0;}' | awk -F',' '{print $1","$2","$15","$17}' > test1.csv

done <"$1"

结果是：

NO1,NF_B000014,45501,Alo1
NO1,N_000000,131,Alo1
NO1,NO_VM31_GRERW0I2_B000002,229,Alo1
NO1,NF_B000014,45501,Alo1
NO1,NG_VM31_B000001,98,Alo2
NO1,NG_VM31_B000001,90,Alo2
NO1,NG_VM31_B000001,23,Alo2
NO1,NG_VM31_B000001,4330,Alo2
NO1,NG_VM31_B000001,43,Alo2
NO1,NL_098JD,99,Alo3
NO1,NL_098JD,99,Alo3
NO1,NL_098JD,99,Alo3
NO1,NL_098JD,38,Alo3
NO1,NL_098JD,31,Alo3
NO1,NL_098JD,3812,Alo3
NO1,NV_IUDS,54,Alo4
NO1,NV_IUDS,57,Alo4
NO1,NV_IUDS,86,Alo4
NO1,NV_IUDS,12,Alo4
NO1,NV_IUDS,62,Alo4

你能告诉我如何用 awk 做到这一点吗？使用 NR、FNR 等？

谢谢！

【问题讨论】：

你在哪里说The result is - 你得到的实际输出是你不想要的还是你想要的输出？如果是后者，请解释为什么存在NO1,NO_VM31_GRERW0I2_B000002,229, 以及为什么缺少包含您的关键词的这么多行。如果是前者，则添加后者。另外 - 你想找到的那些词总是在你的最后一个字段中，还是它们可以出现在行中的任何地方？它们能否成为较长字符串的一部分？
@EdMorton 对不起！我给你错误的文件所以是的，有一个错误！是的，我要查找的单词总是在同一个字段中。我修改了我的帖子，这一次，例子很好！谢谢你的帮助！ :)
@Makmy 我已更新我的答案以反映您的编辑

标签： linux bash shell csv awk

【解决方案1】：

这听起来像你想要做的：

$ awk -F', *' 'NR==FNR{words[$1];next} $NF in words' words file
ANA1,#DEFF24,99460, Alo1
ANA1,#DEFF43,15654,Alo1
ANA1,VM30_0009,587290,Alo4
ANA1,#DEFF29,99RS0, Alo2
ANA1,#DEFF43,18NCSO,Alo3
ANA1,VEZK_IOP,587290,Alo4
ANA1,#DEFF98,9846, Alo2
ANA1,#DEFF47,3476,Alo3
ANA1,VM323_LOp9,49862,Alo4

如果没有，请编辑您的问题以阐明您的要求并提供更具代表性的示例输入和预期输出。

【讨论】：

感谢您的回答！我很感激！我修改了我的帖子并添加了很好的例子！

【解决方案2】：

我需要开发一个脚本来提取我的 CSV 文件的所有行，我们可以在其中找到包含我的关键字的 CSV 文件中存在的单词。

所以对单词的出现位置没有任何限制：

$ grep -Fwf keywords.csv file.csv

编辑后：

脚本中的以下行：

grep -wf "$1" "$2" | awk -F',' '{if(f!=$1)print"\n"; f=$1; print $0;}' | awk -F',' '{print $1","$2","$15","$17}' > test1.csv

可以替换为：

awk '### processing first file starts here
     # read the first file and store list in array a
     (NR==FNR){a[$1];next}
     ### processing second file starts here
     # initialize variables
     # - Set FS and OFS for file 2
     # - reprocess $0 with new FS ($0=$0)
     (FNR==1) { FS=OFS=","; $0=$0 }
     # if 17th field is in a, process
     # - initialize f if this is the first match
     # - print required fields and 
     # - prepend with ORS (\n) or empty string if $1 changed
     ($17 in a) { if (c++==0) f=$1
                  print (f != $1 ? ORS : "") $1,$2,$15,$17
                  f=$1
     }' "$1" "$2" > test1.csv

FS、OFS、NR、FNR 和ORS 的详细信息可以在here 找到。

【讨论】：

该死！效果很好！谢谢！最后一个问题：如果可能的话，你能帮我详细介绍一下你的 awk，一步一步吗？