使用 SED 或 AWK 删除特定 CSV 列中的所有引号

Question

我有一个包含一堆 CSV 行的文件，其中包含带和不带引号的值，如下所示：

"123","456",,17,"hello," how are you this, fine, highly caffienated morning,","2018-05-29T18:58:10-05:00","XYZ", 
"345","737",,16,"Heading to a "meeting", unprepared while trying to be "awake","2018-05-29T18:58:10-05:00","ACD",

第五列是一个文本列，它有转义或非转义双引号。我正在尝试删除此列中的所有引号，使其看起来像这样

"123","456",,17,"hello, how are you this, fine, highly caffeinated morning,","2018-05-29T18:58:10-05:00","XYZ", 
"345","737",,16,"Heading to a meeting, unprepared while trying to be awake","2018-05-29T18:58:10-05:00","ACD",

任何想法如何使用 SED 或 AWK 或任何其他 unix 工具来实现这一点？非常感激！

Answer 1

使用 awk，您可以执行类似的操作来避免 very complex regex。只有第五列被破坏，前面的列不包含逗号，并且我们知道有固定数量的列，这使得修复很容易：

根据 Ed Morton 的建议使用 gsub 进行编辑以实现可移植性

awk '
    BEGIN{FS=OFS=","}
    {
        for(i=6; i<=NF-3;i++){
            $5 = $5 FS $i
        }
    }
    {
         gsub(/"/, "", "g", $5)
    }
    {print $1,$2,$3,$4,"\""$5"\"",$(NF-2),$(NF-1),$NF}
    ' <file>

输出：

"123","456",,17,"hello, how are you this, fine, highly caffienated morning,","2018-05-29T18:58:10-05:00","XYZ", 
"345","737",,16,"Heading to a meeting, unprepared while trying to be awake","2018-05-29T18:58:10-05:00","ACD",

如果你想转义引号，你可以使用这个：

awk '
    BEGIN{FS=OFS=","}
    {
        for(i=6; i<=NF-3;i++){
            $5 = $5 FS $i
        }
    }
    {
         gsub(/^"|"$/,"",$5);
         gsub(/"/,"\\\"",$5);
         $5="\""$5"\"";
    }
    {print $1,$2,$3,$4,$5,$(NF-2),$(NF-1),$NF}
    ' <file>

输出：

"123","456",,17,"hello,\" how are you this, fine, highly caffienated morning,","2018-05-29T18:58:10-05:00","XYZ", 
"345","737",,16,"Heading to a \"meeting\", unprepared while trying to be \"awake","2018-05-29T18:58:10-05:00","ACD",

Answer 2

您的问题很难以通用的方式回答。举个例子：

 "a","b","c","d" 

这是如何解释的（如果我们从感兴趣的字段中删除引号）：

"a","b","c","d"  (4 fields)
"a,b","c","d"    (3 fields, $1 messed up)
"a","b,c","d"    (3 fields, $2 messed up)
"a","b","c,d"    (3 fields, $3 messed up)
"a,b,c","d"      (2 fields, $1 messed up)
"a,b","c,d"      (2 fields, $1 and $2 messed up)
"a","b,c,d"      (2 fields, $2 messed up)
"a,b,c,d"        (1 field , $1 messed up)

解决这个问题的唯一方法是掌握以下知识：

• 我的 CSV 有多少个字段
• 最多有一个字段混乱
• 我们知道哪个字段搞砸了

以下 awk 程序将帮助您修复它：

$ awk 'BEGIN{ere="[^,]*|\042[^\042]"}
       { head=tail=""; mid=$0 }
       # extract the head which is correct
       (n>1) {
          ere_h="^"
          for(i=1;i<n;++i) ere_h = ere_h (ere_h=="^" ? "",",") "(" ere ")"
          match(mid,ere_h); head=substr(mid,RSTART,RLENGTH)
          mid = substr(mid,RLENGTH+1)
       }
       # extract the tail which is correct
       (nf>n) {
          ere_t="$"
          for(i=n+1;i<=nf;++i) ere_t = "(" ere ")" (ere_h=="$" ? "",",") ere_t
          match(mid,ere_t); tail=substr(mid,RSTART,RLENGTH)
          mid = substr(mid,1,RSTART-1)
       }
       # correct the mid part
       { gsub(/\042/,"",mid)
         mid = (mid ~ /^,/) ? ( ",\042" substr(mid,2) ) : ( "\042" mid )
         mid = (mid ~ /,$/) ? ( substr(mid,1,length(mid)-1) "\042," ) : (mid "\042" )
       }
       # print the stuff
       { print head mid tail }' n=5 nf=7 file

Answer 3

使用 GNU awk 将第三个参数 match() 并假设您知道每行应该有多少个字段：

$ cat tst.awk
BEGIN {
    numFlds  = 8
    badFldNr = 5
}
match($0,"^(([^,]*,){"badFldNr-1"})(.*)((,[^,]*){"numFlds-badFldNr"})",a) {
    gsub(/"/,"",a[3])
    print a[1] "\"" a[3] "\"" a[4]
}

$ awk -f tst.awk file
"123","456",,17,"hello, how are you this, fine, highly caffienated morning,","2018-05-29T18:58:10-05:00","XYZ",
"345","737",,16,"Heading to a meeting, unprepared while trying to be awake","2018-05-29T18:58:10-05:00","ACD",

对于其他 awk，您可以通过调用 match() 和变量而不是数组来执行相同的操作。

Answer 4

试试这个正则表达式：

,\d{2}\,(.*),\"\S{25}\",\"\w{3}"

它是根据您的示例制作的。目标只是捕获第五列。就像@Jerry Jeremiah 建议的那样，重点是使用日期，其长度始终为 25 个字符。为了防止一些不匹配，我还考虑了第五个之前的 2 位数字和日期之后的 3 个字母/数字。 Regex101v1

我们还可以通过查找确切的日期匹配来使用“更强”的正则表达式

,\d{2}\,(.*),\"\d{4}-\d{2}-\d{2}\w\d{2}:\d{2}:\d{2}-\d{2}:\d{2}\",\"\w{3}"

Regex101v2

使用这些正则表达式，您将能够使用组提取第五列。要深入了解您的问题，您可以在 bash 中执行此操作：

regex='^(.*,[0-9]{2}\,")(.*)(",\"[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}-[0-9]{2}:[0-9]{2}\",\"[a-zA-Z]{3}".*$)'
while IFS= read -r line
do
    if [[ $line =~ $regex ]]
    then
        before=${BASH_REMATCH[1]}
        fifth=${BASH_REMATCH[2]}
        after=${BASH_REMATCH[3]}
        reworked_fifth="${fifth//\"}"
        echo ${before}${reworked_fifth}${after}
    else
        echo "Line didnt match the regex"
  fi
done < /my/file/path

我不得不更改正则表达式，因为我的 bash 没有采用 \d 和 \w。无需对此进行 sed 或 awk 任何操作。 Bash 可以单独处理。