替换 csv 文件中的列逗号分隔符并用单引号处理字段值

Question

一个系统正在生成一个我无法影响的 csv 文件。

如果数据本身包含逗号，则有两列的值可以用一对单引号括起来。

示例数据 - 4 列

123,'abc,def,ghf',ajajaj,1 
345,abdf,'abc,def,ghi',2
556,abdf,def,3
999,'a,b,d','d,e,f',4

我想使用 powershell 的结果...

不属于数据的逗号 - 表示分隔字段的逗号被替换为指定的分隔符（在竖线星下面的情况下）。一对单引号之间的逗号保留为逗号。

结果

123|*'abc,def,ghf'|*ajajaj|*1 
345|*abdf|*'abc,def,ghi'|*2
556|*abdf|*def|*3
999|*'a,b,d'|*'d,e,f'|*4

如果可能的话，我想使用 reg 表达式来执行此操作，但我不知道如何执行此操作。

Answer 1

虽然我认为这会创建一个格式奇怪的 CSV 文件，但使用 PowerShell，您可以将 switch 与 -Regex 和 -File 参数一起使用。这可能是处理大文件的最快方式，并且只需要几行代码：

# create a regex that will find comma's unless they are inside single quotes
$commaUnlessQuoted = ",(?=([^']*'[^']*')*[^']*$)"

$result = switch -Regex -File 'D:\test.csv' {
    # added -replace "'" to also remove the single quotes as commented
    default { $_ -replace "$commaUnlessQuoted", '|*' -replace "'" }
}

# output to console
$result

# output to new (sort-of) CSV file
$result | Set-Content -Path 'D:\testoutput.csv'

---

更新

作为mklement0 pointed out，上面的代码完成了这项工作，但代价是在写入输出文件之前完全在内存中将更新的数据创建为数组。
如果这是一个问题（文件太大而无法容纳可用内存），您还可以更改代码以读取/替换原始行并立即将该行写出到输出文件。

下一种方法几乎不会占用任何内存，但当然是以在磁盘上执行更多写入操作为代价的。

# make sure this is an absolute path for .NET
$outputFile = 'D:\output.csv'
$inputFile  = 'D:\input.csv'

# create a regex that will find comma's unless they are inside single quotes
$commaUnlessQuoted = ",(?=([^']*'[^']*')*[^']*$)"

# create a StreamWriter object. Uses UTF8Encoding without BOM (Byte Order Mark) by default.
# if you need a different encoding for the output file, use for instance
# $writer = [System.IO.StreamWriter]::new($outputFile, $false, [System.Text.Encoding]::Unicode)
$writer = [System.IO.StreamWriter]::new($outputFile)
switch -Regex -File $inputFile {
    default {
        # added -replace "'" to also remove the single quotes as commented
        $line = $_ -replace "$commaUnlessQuoted", '|*' -replace "'"
        $writer.WriteLine($line)
        # if you want, uncomment the next line to show on console
        # $line
    }
}

# remove the StreamWriter object from memory when done
$writer.Dispose()

结果：

123|*abc,def,ghf|*ajajaj|*1 
345|*abdf|*abc,def,ghi|*2
556|*abdf|*def|*3
999|*a,b,d|*d,e,f|*4

正则表达式详细信息：

,                 Match the character “,” literally
(?=               Assert that the regex below can be matched, starting at this position (positive lookahead)
   (              Match the regular expression below and capture its match into backreference number 1
      [^']        Match any character that is NOT a “'”
         *        Between zero and unlimited times, as many times as possible, giving back as needed (greedy)
      '           Match the character “'” literally
      [^']        Match any character that is NOT a “'”
         *        Between zero and unlimited times, as many times as possible, giving back as needed (greedy)
      '           Match the character “'” literally
   )*             Between zero and unlimited times, as many times as possible, giving back as needed (greedy)
   [^']           Match any character that is NOT a “'”
      *           Between zero and unlimited times, as many times as possible, giving back as needed (greedy)
   $              Assert position at the end of the string (or before the line break at the end of the string, if any)
)

Answer 2

Theo's helpful answer简洁高效。

让我补充以下解决方案：

• 显示如何将每个 CSV 行解析为字段值数组，基于识别嵌入的 '...' 引用（它可以很容易地适应 "..." 引用），没有包括' 字符。在输出中（如果使用 | 等分隔符，则不再需要在语法上。

• 展示了一种更快的写入输出文件的方法，使用System.IO.File.WriteAllLines

# In and output file paths.
# IMPORTANT: To use file paths with .NET methods, as below, always use
#            FULL PATHS, because .NET's current directory differs from PowerShell's
$inPath = "$PWD/input.csv"
$outPath = "$PWD/output.csv"

[IO.File]::WriteAllLines(
  $outPath,
  # CAVEAT: Even though ReadLines() enumerates *lazily* itself,
  #         applying PowerShell's .ForEach() method to it causes the lines
  #         to all be collected in memory  first.
  [IO.File]::ReadLines($inPath).ForEach({
    # Parse the row into field values, whether they're single-quoted or not.
    $fieldValues = $_ -split "(?x) ,? ( '[^']*' | [^,]* ) ,?" -ne '' -replace "'"
    # Join the field values - without single quotes - to form a row with the
    # new delimiter.
    $fieldValues -join '|'
  })
)

<sup>* 为简洁起见，我省略了一个重要的优化：if (-not $_.Contains("'")) { $_.Replace(",", "|") } 可用于处理不包含 ' 字符的行。更快。
* -split，基于正则表达式的string splitting operator 用于将行拆分为字段。
* 内联选项(?x) 用于使正则表达式更具可读性，如this answer 中所述。</sup>

正如代码 cmets 所述，上述解决方案仍然将整个文件加载到内存中。

需要使用 管道 来避免这种情况，这会大大降低解决方案速度，但是：

& {
 foreach ($line in [IO.File]::ReadLines($inPath)) {
    $fieldValues = $line -split "(?x) ,? ( '[^']*' | [^,]* ) ,?" -ne '' -replace "'"
    $fieldValues -join '|'
  }
} | Set-Content -Encoding Utf8 $outPath

无论哪种解决方案，输出文件最终都会包含以下内容（注意缺少' 字符。）：

123|abc,def,ghf|ajajaj|1
345|abdf|abc,def,ghi|2
556|abdf|def|3
999|a,b,d|d,e,f|4