使用换行符和附加逗号解析 csv 文件

Question

我当前的 csv 文件如下所示：

field1, field1, field3, field4, field5, field6  
111, John, Doctor, 1A-jrd, ,"Tuft St  
Peoria, IL 54345  
(12.11111, 43.5555)"  
121, Bob, Teacher, 2A-abcd, 345, "Moore Ave  
Boston, MA 23123  
(67.11111,- 49.5567)"  
131, Kyle, Engineer, 3A-bhbh, , "Barnes St  
San Francisco, CA 34654  
(65.11111, 55.985432)"  

在某些情况下 field5 没有值。此外，field6 在引号内并且有换行符。例如：第一行数据的field6实际上是

"Tuft St  
Peoria, IL 54345  
(12.11111, 43.5575)"  

我需要编写一个脚本来解析这个文件并返回 12.111, 43.557 来代替 field6 的当前值，这样最终的 csv 文件就会看起来像

field1, field1, field3, field4, field5, field6  
111, John, Doctor, 1A-jrd, , "12.111, 43.555"  
121, Bob, Teacher, 2A-abcd, 345, "67.111,- 49.556"  
131, Kyle, Engineer, 3A-bhbh, , "65.111, 55.985"  

我看过 cvsparser，但我的理解是，它只有在整个数据行在一行上且没有任何换行符的情况下才有效。此外，我不能简单地使用逗号来分割行，因为其中有多个逗号的地址。关于如何解析这个 csv 文件的任何建议？

Answer 1

对于这种“非结构化 csv”格式，您可以使用 Marpa::R2，这是一个与通用 BNF 解析器 Marpa 的 Perl 接口。

数据可以在BNF 中描述为this ::= that（~ 运算符定义词法规则）。 ::= 规则中的括号，例如(header [\n]) 意思是“不包含在解析结果中”。

解析器返回一个数据结构（[ id, child1, child2 ... ] 格式的数组数组），可以从中提取数据。

您还可以将semantic actions 定义为Perl sub 在同一个或单独的包中处理数据。

下面是基于您的数据的示例脚本及其输出。

脚本：

use 5.010;
use strict;
use warnings;

use Data::Dumper;
$Data::Dumper::Indent = 1;
$Data::Dumper::Terse = 1;
$Data::Dumper::Deepcopy = 1;

use Marpa::R2;

my $g = Marpa::R2::Scanless::G->new( { source => \(<<'END_OF_SOURCE'),

    :default ::= action => [ name, value]
    lexeme default = action => [ name, value] latm => 1

    csv ::= (header [\n]) lines

    header ::= column+ separator => column_sep
    column_sep ~ ', '
    column ~ 'field' [1-6]

    lines       ::= line+ separator => [\n]
    line        ::= fields1_5 (',') field6
    field_sep   ~ ','
    fields1_5   ::= field1_5+ separator => field_sep
    field1_5    ~ num | word | code
    field6      ~ address

    num ~ [\d]+
    word ~ [A-Za-z]+
    code  ~ num word '-' word
    address ~ '"' address_chars '"'
    address_chars ~ [^\"]+ #"

    :discard ~ space
    space ~ ' '

END_OF_SOURCE
} );

my $input = <<EOI;
field1, field1, field3, field4, field5, field6
111, John, Doctor, 1A-jrd, ,"Tuft St
Peoria, IL 54345
(12.11111, 43.5555)"
121, Bob, Teacher, 2A-abcd, 345, "Moore Ave
Boston, MA 23123
(67.11111,- 49.5567)"
131, Kyle, Engineer, 3A-bhbh, , "Barnes St
San Francisco, CA 34654
(65.11111, 55.985432)"
EOI

say Dumper ${ $g->parse( \$input, { trace_terminals => 0 } ) };

输出：

[
  'csv',
  [
    'lines',
    [
      'line',
      [
        'fields1_5',
        [
          'field1_5',
          '111'
        ],
        [
          'field1_5',
          'John'
        ],
        [
          'field1_5',
          'Doctor'
        ],
        [
          'field1_5',
          '1A-jrd'
        ]
      ],
      [
        'field6',
        '"Tuft St
Peoria, IL 54345
(12.11111, 43.5555)"'
      ]
    ],
    [
      'line',
      [
        'fields1_5',
        [
          'field1_5',
          '121'
        ],
        [
          'field1_5',
          'Bob'
        ],
        [
          'field1_5',
          'Teacher'
        ],
        [
          'field1_5',
          '2A-abcd'
        ],
        [
          'field1_5',
          '345'
        ]
      ],
      [
        'field6',
        '"Moore Ave
Boston, MA 23123
(67.11111,- 49.5567)"'
      ]
    ],
    [
      'line',
      [
        'fields1_5',
        [
          'field1_5',
          '131'
        ],
        [
          'field1_5',
          'Kyle'
        ],
        [
          'field1_5',
          'Engineer'
        ],
        [
          'field1_5',
          '3A-bhbh'
        ]
      ],
      [
        'field6',
        '"Barnes St
San Francisco, CA 34654
(65.11111, 55.985432)"'
      ]
    ]
  ]
]

Answer 2

你不能。由于您允许在字段 6 中使用逗号，因此这是一个有效文件

A,B,C,D,,E a,b,c,d,e,f

而且您无法确定该文件是否包含一个或两个条目，因为第一个数据集的字段 6 可以是 'E' 或 'E \na,b,c,d,e,f'

Answer 3

您可以为此使用csv 库

import csv

with open('myfile.csv') as myfile:
     csv_file = csv.reader(myfile, delimiter = ',')

现在你有了行，随心所欲。

Answer 4

您需要一个 CSV 解析器来处理这类数据。我建议 perl 和 Text::CSV:

类似这样的：

#!/usr/bin/env perl
use strict;
use warnings;

use Text::CSV; 

my $csv = Text::CSV -> new( { 'binary' => 1, eol => "\n" } ); 

open ( my $input_fh, '<', "sample.csv" ) or die $!; 

my $header = $csv -> getline ( $input_fh );
$csv -> print ( \*STDOUT, $header );

while ( my $row = $csv -> getline ( $input_fh ) ) { 
    $row -> [5] =~ s,.*\(,\(,ms;
    $csv -> print ( \*STDOUT, $row );
}

给定源数据：

field1, field1, field3, field4, field5, field6  
111, John, Doctor, 1A-jrd, ,"Tuft St  
Peoria, IL 54345  
(12.11111, 43.5555)"
121, Bob, Teacher, 2A-abcd,345,"Moore Ave  
Boston, MA 23123  
(67.11111,- 49.5567)"
131, Kyle, Engineer, 3A-bhbh, ,"Barnes St  
San Francisco, CA 34654  
(65.11111, 55.985432)"

输出：

field1," field1"," field3"," field4"," field5"," field6  "
111," John"," Doctor"," 1A-jrd"," ","(12.11111, 43.5555)"
121," Bob"," Teacher"," 2A-abcd",345,"(67.11111,- 49.5567)"
131," Kyle"," Engineer"," 3A-bhbh"," ","(65.11111, 55.985432)"

希望您可以清楚地了解如何进一步修改“field6”以精确满足您的规范。