mailgun 报告到 csv 格式 perl

Question

我有一个问题。我想编写一个 perl 脚本来将 Mailgun 输出解析为 csv 格式。我假设'split' 和'join' 函数在这个过程中可以正常工作。以下是一些示例数据：

样本数据

{

    "geolocation": {

    "city": "Random City", 

    "region": "State", 

    "country": "US"
    }, 
    "url": "https://www4.website.com/register/1234567", 

    "timestamp": "1237854980723.0239847"
}


{

    "geolocation": {

    "city": "Random City2", 

    "region": "State2", 

    "country": "mEXICO"
    }, 
    "url": "https://www4.website2.com/register/ABCDE567", 

    "timestamp": "1237854980723.0239847"
}

期望的输出

“城市”、“地区”、“国家”、“网址”、“时间戳”

"随机城市","州","US","https://www4.website.com/register/1234567","1237854980723.0239847"

"Random City_2","State_2","mEXICO","www4.website2.com/ABCDE567","1237854980723.0239847_2"

我的目标是获取我的示例数据并将所需的输出创建为逗号分隔的 CSV 文件。我不确定该怎么做。通常我会尝试在批处理文件中使用一系列单行代码来解决这个问题，但我更喜欢 perl 脚本。真实数据将包含更多信息。但是，只要弄清楚如何解析一般结构就可以了。

这是我在批处理文件中的内容。

代码

    perl -p -i.bak -e "s/(,$|,+ +$|^.*?{$|^.*?}.*?$|^.*?],.*?$)//gi" file.txt

    rem Removes all unnecessary characters and lines with { and }. ^

    perl -p -i.bak -e "s/(^ +| +$)//gi" file.txt    

    perl -p -i.bak -e "s/^\n$//gi" file.txt


rem Removes all blank lines in initial file. Next one-liner takes care of trailing and beginning 

rem whitespace.  The file is nice and clean now.

perl -p -e "s/(^\".*?\"):.*?$/$1/gi" file.txt > header.txt

rem retains only header info and puts into 'header.txt' ^

perl -p -e "s/^\".*?\": +(\".*?\"$)/$1/gi" file.txt > data.txt

rem retains only data that is associated with each field.

perl -p -i.bak -e "s/\n/,/gi" data.txt

rem replaces new line character with ',' delimiter.

perl -p -i.bak -e "s/^/\n/gi" data.txt

rem drops data down a line

perl -p -i.bak -e "s/\n/,/gi" header.txt

rem replaces new line character with ',' delimiter.

copy header.txt+data.txt report.txt

rem copies both files together.  Since there is the same amount of fields as there are data   

rem delimiters, the columns and headers match.

我的输出

“城市”、“地区”、“国家”、“网址”、“时间戳”

"随机城市","州","US","https://www4.website.com/register/1234567",1237854980723.0239847

这可以解决问题，但精简的脚本会更好。不同的情况会影响这个批处理脚本我需要更可靠的东西。有什么建议吗？？

Answer 1

您可以将单个 Perl 脚本与一个正则表达式一起使用

#!/usr/bin/env perl
use v5.10;
use Data::Dumper;

$_ = <<TXT;
{

    "geolocation": {

    "city": "Random City",

    "region": "State",

    "country": "US"
    },
    "url": "https://www4.website.com/register/1234567",

    "timestamp": "1237854980723.0239847"
}
TXT

my @matches = /\s*\s*("[^"]+")\s*\s*:\s*("[^"]+")/gmx;
my %hash = @matches;

say join(",", keys %hash);
say join(",", values %hash);          

哪个输出这个：

"city","country","region","timestamp","url"
"Random City","US","State","1237854980723.0239847","https://www4.website.com/register/1234567"

当然，如果您想使用 STDIN 代替，您可以将字符串定义替换为：

local $/ = undef;
$_ = <>;

如果你想要一个更健壮的代码，我建议首先匹配包含在大括号中的数据块。然后您将搜索键：值。

我会写这个program.pl 文件：

#!/usr/bin/env perl
use v5.10;
use Data::Dumper;

local $/ = undef;    
open FILE, $ARGV[0] or die $!;
$_ = <FILE>;
close FILE;

# Match all group { ... }
my @groups = /((?&BRACKETED))
(?(DEFINE)
    (?<WORD>     [^\{\}]+ )
    (?<BRACKETED> \s* \{ (?&TEXT)? \s* \} )
    (?<TEXT>      (?: (?&WORD) | (?&BRACKETED) )+ )
)/gmx;

# Match any key:value pairs inside each group
my @results;
for(grep($_,@groups)) {
    push @results, {/\s*\s*"([^"]+)"\s*\s*:\s*("[^"]+")/gmx};
}

# For each result, we print the keys we want
for(@results) {
    say join ",", @$_{qw/city region country url timestamp/};
}

然后一个批处理文件调用脚本：

rem How to call it...
@perl program.pl text.txt > report.txt

Answer 2

不要嘲笑@coin 的 regex-fu，但使用 CPAN 模块的优势包括获得更灵活的解决方案，您可以在此基础上继续构建，并利用其他人已经解决的边缘情况处理。

此解决方案使用 JSON 模块来解析您的传入数据（我假设它仍然看起来像 JSON），并使用 CSV 模块来生成高质量的 CSV，它负责处理您的内嵌引号和逗号等事情数据。

use warnings;
use strict;

use JSON qw/decode_json/;
use Text::CSV_XS;

my $json_data_as_string = <<EOL;
{
    "geolocation": {
        "city": "Random City", 
        "region": "State", 
        "country": "US"
    }, 
    "url": "https://www4.website.com/register/1234567", 
    "timestamp": "1237854980723.0239847"
}
EOL

my $s = decode_json($json_data_as_string);

my $csv = Text::CSV_XS->new({ binary => 1 });

$csv->combine(
    $s->{geolocation}{city},
    $s->{geolocation}{region},
    $s->{geolocation}{country},
    $s->{url},
    $s->{timestamp},
) || die $csv->error_diag;;

print $csv->string, "\n";

要将文件中的数据读入 $json_data_as_string，您可以使用@coin 解决方案中的代码。