无法在 perl6 中编写语法来解析带有特殊字符的行

Question

我的代码在：https://gist.github.com/ravbell/d94b37f1a346a1f73b5a827d9eaf7c92

use v6;
#use Grammar::Tracer;


grammar invoice {

    token ws { \h*};
    token super-word {\S+};
    token super-phrase { <super-word> [\h  <super-word>]*}
    token line {^^ \h* [ <super-word> \h+]* <super-word>* \n};

    token invoice-prelude-start {^^'Invoice Summary'\n}
    token invoice-prelude-end {<line> <?before 'Start Invoice Details'\n>};

    rule invoice-prelude {
        <invoice-prelude-start>
        <line>*?
        <invoice-prelude-end>
        <line>
    }
}

multi sub MAIN(){ 

    my $t = q :to/EOQ/; 
    Invoice Summary
    asd fasdf
    asdfasdf
    asd 123-fasdf $1234.00
    qwe {rq} [we-r_q] we
    Start Invoice Details 
    EOQ


    say $t;
    say invoice.parse($t,:rule<invoice-prelude>);
}

multi sub MAIN('test'){
    use Test;
    ok invoice.parse('Invoice Summary' ~ "\n", rule => <invoice-prelude-start>);

    ok invoice.parse('asdfa {sf} asd-[fasdf] #werwerw'~"\n", rule => <line>);
    ok invoice.parse('asdfawerwerw'~"\n", rule => <line>);

    ok invoice.subparse('fasdff;kjaf asdf asderwret'~"\n"~'Start Invoice Details'~"\n",rule => <invoice-prelude-end>);
    ok invoice.parse('fasdff;kjaf asdf asderwret'~"\n"~'Start Invoice Details'~"\n",rule => <invoice-prelude-end>);
    done-testing;
}

我无法弄清楚为什么rule <invoice-prelude> 上的解析失败并显示Nil。请注意，即使.subparse 也会失败。

通过运行带有'test' 参数的MAIN 可以看到单个令牌的测试通过（当然<invoice-prelude> 上的.parse 失败，因为它不是完整的字符串）。

应该在rule <invoice-prelude> 中修改什么，以便可以正确解析MAIN() 中的整个字符串$t？

Answer 1

注意$t字符串的最后一行末尾有一个隐藏空格：

my $t = q :to/EOQ/; 
    Invoice Summary
    asd fasdf
    asdfasdf
    asd 123-fasdf $1234.00
    qwe {rq} [we-r_q] we
    Start Invoice Details␣   <-- Space at the end of the line
    EOQ

这会使<invoice-prelude-end> 令牌失败，因为它包含前瞻正则表达式<?before 'Start Invoice Details'\n>。这种前瞻在行尾不包括可能的空格（由于前瞻末尾的显式换行符\n）。因此，<invoice-prelude> 规则也不能匹配。

快速解决方法是删除Start Invoice Details 行末尾的空格。

Answer 2

首先，没有回溯的节俭量词*? 可能每次都匹配空字符串。您可以使用regex 代替rule。

其次，行尾有一个空格，以Start Invoice Details开头。

rule invoice-prelude-end {<line> <?before 'Start Invoice Details' \n>};

regex invoice-prelude {
    <invoice-prelude-start>
    <line>*?
    <invoice-prelude-end>
    <line>
}

如果您想避免回溯，可以使用负前瞻。

token invoice-prelude-end { <line> };

rule invoice-prelude {
    <invoice-prelude-start>
    [<line> <!before 'Start Invoice Details' \n>]*
    <invoice-prelude-end>
    <line>
}

以一些变化为灵感的整个例子：

use v6;
#use Grammar::Tracer;


grammar invoice {
    token ws { <!ww>\h* }
    token super-word { \S+ }
    token line { <super-word>* % <.ws> }

    token invoice-prelude-start   { 'Invoice Summary' }
    rule  invoice-prelude-midline { <line> <!before \n <invoice-details-start> \n> }
    token invoice-prelude-end     { <line> }
    token invoice-details-start   { 'Start Invoice Details' }

    rule invoice-prelude {
        <invoice-prelude-start> \n
        <invoice-prelude-midline> * %% \n
        <invoice-prelude-end> \n
        <invoice-details-start> \n
    }
}

multi sub MAIN(){

    my $t = q :to/EOQ/;
    Invoice Summary
    asd fasdf
    asdfasdf
    asd 123-fasdf $1234.00
    qwe {rq} [we-r_q] we
    Start Invoice Details 
    EOQ


    say $t;
    say invoice.parse($t,:rule<invoice-prelude>);
}

Answer 3

TLDR：问题是带有Start Invoice Details  的测试输入行以您没有处理的水平空格结尾。

两种处理方式（除了改变输入）

# Explicitly:                                                       vvv
token invoice-prelude-end { <line> <?before 'Start Invoice Details' \h* \n>}

# Implicitly:
rule  invoice-prelude-end { <line><?before 'Start Invoice Details' \n>}
# ^ must be a rule                      and there must be a space ^
# (uses the fact that you wrote your own <ws> token)

---

以下是我认为会有所帮助的更多内容

我会在line 和super-phrase 中使用% 的“分隔”功能

token super-phrase { <super-word>+ % \h } # single % doesn't capture trailing separator

token line {
  ^^ \h*
  <super-word>* %% \h+ # double %% can capture optional trailing separator
  \n
}

这些 [几乎] 完全等同于您所写的。 （你写的内容必须在<line> 中匹配<super-word> 两次失败，但这只需要失败一次。）

---

我会在invoice-prelude 中使用环绕功能~

token invoice-prelude {
    # zero or more <line>s surrounded by <invoice-prelude-start> and <invoice-prelude-end>
    <invoice-prelude-start> ~ <invoice-prelude-end> <line>*?

    <line> # I assume this is here for debugging
}

请注意，作为rule 实际上并没有获得任何好处，因为所有水平空白都已由其余代码处理。

---

我不认为发票前奏的最后一行有什么特别之处，所以从invoice-prelude-end 中删除<line>。 （invoice-prelude 中的<line>*? 将捕获它。）

token invoice-prelude-end {<?before 'Start Invoice Details' \h* \n>}

---

唯一可以从rule 中受益的正则表达式是invoice-prelude-start 和invoice-prelude-end。

rule  invoice-prelude-start {^^ Invoice Summary \n}
# `^^` is needed  so the space ^ will match <.ws>

rule  invoice-prelude-end {<?before ^^ Start Invoice Details $$>}

这只有在你可以匹配     Invoice    Summary    ␤之类的东西时才有效。

注意invoice-prelude-start 需要使用\n 来捕获它，但invoice-prelude-end 可以使用$$ 代替，因为它无论如何都不会捕获\n。

---

如果您将super-word 更改为\S+ 以外的其他值，那么您可能还希望将ws 更改为\h+ | <.wb> 之类的值。 （单词边界）

---

#! /usr/bin/env perl6
use v6.d;

grammar invoice {
    token TOP { # testing
         <invoice-prelude>
         <line>
    }

    token ws { \h* | <.wb> };
    token super-word { \S+ };
    token super-phrase { <super-word>+ % \h }
    token line {
        ^^ \h*
        <super-word>* %% \h+
        \n
    };

    rule invoice-prelude-start {^^ Invoice Summary \n}
    rule invoice-prelude-end {<?before ^^ Start Invoice Details $$>};

    token invoice-prelude {
        <invoice-prelude-start> ~ <invoice-prelude-end>
            <line>*?
    }
}

multi sub MAIN(){ 
    my $t = q :to/EOQ/; 
    Invoice Summary
    asd fasdf
    asdfasdf
    asd 123-fasdf $1234.00
    qwe {rq} [we-r_q] we
    Start Invoice Details 
    EOQ


    say $t;
    say invoice.parse($t);
}