根据终端中的括号或括号（仅限顶级）拆分文本文件

Question

我有几个要在 shell 脚本中处理的文本文件 (utf-8)。它们的格式不完全相同，但如果我只能将它们分成可食用的块，我可以处理。 这可以用 C 或 python 编程，但我不喜欢。

编辑：我用 C 写了一个解决方案；看我自己的答案。我认为这可能是最简单的方法。如果您认为我错了，请根据下面我的答案中更复杂的示例输入来测试您的解决方案。

--jcxz100

为了清楚起见（并且能够更轻松地调试），我希望将块保存为子文件夹中的单独文本文件。

所有类型的输入文件包括：

1. 垃圾线
2. 带有垃圾文本的行，后跟开始括号或括号 - 即 '[' '{' '
3. 有效载荷行
4. 带有括号或圆括号的行嵌套在顶级对中；也被视为有效载荷
5. 带有结束括号或括号的有效负载行 - 即 ']' '}' '>' 或 ')' - 后面可能跟一些内容（垃圾文本和/或新有效负载的开头）

我想仅根据匹配的 top-level 括号/括号对分解输入。 不得更改这些对中的有效负载（包括换行符和空格）。 顶级对之外的所有内容都应作为垃圾丢弃。

双引号内的任何垃圾或有效负载都必须被视为原子（作为原始文本处理，因此其中的任何括号或括号也应被视为文本）。

这是一个示例（仅使用 {} 对）：

junk text
"atomic junk"

some junk text followed by a start bracket { here is the actual payload
   more payload
   "atomic payload"
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
      "yet more atomic payload; this one's got a smiley ;-)"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
} trailing junk
intermittent junk
{
   payload that goes in second output file    }
end junk

...对不起：有些输入文件确实像那样乱七八糟。

第一个输出文件应该是：

{ here is the actual payload
   more payload
   "atomic payload"
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
      "yet more atomic payload; this one's got a smiley ;-)"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
}

...和第二个输出文件：

{
   payload that goes in second output file    }

注意：

• 我还没有完全决定是否有必要在输出中保留这对开始/结束字符，或者它们本身是否应该作为垃圾丢弃。 我认为保留它们的解决方案更通用。

• 同一输入文件中可以有多种类型的顶级括号/括号对。

• 注意：输入文件中有 * 和 $ 字符，所以请避免混淆 bash ;-)

• 比起简洁，我更喜欢可读性；但不会以指数级的速度为代价。

锦上添花：

• 文本中有反斜杠转义的双引号；最好应该处理它们 （我有一个 hack，但它并不漂亮）。

• 脚本不应破坏垃圾和/或有效负载中不匹配的括号/括号对（注意：在原子内部，它们必须被允许！）

更多精彩：

• 我还没有看到它，但可以推测某些输入可能有单引号而不是双引号来表示原子内容......甚至两者兼而有之。

• 如果可以轻松修改脚本以解析类似结构但具有不同开始/结束字符或字符串的输入，那就太好了。

我可以看出这是一口难言，但我认为如果我把它分解成更简单的问题，它不会给出一个可靠的解决方案。

主要问题是正确拆分输入 - 其他所有内容都可以通过 hack 忽略或“解决”，所以 随意忽略 nice-to-haves 和 more-far-out-nice-to-haves。

Answer 1

给定：

$ cat file
junk text
"atomic junk"

some junk text followed by a start bracket { here is the actual payload
   more payload
   "atomic payload"
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
      "yet more atomic payload; this one's got a smiley ;-)"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
} trailing junk
intermittent junk
{
   payload that goes in second output file    }
end junk

此 perl 文件会将您描述的块提取到文件 block_1、block_2 等中：

#!/usr/bin/perl
use v5.10;
use warnings;
use strict;

use Text::Balanced qw(extract_multiple extract_bracketed);

my $txt;

while (<>){$txt.=$_;}  # slurp the file

my @blocks = extract_multiple(
    $txt,
    [
        # Extract {...}
        sub { extract_bracketed($_[0], '{}') },
    ],
    # Return all the fields
    undef,
    # Throw out anything which does not match
    1
);
chdir "/tmp";
my $base="block_";
my $cnt=1;
for my $block (@blocks){ my $fn="$base$cnt";
                         say "writing $fn";
                         open (my $fh, '>', $fn) or die "Could not open file '$fn' $!";
                         print $fh "$block\n";
                         close $fh;
                         $cnt++;}

现在是文件：

$ cat block_1
{ here is the actual payload
   more payload
   "atomic payload"
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
      "yet more atomic payload; this one's got a smiley ;-)"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
}

$ cat block_2
{
   payload that goes in second output file    }

使用Text::Balanced 是可靠的并且可能是最佳解决方案。

您可以使用单个 Perl regex 来完成这些块：

$ perl -0777 -nlE 'while (/(\{(?:(?1)|[^{}]*+)++\})|[^{}\s]++/g) {if ($1) {$cnt++; say "block $cnt:== start:\n$1\n== end";}}' file
block 1:== start:
{ here is the actual payload
   more payload
   "atomic payload"
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
      "yet more atomic payload; this one's got a smiley ;-)"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
}
== end
block 2:== start:
{
   payload that goes in second output file    }
== end

但这比使用像Text::Balanced这样的适当解析器要脆弱一些...

Answer 2

我有一个 C 语言的解决方案。这似乎太复杂了，无法在 shell 脚本中轻松实现。 该程序并不过分复杂，但有 200 多行代码，其中包括错误检查、一些速度优化和其他细节。

源文件split-brackets-to-chunks.c：

#include <stdio.h>

/* Example code by jcxz100 - your problem if you use it! */

#define BUFF_IN_MAX 255
#define BUFF_IN_SIZE (BUFF_IN_MAX+1)

#define OUT_NAME_MAX 31
#define OUT_NAME_SIZE (OUT_NAME_MAX+1)

#define NO_CHAR '\0'

int main()
{
    char pcBuff[BUFF_IN_SIZE];
    size_t iReadActual;
    FILE *pFileIn, *pFileOut;
    int iNumberOfOutputFiles;
    char pszOutName[OUT_NAME_SIZE];
    char cLiteralChar, cAtomicChar, cChunkStartChar, cChunkEndChar;
    int iChunkNesting;
    char *pcOutputStart;
    size_t iOutputLen;

    pcBuff[BUFF_IN_MAX] = '\0';  /* ... just to be sure. */
    iReadActual = 0;
    pFileIn = pFileOut = NULL;
    iNumberOfOutputFiles = 0;
    pszOutName[OUT_NAME_MAX] = '\0';  /* ... just to be sure. */
    cLiteralChar = cAtomicChar = cChunkStartChar = cChunkEndChar = NO_CHAR;
    iChunkNesting = 0;
    pcOutputStart = (char*)pcBuff;
    iOutputLen = 0;

    if ((pFileIn = fopen("input-utf-8.txt", "r")) == NULL)
    {
        printf("What? Where?\n");
        return 1;
    }

    while ((iReadActual = fread(pcBuff, sizeof(char), BUFF_IN_MAX, pFileIn)) > 0)
    {
        char *pcPivot, *pcStop;

        pcBuff[iReadActual] = '\0'; /* ... just to be sure. */
        pcPivot = (char*)pcBuff;
        pcStop = (char*)pcBuff + iReadActual;

        while (pcPivot < pcStop)
        {
            if (cLiteralChar != NO_CHAR) /* Ignore this char? */
            {
                /* Yes, ignore this char. */

                if (cChunkStartChar != NO_CHAR)
                {
                    /* ... just write it out: */
                    fprintf(pFileOut, "%c", *pcPivot);
                }
                pcPivot++;
                cLiteralChar = NO_CHAR;

                /* End of "Yes, ignore this char." */
            }
            else if (cAtomicChar != NO_CHAR) /* Are we inside an atomic string? */
            {
                /* Yup; we are inside an atomic string. */

                int bBreakInnerWhile;
                bBreakInnerWhile = 0;

                pcOutputStart = pcPivot;
                while (bBreakInnerWhile == 0)
                {
                    if (*pcPivot == '\\') /* Treat next char as literal? */
                    {
                        cLiteralChar = '\\'; /* Yes. */
                        bBreakInnerWhile = 1;
                    }
                    else if (*pcPivot == cAtomicChar) /* End of atomic? */
                    {
                        cAtomicChar = NO_CHAR; /* Yes. */
                        bBreakInnerWhile = 1;
                    }
                    if (++pcPivot == pcStop) bBreakInnerWhile = 1;
                }
                if (cChunkStartChar != NO_CHAR)
                {
                    /* The atomic string is part of a chunk. */
                    iOutputLen = (size_t)(pcPivot-pcOutputStart);
                    fprintf(pFileOut, "%.*s", iOutputLen, pcOutputStart);
                }

                /* End of "Yup; we are inside an atomic string." */
            }
            else if (cChunkStartChar == NO_CHAR) /* Are we inside a chunk? */
            {
                /* No, we are outside a chunk. */

                int bBreakInnerWhile;
                bBreakInnerWhile = 0;
                while (bBreakInnerWhile == 0)
                {
                    /* Detect start of anything interesting: */
                    switch (*pcPivot)
                    {
                        /* Start of atomic? */
                        case '"':
                        case '\'':
                            cAtomicChar = *pcPivot;
                            bBreakInnerWhile = 1;
                            break;

                        /* Start of chunk? */
                        case '{':
                            cChunkStartChar = *pcPivot;
                            cChunkEndChar = '}';
                            break;
                        case '[':
                            cChunkStartChar = *pcPivot;
                            cChunkEndChar = ']';
                            break;
                        case '(':
                            cChunkStartChar = *pcPivot;
                            cChunkEndChar = ')';
                            break;
                        case '<':
                            cChunkStartChar = *pcPivot;
                            cChunkEndChar = '>';
                            break;
                    }
                    if (cChunkStartChar != NO_CHAR)
                    {
                        iNumberOfOutputFiles++;
                        printf("Start '%c' '%c' chunk (file %04d.txt)\n", *pcPivot, cChunkEndChar, iNumberOfOutputFiles);
                        sprintf((char*)pszOutName, "output/%04d.txt", iNumberOfOutputFiles);
                        if ((pFileOut = fopen(pszOutName, "w")) == NULL)
                        {
                            printf("What? How?\n");
                            fclose(pFileIn);
                            return 2;
                        }
                        bBreakInnerWhile = 1;
                    }
                    else if (++pcPivot == pcStop)
                    {
                        bBreakInnerWhile = 1;
                    }
                }

                /* End of "No, we are outside a chunk." */
            }
            else
            {
                /* Yes, we are inside a chunk. */

                int bBreakInnerWhile;
                bBreakInnerWhile = 0;

                pcOutputStart = pcPivot;
                while (bBreakInnerWhile == 0)
                {
                    if (*pcPivot == cChunkStartChar)
                    {
                        /* Increase level of brackets/parantheses: */
                        iChunkNesting++;
                    }
                    else if (*pcPivot == cChunkEndChar)
                    {
                        /* Decrease level of brackets/parantheses: */
                        iChunkNesting--;
                        if (iChunkNesting == 0)
                        {
                            /* We are now outside chunk. */
                            bBreakInnerWhile = 1;
                        }
                    }
                    else
                    {
                        /* Detect atomic start: */
                        switch (*pcPivot)
                        {
                            case '"':
                            case '\'':
                                cAtomicChar = *pcPivot;
                                bBreakInnerWhile = 1;
                                break;
                        }
                    }
                    if (++pcPivot == pcStop) bBreakInnerWhile = 1;
                }
                iOutputLen = (size_t)(pcPivot-pcOutputStart);
                fprintf(pFileOut, "%.*s", iOutputLen, pcOutputStart);
                if (iChunkNesting == 0)
                {
                    printf("File done.\n");
                    cChunkStartChar = cChunkEndChar = NO_CHAR;
                    fclose(pFileOut);
                    pFileOut = NULL;
                }

                /* End of "Yes, we are inside a chunk." */
            }
        }
    }
    if (cChunkStartChar != NO_CHAR)
    {
        printf("Chunk exceeds end-of-file. Exiting gracefully.\n");
        fclose(pFileOut);
        pFileOut = NULL;
    }

    if (iNumberOfOutputFiles == 0) printf("Nothing to do...\n");
    else printf("All done.\n");
    fclose(pFileIn);
    return 0;
}

我已经解决了 nice-to-haves 和 more-far-out-nice-to-haves 之一。 为了说明这一点，输入比问题中的示例复杂一点：

junk text
"atomic junk"

some junk text followed by a start bracket { here is the actual payload
   more payload
   'atomic payload { with start bracket that should be ignored'
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
"this atomic has a literal double-quote \" inside"
      "yet more atomic payload; this one's got a smiley ;-) and a heart <3"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
   "here's a totally unprovoked $ sign and an * asterisk"
} trailing junk
intermittent junk
<
   payload that goes in second output file } mismatched end bracket should be ignored     >
end junk

结果文件output/0001.txt：

{ here is the actual payload
   more payload
   'atomic payload { with start bracket that should be ignored'
   nested start bracket { - all of this line is untouchable payload too
      here is more payload
"this atomic has a literal double-quote \" inside"
      "yet more atomic payload; this one's got a smiley ;-) and a heart <3"
   end of nested bracket pair } - all of this line is untouchable payload too
   this is payload too
   "here's a totally unprovoked $ sign and an * asterisk"
}

... 和结果文件 output/0002.txt:

<
   payload that goes in second output file } mismatched end bracket should be ignored     >

感谢@dawg 的帮助:)