如何在 Python 中从文本数据文件中提取数据子集

Question

我有一个文本文件，其中相关数据（行 x 列）仅出现在“开始”和“结束”关键字之间。见下文。我想编写一个可以提取这些数据子集的代码。如果一行以“start”开头，后跟数据，但不跟随后的“end”关键字，那么我想忽略该数据。在下面的示例中，data1 和 data3 相关，但 data2 不相关，因为它没有包含在“start”和“end”关键字中。

start
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end"
end
start
data2 (matrix of data) /not relevant because there is no "end"
. 
start
data3 (matrix of data) /relevant for same reason as for data1
end
.
.
and so on

我想我可以开始：

with open(file_path,'r') as file:

    text = file.readlines()
    start_indexes = []
    end_indexes = []

    for i, line in enumerate(text):
        if line.startswith('start'):
            start_indexes.append(i)
        elif line.startswith('end'):
            end_indexes.append(i)

    for i in range(len(start_indexes)):
        for j in range(len(end_indexes)):
            if (start_indexes[i] < end_indexes[j] < start_indexes[i+1]):
                print(start_indexes[i],end_indexes[j])

上面的代码给了我有相关数据的起始行号和结束行号。这是我有点卡住的地方。我现在如何提取相关数据？在下面的示例中，它将是 data1，data3。我是否以“正确”的方式解决问题？我应该求助于熊猫吗？有没有更高效直接的方法？

Answer 1

嵌套 For 循环？

您正在经历开始和结束范围的每一种组合。您只需要与同一条数据相对应的那些。

用这样的东西替换你的 for 循环：

for start, end in zip(start_indexes, end_indexes):
    print(text[start + 1:end])

zip(a, b, ...) 返回一个包含a, b, ... 列的新列表，本质上给出[(a[0], b[0], ...), (a[1], b[1], ...), ...]。您遍历start_indexes, end_indexes 的每一列，为您提供相应的开始和结束值，然后使用列表切片访问来获取这些行的数据。

Answer 2

我会使用另一种方式，仅按顺序读取文件（假设“开始”-“结束”-块中的数据不太大）。我会创建一个data 变量来收集当前块的数据（无论是否相关）和一个带有状态转换的state 变量。

一些伪 Python：

if state == OUTSIDE_BLOCK and line.startswith('start'):
    state = INSIDE_BLOCK
elif state == INSIDE_BLOCK and <line contains data>:
    <Add to data variable>
elif state == INSIDE_BLOCK and line.startswith('end'):
    state = OUTSIDE_BLOCK
    <Process collected data>
elif state == INSIDE_BLOCK and line.startswith('start'):
    <Throw away possibly collected data because it was irrelevant>

Answer 3

设置：

s = '''start
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end"
end
start
data2 (matrix of data) /not relevant because there is no "end"
start
data3 (matrix of data) /relevant for same reason as for data1
end
start
data4 blah
'''
import io
f = io.StringIO(s)

遍历文件一次，测试每一行的开头；找出将有效数据块放入子列表并将它们附加到结果列表所需的逻辑...

result = []
sub = []

for line in f:
    if line.startswith('start'):
        # possible new data block
        if sub:
            # if it isn't empty it must contain
            # a start --> data block with no end
            result.append(sub)
            sub = []
        sub = [line]
    elif line.startswith('end'):
        # start over
        sub = []
    elif line.startswith('data'):
        sub.append(line)
    else:
        # for lines that don't startwith data, start or end - if any 
        sub.append(line)

if sub:
    # start --> data --> EOF or end of string
    result.append(sub)

Answer 4

我个人认为正则表达式是处理这种情况的最佳方式：

import re woof0='''start data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" end start data2 (matrix of data) /not relevant because there is no "end" . start data3 (matrix of data) /relevant for same reason as for data1 end . . and so on ''' re.findall(r'start(\sdata.*|\Sdata.*)\nend',woof0)

输出：

['\ndata1 (matrix of data) /relevant because data1 is enclosed by "start" and "end"', '\ndata3 (matrix of data) /relevant for same reason as for data1']