Python：由许多标题分割一个大文本文件

Question

我有一个大文本文件，如下所示：

lat lon altitude pressure
3 lines data group bsas
2.3 4.5 45.0 875
5.6 6.5 46.2 676
3.4 3.4 48.2 565
6 lines data group sdad
3.4 4.5 56.1 535
5.6 6.5 46.2 676    
3.4 4.5 56.1 535
2.3 4.5 45.0 875
5.6 6.5 46.2 676
3.4 3.4 48.2 565
50 lines data group asdasd
5.5 6.6 44.5 343
...
3.7 8.4 56.5 456
... and so on

我想将整个文本文件拆分为单独的数据组，每个数据组将存储在一个二维数组中。到目前为止，我已经尝试了两种方法。

第一种方式是遍历每一行，获取数据如下：

# define an object class called Wave here
# each object has 4 attributes: lat, lon, altitude, pressure
wave_list = []
with open(filename, 'r') as f:
    next(f) # skip the header
    wave = Wave()
    for i, line in enumerate(f):
        if 'data' in line:
            if wave is not empty:
                wave_list.append(wave)
            wave = Wave()
        else:
            wave.lat.append(line.split()[0])
            wave.lon.append(line.split()[1])
            wave.altitude.append(line.split()[2])
            wave.pressure.append(line.split()[3])
        wave_list.append(wave)
return wave_list

第二种方法是使用numpy loadtext：

f = open(filename, 'r')
txt = f.read()
# split by "data", remove the first element
raw_chunks = txt.split("data")[1:]
# define a new list to store results
wave_list = []
# go through each chunk
for rc in raw_chunks:
    # find the fisrt index of "\n"
    first_id = rc.find("\n")
    # find the last index of "\n"
    last_id = rc.rfind("\n")
    # temporary chunk
    temp_chunk = rc[first_id:last_id]
    # load data using loadtxt
    data = np.loadtxt(StringIO(temp_chunk)          
    wave = Wave()
    wave.lat = data.T[0]
    wave.lon = data.T[1]
    wave.altitude = data.T[2]
    wave.pressure = data.T[3]
    wave_list.append(wave)
return wave_list

但是，这两种方法都很慢。我查看了 pandas 文档，但找不到避免文件中间出现标题的方法。我还查看了不同的问题作为示例：

Splitting a file based on text in Python

Split the text file in python

How to split and parse a big text file in python in a memory-efficient way?

但它们都不能解决我的问题。有没有更快的方法来读取这种文本文件。提前谢谢你。

Answer 1

搜索以<number> lines data group <something>开头的行，存储组（<something>）和要读取的行数（<number>），然后当匹配时，存储n该组的行数，例如：

给定以下代码：

from itertools import islice
from collections import defaultdict
import re

data = defaultdict(list) 
with open(filename) as fin:
    header = next(fin, '').split()
    for line in fin:
        m = re.match(r'(\d+) lines.*(\b\w+)$', line)
        if m:
            data[m.group(2)].extend(islice(fin, int(m.group(1))))

给定输入：

lat lon altitude pressure
3 lines data group bsas
2.3 4.5 45.0 875
5.6 6.5 46.2 676
3.4 3.4 48.2 565
6 lines data group sdad
3.4 4.5 56.1 535
5.6 6.5 46.2 676    
3.4 4.5 56.1 535
2.3 4.5 45.0 875
5.6 6.5 46.2 676
3.4 3.4 48.2 565

给你data：

{'bsas': ['2.3 4.5 45.0 875\n', '5.6 6.5 46.2 676\n', '3.4 3.4 48.2 565\n'],
 'sdad': ['3.4 4.5 56.1 535\n',
          '5.6 6.5 46.2 676    \n',
          '3.4 4.5 56.1 535\n',
          '2.3 4.5 45.0 875\n',
          '5.6 6.5 46.2 676\n',
          '3.4 3.4 48.2 565\n']}

进一步说，如果“组”是微不足道的，那么：

data = []
with open(filename) as fin:
    header = next(fin, '').split()
    for line in fin:
        m = re.match(r'(\d+) lines.*(\b\w+)$', line)
        if m:
            data.append(list(islice(fin, int(m.group(1)))))