读取文本文件时如何更正列之间的空格？

Question

我想从文本文件中读取数据并将其写入 hdf5 格式。但不知何故，在数据文件的中间，列之间的空间消失了。 small part of the file 数据看起来是这样的：

Generated by trjconv : P/L=1/400 t=   0.00000
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
.
.
 9994LI     aLI 9994  24.60  41.14   5.32
 9995LI     aLI 9995  88.47  43.02   5.72
 9996LI     aLI 9996  18.98  40.60   5.56
 9997LI     aLI 9997  35.63  46.43   5.68
 9998LI     aLI 9998  33.81  52.15   5.41
 9999LI     aLI 9999  38.72  57.18   5.32
10000LI     aLI10000  29.36  47.12   5.55
10001LI     aLI10001  82.55  44.80   5.50
10002LI     aLI10002  42.52  51.00   5.19
10003LI     aLI10003  28.61  40.21   5.70
10004LI     aLI10004  38.16  42.85   5.33
Generated by trjconv : P/L=1/400 t=   1000.00
11214
    1P1     aP1    1  80.48  35.36   4.25
    2P1     aP1    2  37.45   3.92   3.96
    3P2     aP2    3  18.53  -9.69   4.68
    4P2     aP2    4  55.39  74.34   4.60
    5P3     aP3    5  22.11  68.71   3.85
.
.
 9994LI     aLI 9994  24.60  41.14   5.32
 9995LI     aLI 9995  88.47  43.02   5.72
 9996LI     aLI 9996  18.98  40.60   5.56
 9997LI     aLI 9997  35.63  46.43   5.68
 9998LI     aLI 9998  33.81  52.15   5.41
 9999LI     aLI 9999  38.72  57.18   5.32
10000LI     aLI10000  29.36  47.12   5.55
10001LI     aLI10001  82.55  44.80   5.50
10002LI     aLI10002  42.52  51.00   5.19
10003LI     aLI10003  28.61  40.21   5.70
10004LI     aLI10004  38.16  42.85   5.33
..
..
..

数据是 t=1000 帧的集合，有一百万帧。正如您在帧末尾看到的那样，第 2 列和第 3 列相互接触。我想在读取数据时在它们之间创建空间。我遇到的另一个问题是重复的标题 Generated by..。由于 h5 文件不支持字符串，如何将它们读写到 hdf5 文件中？有没有办法手动添加它们？代码如下：

import h5py
import numpy as np

#define a np.dtype for gro array/dataset (hard-coded for now)
gro_dt = np.dtype([('col1', 'S4'), ('col2', 'S4'), ('col3', int), 
                   ('col4', float), ('col5', float), ('col6', float)])

# Next, create an empty .h5 file with the dtype
with h5py.File('xaa.h5', 'w') as hdf:
    ds= hdf.create_dataset('dataset1', dtype=gro_dt, shape=(20,), maxshape=(None,)) 

    # Next read line 1 of .gro file
    f = open('xaa', 'r')
    data = f.readlines()
    ds.attrs["Source"]=data[0]
    f.close()

    # loop to read rows from 2 until end
    skip, incr, row0 = 2, 20, 0 
    read_gro = True
    while read_gro:
        arr = np.genfromtxt('xaa', skip_header=skip, max_rows=incr, dtype=gro_dt)
        rows = arr.shape[0]
        if rows == 0:
            read_gro = False 
        else:    
            if row0+rows > ds.shape[0] :
                ds.resize((row0+rows,))
            ds[row0:row0+rows] = arr
            skip += rows
            row0 += rows

我可以跳过第一个标题，但是如何处理即将到来的标题？如果有人需要，我可以提供标题的行号。列抛出 valueError

 ValueError: Some errors were detected !
    Line #7 (got 5 columns instead of 6)
    Line #8 (got 5 columns instead of 6)
    Line #9 (got 5 columns instead of 6)

Answer 1

答案于 2021 年 9 月 9 日更新：
根据 cmets 中的请求，我添加了 2 个使用 f.readline() 的新方法。一个用索引分割线，另一个使用struct 包来解包字段。 struct 应该更快，但我没有看到测试文件的性能有显着差异（75 个时间步长）。

另外，我修改了代码以使用while True: 循环并在文件末尾中断。这避免了输入时间步数的需要。

这是我根据您对上一个问题的回答所遇到的问题而写的答案。 （参考：Reading data from gromacs file and write it to the hdf5 file format。此答案使用readlines() 将数据读入列表。（这可能是您的大文件的问题。如果是这样，可以修改解决方案以使用@987654327 逐行读取@.) 它使用与字段宽度对齐的索引对每一行的数据进行切片。警告：读取 50e6 行可能需要一段时间。注意：HDF5 支持字符串（但 h5py 不支持 NumPy Unicode 字符串）。

方法一：使用f.readlines()和进程列表。
通过使用索引对每一行进行切片来获取值：

import h5py
import numpy as np

csv_file = 'xaa.txt' # data from link in question

# define a np.dtype for gro array/dataset (hard-coded for now)
gro_dt = np.dtype([('col1', 'S7'), ('col2', 'S8'), ('col3', int), 
                   ('col4', float), ('col5', float), ('col6', float)])
 
c1, c2, c3, c4, c5 = 7, 15, 20, 27, 34
# The values above are used as indices to slice line 
# into the following fields in the loop on data[]: 
# [:7], [7:15], [15:20], [20:27], [27:34], [34:]

# Open the file for reading and
# create an empty .h5 file with the dtype above   
with open(csv_file, 'r') as f, \
     h5py.File('xaa.h5', 'w') as hdf:

    data = f.readlines()
    skip = 0
    step = 0
    while True:
        # Read text header line for THIS time step
        if skip == len(data):
            print("End Of File")
            break
        else:
            header = data[skip]
            print(header)
            skip += 1
        
        # get number of data rows
        no_rows = int(data[skip]) 
        skip += 1
        arr = np.empty(shape=(no_rows,), dtype=gro_dt)
        for row, line in enumerate(data[skip:skip+no_rows]):
            arr[row]['col1'] = line[:c1].strip()            
            arr[row]['col2'] = line[c1:c2].strip()            
            arr[row]['col3'] = int(line[c2:c3])
            arr[row]['col4'] = float(line[c3:c4])
            arr[row]['col5'] = float(line[c4:c5])
            arr[row]['col6'] = float(line[c5:])
            
        if arr.shape[0] > 0:
            # create a dataset for THIS time step
            ds= hdf.create_dataset(f'dataset_{step:04}', data=arr) 
            #create attributes for this dataset / time step
            hdr_tokens = header.split()
            ds.attrs['raw_header'] = header
            ds.attrs['Generated by'] = hdr_tokens[2]
            ds.attrs['P/L'] = hdr_tokens[4].split('=')[1]
            ds.attrs['Time'] = hdr_tokens[6]
            
        # increment by rows plus footer line that follows
        skip += 1 + no_rows

方法二：使用f.readline()逐行读取。
通过使用索引对每一行进行切片来获取值：

import h5py
import numpy as np

csv_file = 'xaa.txt'

#define a np.dtype for gro array/dataset (hard-coded for now)
gro_dt = np.dtype([('col1', 'S7'), ('col2', 'S8'), ('col3', int), 
                   ('col4', float), ('col5', float), ('col6', float)])

## gro_fmt=[0:7], [7:15], [15:20], [20:27], [27:34], [34:41]
c1, c2, c3, c4, c5 = 7, 15, 20, 27, 34
# Open the file for reading and
# create an empty .h5 file with the dtype above   
with open(csv_file, 'r') as f, \
     h5py.File('xaa.h5', 'w') as hdf:

    step = 0
    while True:
    # Read text header line for THIS time step
        header = f.readline()
        if not header:
            print("End Of File")
            break
        else:
            print(header)

        # get number of data rows
        no_rows = int(f.readline()) 
        arr = np.empty(shape=(no_rows,), dtype=gro_dt)
        for row in range(no_rows):
            line = f.readline()
            arr[row]['col1'] = line[:c1].strip()            
            arr[row]['col2'] = line[c1:c2].strip()            
            arr[row]['col3'] = int(line[c2:c3])
            arr[row]['col4'] = float(line[c3:c4])
            arr[row]['col5'] = float(line[c4:c5])
            arr[row]['col6'] = float(line[c5:])
            
        if arr.shape[0] > 0:
            # create a dataset for THIS time step
            ds= hdf.create_dataset(f'dataset_{step:04}', data=arr) 
            #create attributes for this dataset / time step
            print(header)
            hdr_tokens = header.split()
            ds.attrs['raw_header'] = header
            ds.attrs['Generated by'] = hdr_tokens[2]
            ds.attrs['P/L'] = hdr_tokens[4].split('=')[1]
            ds.attrs['Time'] = hdr_tokens[6]
            
        footer = f.readline()
        step += 1

方法三：使用f.readlines()逐行读取。
使用struct 包从每一行解包值：

import struct
import numpy as np
import h5py

csv_file = 'xaa.txt'

fmtstring = '7s 8s 5s 7s 7s 7s'
fieldstruct = struct.Struct(fmtstring)
parse = fieldstruct.unpack_from

#define a np.dtype for gro array/dataset (hard-coded for now)
gro_dt = np.dtype([('col1', 'S7'), ('col2', 'S8'), ('col3', int), 
                   ('col4', float), ('col5', float), ('col6', float)])

with open(csv_file, 'r') as f, \
     h5py.File('xaa.h5', 'w') as hdf:
         
    step = 0
    while True:         
        header = f.readline()
        if not header:
            print("End Of File")
            break
        else:
            print(header)

        # get number of data rows
        no_rows = int(f.readline())
        arr = np.empty(shape=(no_rows,), dtype=gro_dt)
        for row in range(no_rows):
            fields = parse( f.readline().encode('utf-8') )
            arr[row]['col1'] = fields[0].strip()            
            arr[row]['col2'] = fields[1].strip()            
            arr[row]['col3'] = int(fields[2])
            arr[row]['col4'] = float(fields[3])
            arr[row]['col5'] = float(fields[4])
            arr[row]['col6'] = float(fields[5])
        if arr.shape[0] > 0:
            # create a dataset for THIS time step
            ds= hdf.create_dataset(f'dataset_{step:04}', data=arr) 
            #create attributes for this dataset / time step
            hdr_tokens = header.split()
            ds.attrs['raw_header'] = header
            ds.attrs['Generated by'] = hdr_tokens[2]
            ds.attrs['P/L'] = hdr_tokens[4].split('=')[1]
            ds.attrs['Time'] = hdr_tokens[6]
            
        footer = f.readline()
        step += 1