某种方式以相反的顺序逐行读取文本文件？ [复制]

Question

我想逐行读取下面给出的文本文件。我不想使用readlines() 或read()。

a.txt

2018/03/25-00:08:48.638553  508     7FF4A8F3D704     snononsonfvnosnovoosr
2018/03/25-10:08:48.985053 346K     7FE9D2D51706     ahelooa afoaona woom
2018/03/25-20:08:50.486601 1.5M     7FE9D3D41706     qojfcmqcacaeia
2018/03/25-24:08:50.980519  16K     7FE9BD1AF707     user: number is 93823004
2018/03/26-00:08:50.981908 1389     7FE9BDC2B707     user 7fb31ecfa700
2018/03/26-10:08:51.066967    0     7FE9BDC91700     Exit Status = 0x0
2018/03/26-15:08:51.066968    1     7FE9BDC91700     std:ZMD:

预期结果：

2018/03/26-15:08:51.066968    1     7FE9BDC91700     std:ZMD:
2018/03/26-10:08:51.066967    0     7FE9BDC91700     Exit Status = 0x0
2018/03/26-00:08:50.981908 1389     7FE9BDC2B707     user 7fb31ecfa700
2018/03/25-24:08:50.980519  16K     7FE9BD1AF707     user: number is 93823004
2018/03/25-20:08:50.486601 1.5M     7FE9D3D41706     qojfcmqcacaeia
2018/03/25-10:08:48.985053 346K     7FE9D2D51706     ahelooa afoaona woom
2018/03/25-00:08:48.638553  508     7FF4A8F3D704     snononsonfvnosnovoosr

我的解决方案：

with open('a.txt') as lines:
    for line in reversed(lines):
        print(line)

Answer 1

这是一种无需一次性将整个文件读入内存的方法。它确实需要首先读取整个文件，但只存储每行开始的位置。一旦知道了，它就可以使用seek() 方法以任何所需的顺序随机访问每一个。

这是一个使用您的输入文件的示例：

# Preprocess - read whole file and note where lines start.
# (Needs to be done in binary mode.)
with open('text_file.txt', 'rb') as file:
    offsets = [0]  # First line is always at offset 0.
    for line in file:
        offsets.append(file.tell())  # Append where *next* line would start.

# Now reread lines in file in reverse order.
with open('text_file.txt', 'rb') as file:
    for index in reversed(range(len(offsets)-1)):
        file.seek(offsets[index])
        size = offsets[index+1] - offsets[index]  # Difference with next.
        # Read bytes, convert them to a string, and remove whitespace at end.
        line = file.read(size).decode().rstrip()
        print(line)

输出：

2018/03/26-15:08:51.066968    1     7FE9BDC91700     std:ZMD:
2018/03/26-10:08:51.066967    0     7FE9BDC91700     Exit Status = 0x0
2018/03/26-00:08:50.981908 1389     7FE9BDC2B707     user 7fb31ecfa700
2018/03/25-24:08:50.980519  16K     7FE9BD1AF707     user: number is 93823004
2018/03/25-20:08:50.486601 1.5M     7FE9D3D41706     qojfcmqcacaeia
2018/03/25-10:08:48.985053 346K     7FE9D2D51706     ahelooa afoaona woom
2018/03/25-00:08:48.638553  508     7FF4A8F3D704     snononsonfvnosnovoosr

更新

这是一个执行相同操作但使用 Python 的 mmap 模块到 memory-map 文件的版本，该文件应该通过利用您的操作系统/硬件的虚拟内存功能来提供更好的性能。

这是因为，正如PyMOTW-3 所说：

内存映射通常会提高 I/O 性能，因为它不涉及每次访问的单独系统调用，也不需要在缓冲区之间复制数据 - 内存由内核和用户应用程序直接访问。

代码：

import mmap

with open('text_file.txt', 'rb') as file:
    with mmap.mmap(file.fileno(), length=0, access=mmap.ACCESS_READ) as mm_file:

        # First preprocess the file and note where lines start.
        # (Needs to be done in binary mode.)
        offsets = [0]  # First line is always at offset 0.
        for line in iter(mm_file.readline, b""):
            offsets.append(mm_file.tell())  # Append where *next* line would start.

        # Now process the lines in file in reverse order.
        for index in reversed(range(len(offsets)-1)):
            mm_file.seek(offsets[index])
            size = offsets[index+1] - offsets[index]  # Difference with next.
            # Read bytes, convert them to a string, and remove whitespace at end.
            line = mm_file.read(size).decode().rstrip()
            print(line)

Answer 2

不，没有更好的方法可以做到这一点。根据定义，文件是一些基本数据类型的顺序组织。文本文件的类型是字符。您正在尝试对文件强加不同的组织，字符串由换行符分隔。

因此，您必须完成读取文件的工作，重新转换为所需的格式，然后然后以相反的顺序对该组织进行处理。例如，您是否需要多次...将文件作为行读取，将行存储为数据库记录，然后在您认为合适的时候遍历记录。

file 接口只在一个方向读取。您可以seek() 到另一个位置，但标准 I/O 操作仅适用于增加位置描述。

为了使您的解决方案正常工作，您需要读入整个文件——您不能reverse 文件描述符的隐式迭代器。

Answer 3

虽然@martineau 的解决方案在不将整个文件加载到内存的情况下完成了工作，但它还是浪费了两次读取整个文件。

一种更有效的一次性方法是从文件末尾以相当大的块读取到缓冲区中，从缓冲区末尾查找下一个换行符（减去最后一个字符的尾随换行符） )，如果未找到，则向后搜索并继续读取块并将块添加到缓冲区，直到找到换行符。只要在内存限制内，就使用更大的块大小来提高读取效率：

class ReversedTextReader:
    def __init__(self, file, chunk_size=50):
        self.file = file
        file.seek(0, 2)
        self.position = file.tell()
        self.chunk_size = chunk_size
        self.buffer = ''

    def __iter__(self):
        return self

    def __next__(self):
        if not self.position and not self.buffer:
            raise StopIteration
        chunk = self.buffer
        while True:
            line_start = chunk.rfind('\n', 0, len(chunk) - 1 - (chunk is self.buffer))
            if line_start != -1:
                break
            chunk_size = min(self.chunk_size, self.position)
            self.position -= chunk_size
            self.file.seek(self.position)
            chunk = self.file.read(chunk_size)
            if not chunk:
                line = self.buffer
                self.buffer = ''
                return line
            self.buffer = chunk + self.buffer
        line_start += 1
        line = self.buffer[line_start:]
        self.buffer = self.buffer[:line_start]
        return line

这样：

from io import StringIO

f = StringIO('''2018/03/25-00:08:48.638553  508     7FF4A8F3D704     snononsonfvnosnovoosr
2018/03/25-10:08:48.985053 346K     7FE9D2D51706     ahelooa afoaona woom
2018/03/25-20:08:50.486601 1.5M     7FE9D3D41706     qojfcmqcacaeia
2018/03/25-24:08:50.980519  16K     7FE9BD1AF707     user: number is 93823004
2018/03/26-00:08:50.981908 1389     7FE9BDC2B707     user 7fb31ecfa700
2018/03/26-10:08:51.066967    0     7FE9BDC91700     Exit Status = 0x0
2018/03/26-15:08:51.066968    1     7FE9BDC91700     std:ZMD:
''')

for line in ReversedTextReader(f):
    print(line, end='')

输出：

2018/03/26-15:08:51.066968    1     7FE9BDC91700     std:ZMD:
2018/03/26-10:08:51.066967    0     7FE9BDC91700     Exit Status = 0x0
2018/03/26-00:08:50.981908 1389     7FE9BDC2B707     user 7fb31ecfa700
2018/03/25-24:08:50.980519  16K     7FE9BD1AF707     user: number is 93823004
2018/03/25-20:08:50.486601 1.5M     7FE9D3D41706     qojfcmqcacaeia
2018/03/25-10:08:48.985053 346K     7FE9D2D51706     ahelooa afoaona woom
2018/03/25-00:08:48.638553  508     7FF4A8F3D704     snononsonfvnosnovoosr