如何从巨大的 tar.gz 中的 csv 文件中逐块获取 pandas 数据帧，而无需解压缩和迭代它们？

Question

我有一个巨大的压缩文件，我有兴趣在其中读取各个数据帧，以免内存不足。

另外，由于时间和空间的关系，我无法解压.tar.gz。

这是我到目前为止得到的代码：

import pandas as pd
# With this lib we can navigate on a compressed files
# without even extracting its content
import tarfile
import io

tar_file = tarfile.open(r'\\path\to\the\tar\file.tar.gz')

# With the following code we can iterate over the csv contained in the compressed file
def generate_individual_df(tar_file):
    return \
        (
            (
                member.name, \
                pd.read_csv(io.StringIO(tar_file.extractfile(member).read().decode('ascii')), header=None)
            )
               for member in tar_file
                   if member.isreg()\
        )

for filename, dataframe in generate_individual_df(tar_file):
    # But dataframe is the whole file, which is too big

尝试了How to create Panda Dataframe from csv that is compressed in tar.gz?，但仍然无法解决...

Answer 1

您实际上可以使用以下函数遍历压缩文件中的块：

def generate_individual_df(tar_file, chunksize=10**4):
    return \
        (
            (
                member.name, \
                chunk
            )
            for member in tar_file
                if member.isreg()\
                for chunk in pd.read_csv(io.StringIO(tar_file.extractfile(member)\
                  .read().decode('ascii')), header=None, chunksize=chunksize)
        )

Answer 2

您可以使用 glob 模块通过 glob 获取 zip 中的某些文件 例如我希望 cv2 读取文件中的图像

 import glob
 import cv2
    
    file1 = glob.glob(filepath/ "*.extension")
    for image in file1:
       image = cv2.imread(image)
hope it works