Pandas：读取几个大的 .bz2 文件并附加它答案

【问题标题】：Pandas: Reading in several large .bz2 files and appending itPandas：读取几个大的 .bz2 文件并附加它
【发布时间】：2020-04-30 16:38:12
【问题描述】：

我有 30 个要读入的 .bz2 文件。每个文件都太大而无法读入，因此每个文件的 x 大小块就足够了。然后我想将所有这 30 个文件连接在一起。

import pandas as pd
import numpy as np
import glob
path = r'/content/drive/My Drive/'                     # use your path
all_files = glob.glob(os.path.join(path, "*.bz2"))     # advisable to use os.path.join as this makes concatenation OS independent

# Below I read 10,000 lines X 11 in for each file because of RAM limit and append it together. 
# How do I make it so it also appends each of the 30 files together?? I made an attempt below.

chunks = (pd.read_json(f, lines=True, chunksize = 1000) for f in all_files)
i = 0
chunk_list = []
for chunk in chunks:
    if i >= 11:
        break
    i += 1
    chunk_list.append(chunk)
    df = pd.concat(chunk_list, sort = True)
#print(df)
df

.bz2 数据示例可在以下位置找到： https://csr.lanl.gov/data/2017.html

【问题讨论】：

标签： python json pandas for-loop glob

【解决方案1】：

import os, json
import pandas as pd
import numpy as np
import glob
pd.set_option('display.max_columns', None)

temp = pd.DataFrame()

path_to_json = '/content/drive/My Drive/' 

json_pattern = os.path.join(path_to_json,'*.bz2')
file_list = glob.glob(json_pattern)

for file in file_list:
    chunks = pd.read_json(file, lines=True, chunksize=1000)
    i = 0
    chunk_list = []
    for chunk in chunks:
        if i >= 10:
            break
        i += 1
        chunk_list.append(chunk)
        df = pd.concat(chunk_list, sort = True)
    temp = temp.append(df, sort = True)
temp

这似乎有效

【讨论】：