使用 pandas 将所有 csv 文件合并为每个文件夹的一个大 csv 文件

Question

我有一个包含多个子目录和子子目录的目录。 我的目标是读取所有 csv 文件并将它们附加到每个子目录一个大 csv 文件中（PS 不是每个子子目录一个大 csv 文件，而是每个子目录一个文件）。

以下是我用来将所有 csv 文件附加到 1 个大 csv 文件中的代码，但此代码的唯一问题是 最终的 csv 文件大约为 ~890 MB，这使得它当您尝试在 Microsoft Excel 中打开文件时无法读取，因为它遇到“内存问题”。

dirName = 'FTP_Data/2019/'

allFilesList = list()
print('[info.] looking for .csv files.')
for (dirpath, dirnames, filenames) in os.walk(dirName):
    for file in filenames:
        if file.endswith('.csv'):
            allFilesList += [os.path.join(dirpath, file).replace('\\','/')]

print('[info.] combining all the .csv files.')
combined_csv = pd.concat([pd.read_csv(file, sort=True) for file in allFilesList])
print('[info.] exporitng to one final output file.')
combined_csv.to_csv('FinalOutputFrom2019.csv', index=False, encoding='utf-8')

以上代码生成一个名为“FinalOutputFrom2019.csv”的大 csv 文件。

现在如果目录结构如下，我希望它每月生成 12 个文件 - 2019-01.csv,2019-02.csv,2019-03.csv,2019-04.csv,2019-05.csv,2019-06.csv,@987654328 @,2019-08.csv,2019-09.csv,2019-10.csv,2019-11.csv,2019-12.csv

FTP_data
|-- 2019
    |-- 2019-01
        |-- 2019-01-01
            |-- foo.csv
            |-- bar.csv
            |-- foobar.xml
        |-- 2019-01-03
            |-- foobar.csv
        |-- ...
    |-- 2019-02
        |-- 2019-02-02
            |-- ...
        |-- 2019-02-03
            |-- ...
        |-- 2019-02-05
            |-- ...
    |-- ...

Answer 1

只需将您的工作 sn-p 调整为贯穿所有月份的 for 循环。试试这个：

设置

yr = 2019
dirName = f'FTP_Data/{yr}/'

def export_csv(dirName, csv_name):
    allFilesList = list()
    print('[info.] looking for .csv files.')
    for dirpath, dirnames, filenames in os.walk(dirName):
        for file in filenames:
            if file.endswith('.csv'):
                allFilesList.append(os.path.join(dirpath, file).replace('\\', '/'))

    print('[info.] combining all the .csv files.')
    combined_csv = pd.concat([pd.read_csv(file, sort=True) for file in allFilesList])
    print('[info.] exporting to one final output file.')
    combined_csv.to_csv(f'{csv_name}.csv', index=False, encoding='utf-8')

执行

for mon in range(1, 13):
    # yr = dirName.split("/")[1]
    csv_name = f"{yr}-{mon:02}"
    subdirName = f"{dirName}{csv_name}"
    export_csv(subdirName, csv_name)

Answer 2

import os
import pandas as pd


dirName = 'FTP_Data/2019/2019-'
Months = ('01', '02', '03', '04', '05', '06','07', '08', '09', '10', '11', '12')
# months are hard coded but if you prefer you can loop too.

Monthly_dirNames = []
i = 0
for mn in Months:
    path = dirName+mn

    #Get the list of csv files in each Sub directory:
    Monthly_Csv_Files = [f for f in os.listdir(path) if f.endswith('.csv')]

    #*Optional* Show the csv files to be merged to the user
    print(f'CSV files for month_{mn} are :', Monthly_Csv_Files)

    # Read and concat all csv files in that sub directory to a Data Frame:
    combined_csv = pd.concat([pd.read_csv(path+'/'+file) for file in Monthly_Csv_Files])

    # Write Combined Data Frame as a csv file in that subfolder:
    # Removing 'path+' from below will write all csv files to .py's directory:
    combined_csv.to_csv(path+f'/Monthly_Output_From_2019_{mn}.csv', index=False, encoding='utf-8')
    i += 1