shutil.copyfileobj 但没有标题或跳过第一行

Question

我有大约 8000 个 text 文件，其中包含 csv 数据，例如

CustomerID,Gender,Day,SaleAmount
18,Male,Monday,71.55
24,Female,Monday,219.66
112,Male,Friday,150.44

我的代码循环遍历所有文件，然后将其附加到final.txt-

with open('final.txt', 'wb') as outfile:
    for filename in files:
        with open(filename, 'rb') as readfile:
            shutil.copyfileobj(readfile, outfile)

现在的问题是因为每个文件都有自己的标题，即

+------------+--------+-----+------------+
| CustomerID | Gender | Day | SaleAmount |
+------------+--------+-----+------------+

我的最终内容是这样的 -

+------------+--------+--------+------------+
| CustomerID | Gender |  Day   | SaleAmount |
+------------+--------+--------+------------+
| 18         | Male   | Monday | 71.55      |
| 24         | Female | Monday | 219.66     |
| 112        | Male   | Friday | 150.44     |
| CustomerID | Gender | Day    | SaleAmount |
| 28         | Male   | Monday | 7.55       |
| 34         | Female | Monday | 19.66      |
| 12         | Female | Friday | 150.44     |
| CustomerID | Gender | Day    | SaleAmount |
| 28         | Male   | Monday | 7.55       |
| 34         | Female | Monday | 19.66      |
| 12         | Female | Friday | 150.44     |
+------------+--------+--------+------------+

有没有办法使用 shutil.copyfileobj 将所有 8000 个 txt 文件合并为一个只保留一个标题？

我尝试过使用 pd.read_csv 但 copyfileobj 的速度是原来的两倍。还有其他更快的方法吗？

编辑 - 我直接从 txt 文件而不是数据帧中读取。

Answer 1

使用这个方法

def copy_csv(fname):
    allFiles = glob.glob(fname)
    allFiles.sort()  # glob lacks reliable ordering, so impose your own if output order matters
    with open(fname+'.csv', 'wb') as outfile:
        for i, fname in enumerate(allFiles):
            with open(fname, 'rb') as infile:
                if i != 0:
                    infile.readline()  # Throw away header on all but first file
                # Block copy rest of file from input to output without parsing
                shutil.copyfileobj(infile, outfile)
            infile.close()
    outfile.close()

Answer 2

import pandas as pd
from pathlib import Path

files = Path.cwd().glob('**/*.csv')

or

files = Path('c:/path_to_files').glob('**/*.csv')  # ** looks in all subdirectories
df = pd.concat([pd.read_csv(file) for file in files])

df.reset_index(inplace=True)  # if you want

df.to_csv('new.csv', index=False, sep=',')