如何根据给定的标准将 csv 文件拆分为多个 csv？

Question

我需要根据给定时间拆分几个 csv 文件。在这些文件中，时间值以秒为单位并在“时间”列中给出。

例如，如果我想在 0.1 秒内拆分 aaa.csv 文件，则需要将时间为 0.0 到 0.1 的第一组行（附件中没有 1 到 8）写入 aaa1.csv，然后时间大于 0.1 到 0.2 的行（附件中的第 9 到 21 号）到aaa2.csv 等等...（基本上是给定时间的倍数）。

输出文件需要与输入文件同名，并在末尾加上一个数字。并且输出文件需要写入不同的位置/文件夹。 时间值需要是一个变量。所以一次我可以在 0.1 秒内拆分文件，另一次我可以在 0.7 秒内拆分文件，依此类推。

请问我该如何为此编写一个 python 脚本？文件如下（整个119K文件可以从https://fil.email/vnsZsp7b下载）：

No.,Time,Length
1,0,146
2,0.006752,116
3,0.019767,156
4,0.039635,144
5,0.06009,147
6,0.069165,138
7,0.0797,133
8,0.099397,135
9,0.120142,135
10,0.139721,148
11,0.1401,126
12,0.1401,120
13,0.140101,123
14,0.140101,120
15,0.141294,118
16,0.141295,118
17,0.141295,114
18,0.144909,118
19,0.160639,119
20,0.161214,152
21,0.185625,143
... etc

在@Serafeim 的回答之后，我尝试了这个：

import pandas as pd
import numpy as np
import glob
import os

path = '/root/Desktop/TT1/'
mystep = 0.4


for filename in glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    def data_splitter(df):
        max_time = df['Time'].max() # get max value of Time for the current csv file (df)
        myrange= np.arange(0, max_time, mystep) # build the threshold range
        for k in range(len(myrange)):
            # build the upper values
            temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
            #temp.to_csv("/root/Desktop/T1/xx_{}.csv".format(k))
            temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(filename, k))

data_splitter(df)

Answer 1

您只需要使用pandas 对数据框应用逻辑操作。 ✔️

在这个答案的最后，我有一个“脚本想法”来自动执行此操作，但首先让我们一步一步来：

# Load the files using pandas
import pandas as pd

df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")

# Get the desired elements based on 'Time' column
mask = df['Time'] < 0.1

# Write the new file
df_1 = df[mask] # or directly use: df_1 = df[df['Time'] < 0.1]

# save it 
df_1.to_csv("Testfile1.csv")

print(df_1)
    No.      Time  Length
0    1  0.000000     146
1    2  0.006752     116
2    3  0.019767     156
3    4  0.039635     144
4    5  0.060090     147
5    6  0.069165     138
6    7  0.079700     133
7    8  0.099397     135

#For 0.1 to 0.2 applying 2 logical conditions
df_2 = df[(df['Time'] > 0.1) & (df['Time'] < 0.2)]

---

剧本构思：

import pandas as pd
import numpy as np

mystep = 0.2 # the step e.g. 0.2, 0.4, 0.6 

#define the function
def data_splitter(df):
    max_time = df['Time'].max() # get max value of Time for the current csv file (df)
    myrange= np.arange(0, max_time, mystep) # build the threshold range
    for k in range(len(myrange)):
        # build the upper values 
        temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
        temp.to_csv("/Users/serafeim/Downloads/aaa_{}.csv".format(k))

现在，调用函数：

df = pd.read_csv("/Users/serafeim/Downloads/Testfile.csv")
data_splitter(df) # pass the df to the function and call the function

最后，您可以创建一个循环，并在data_splitter() 函数中一一传递每个df。

为了更清楚地说明函数的作用如下：

for k in range(len(myrange)):
    print myrange[k], myrange[k]+step

打印出来：

0.0 0.2
0.2 0.4
0.4 0.6000000000000001
0.6000000000000001 0.8
0.8 1.0

因此它会根据当前 .csv 文件的 Time 列的最大值自动创建下限和上限阈值。

编辑 2：

import glob, os
path = '/Volumes/'

mystep = 0.2 

for filename in glob.glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    data_splitter(df)

把所有的放在一起：

import pandas as pd
import numpy as np
import glob
import os

path = '/root/Desktop/TT1/'
mystep = 0.4

#define the function
def data_splitter(df, name):
    max_time = df['Time'].max() # get max value of Time for the current csv file (df)
    myrange= np.arange(0, max_time, mystep) # build the threshold range
    for k in range(len(myrange)):
        # build the upper values 
        temp = df[(df['Time'] >= myrange[k]) & (df['Time'] < myrange[k] + mystep)]
        temp.to_csv("/root/Desktop/T1/{}_{}.csv".format(name, k))

for filename in glob.glob(os.path.join(path, '*.csv')):
    df = pd.read_csv(filename)
    name = os.path.split(filename)[1] # get the name of the file
    data_splitter(df, name) # call the splitting function

Answer 2

假设您有 2 个目录：Source 和 Test。 Source 包含所有源 csv 文件，Test 目录将包含所有输出文件。

import os
import glob

os.chdir("/home/prasanth-8508/Downloads/Source")
for csv_file in glob.glob("*.csv"):
    contents, output_list = list(), list()
    with open(csv_file) as f:
        contents.append(f.read().replace('"', ""))

    contents = ''.join(contents).split('\n')
    header = contents[0]
    contents = contents[1:]
    op_file_counter = 1
    split_factor = float(input("Enter split factor:"))
    split_num = split_factor
    i = 0
    contents = list(filter(None, contents))

    while i < len(contents)-1:
        try:
            row = contents[i].split(",")
            if not(str(float(row[1])).startswith(str(split_num)[0:str(split_num).index(".")+2], 0, str(split_num).index(".")+2)):
                output_list.append(contents[i])
                i += 1
            else:
                if len(output_list) > 0:
                    with open("/home/prasanth-8508/Downloads/Test/file" + str(op_file_counter) + ".csv", "a+") as f:
                        f.write(header+'\n')
                        for j in output_list:
                            f.write(j+'\n')
                    op_file_counter += 1
                    output_list = list()
                split_num += split_factor
                split_num = round(split_num,1)
                print(split_num)
        except IndexError:
            break

    with open("/home/prasanth-8508/Downloads/Test/file" + str(op_file_counter) + ".csv", "a+") as f:
        f.write(header+'\n')
        for j in output_list:
            f.write(j+'\n')

    print(csv_file+" processed successfully")

运行程序后我得到了 600 多个文件，这些文件太大而无法共享。