优化代码以提高性能并减少执行时间

Question

我有一个完美运行的代码。但是，当我运行一个大的 CSV 文件（大约 2GB）时，代码的完整执行大约需要 15-20 分钟。有没有一种方法可以优化下面的代码以减少执行时间从而提高性能？

from csv import reader, writer
import pandas as pd

path = (r"data.csv")

data = pd.read_csv(path, header=None)

last_column = data.iloc[: , -1]

arr = [i+1 for i in range(len(last_column)-1) if (last_column[i] == 1 and last_column[i+1] == 0)]

ch_0_6 = []
ch_7_14 = []
ch_16_22 = []

with open(path, 'r') as read_obj:
    csv_reader = reader(read_obj)
    rows = list(csv_reader)

for j in arr:
    
    # Channel 1-7
    ch_0_6_init = [int(rows[j][k]) for k in range(1,8)]
    bin_num = ''.join([str(x) for x in ch_0_6_init])
    dec_num = int(f'{bin_num}', 2)
    ch_0_6.append(dec_num)
    ch_0_6_init = []

    # Channel 8-15
    ch_7_14_init = [int(rows[j][k]) for k in range(8,16)]
    bin_num = ''.join([str(x) for x in ch_7_14_init])
    dec_num = int(f'{bin_num}', 2)
    ch_7_14.append(dec_num)
    ch_7_14_init = []

    # Channel 16-22
    ch_16_22_init = [int(rows[j][k]) for k in range(16,23)]
    bin_num = ''.join([str(x) for x in ch_16_22_init])
    dec_num = int(f'{bin_num}', 2)
    ch_16_22.append(dec_num)
    ch_16_22_init = []

样本数据：

0.0114,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,1,0,0,0,1
0.0112,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,1,0,0,0,0
0.0115,0,1,0,1,1,1,0,1,0,0,1,0,0,0,1,1,1,0,1,0,0,0,1
0.0117,0,1,0,1,1,1,0,1,0,0,1,0,0,0,1,1,1,0,1,0,0,0,0
0.0118,0,1,0,0,1,1,0,0,0,1,0,1,0,0,1,1,1,0,1,0,0,0,1

根据所选择的通道，将二进制数字连接起来形成一个十进制数。

Answer 1

仅使用 csv 模块，您可以尝试以下类型方法：

from csv import reader, writer

ch_0_6 = []
ch_7_14 = []
ch_16_22 = []

with open('data.csv', 'r') as f_input:
    csv_input = reader(f_input)
    last_row = ['0']

    for row in csv_input:
        if last_row[-1] == '1' and row[-1] == '0':
            ch_0_6.append(int(''.join(row[1:8]), 2))
            ch_7_14.append(int(''.join(row[8:16]), 2))
            ch_16_22.append(int(''.join(row[16:23]), 2))
            
        last_row = row
    
print(ch_0_6)
print(ch_7_14)
print(ch_16_22)

对于您的示例数据，这将显示：

[32, 46]
[1, 145]
[104, 104]

如前所述，您最初的方法是将整个文件两次读入内存。第一遍只是确定要解析哪些行。这可以在读取时通过跟踪循环中的前一行来完成。仅此一项就应该会显着加快速度。

从二进制列表元素到十进制元素的转换也效率更高一些。

这种方法也适用于更大的文件。