Python循环从csv导入中删除重复项+原始项？

Question

所以我有一个要导入的 csv 文件，并希望根据第一列中的用户编号跳过从 csv 文件中导入重复行和原始行，并且我正在使用 StringIO 模块。我目前这样做的方式低于这是不正确的，因为即使它跳过了重复的行，它仍然会导入我相信的原始行。跳过从 csv 导入重复行和原始行的最佳方法是什么？

    def csv_import(stream):

        ostream = StringIO()
        headers = stream.readline()
        ostream.write(headers)

        seen_user_numbers = {}

        for row in stream:
            list_row = row.split(',')
            user_number = list_row[0]

            if user_number in seen_user_numbers:
                seen_user_numbers.pop(user_number)
                continue

            seen_user_numbers[user_number] = True
            ostream.write(row)

        ostream.seek(0)
        return ostream
   

Answer 1

因为在到达输入文件末尾之前您无法确定是否包含行，因此您需要先将所有未排除的行存储在内存中，然后才能将它们写入文件。

你可以用字典来做到这一点：

def csv_import(stream):

    ostream = StringIO()
    headers = stream.readline()
    ostream.write(headers)

    outputLines = dict()  # will use None for lines to exclude

    for row in stream:
        list_row = row.split(',')
        user_number = list_row[0]
        
        if user_number in outputLines:
            outputLines[user_number] = None
        else:
            outputLines[user_number] = row
        
    for row in filter(None,outputLines.values()):
        ostream.write(row)

    ostream.seek(0)
    return ostream