将数据从一个 csv 写入另一个 python

Question

我有三个具有 Product_ID、名称、成本、描述属性的 CSV 文件。每个文件都包含 Product_ID。我想将 Name (file1)、Cost(file2)、Description(File3) 与 Product_ID 和上述所有三个属性组合到新的 CSV 文件中。我需要高效的代码，因为文件包含超过 130000 行。

将所有数据合并到新文件后，我必须将该数据加载到字典中。 比如：Product_Id 作为 Key 和 Name，Cost，Description 作为 Value。

Answer 1

在创建聚合结果之前将每个输入 .csv 读入字典可能会更有效。

这是读取每个文件并将列存储在字典中的解决方案，其中 Product_IDs 作为键。我假设每个文件中都存在每个 Product_ID 值并且包含标题。我还假设除了 Product_ID 之外的文件中没有重复的列。

import csv
from collections import defaultdict

entries = defaultdict(list)
files = ['names.csv', 'costs.csv', 'descriptions.csv']
headers = ['Product_ID']

for filename in files:
   with open(filename, 'rU') as f:      # Open each file in files.
      reader = csv.reader(f)            # Create a reader to iterate csv lines
      heads = next(reader)              # Grab first line (headers)

      pk = heads.index(headers[0])      # Get the position of 'Product_ID' in
                                        # the list of headers
      # Add the rest of the headers to the list of collected columns (skip 'Product_ID')
      headers.extend([x for i,x in enumerate(heads) if i != pk])

      for row in reader:
         # For each line, add new values (except 'Product_ID') to the
         # entries dict with the line's Product_ID value as the key
         entries[row[pk]].extend([x for i,x in enumerate(row) if i != pk])

writer = csv.writer(open('result.csv', 'wb'))    # Open file to write csv lines
writer.writerow(headers)                         # Write the headers first
for key, value in entries.items():
   writer.writerow([key] + value)      # Write the product IDs
   # concatenated with the other values

Answer 2

为每个id 生成记录（可能不完整）的通用解决方案需要使用专门的数据结构，幸运的是它只是一个列表，具有预先分配的插槽数

d = {id:[name,None,None] for id, name in [line.strip().split(',') for line in open(fn1)]}
for line in open(fn2):
    id, cost = line.strip().split(',')
    if id in d:
        d[id][1] = cost
    else:
        d[id] = [None, cost, None]
for line in open(fn3):
    id, desc = line.strip().split(',')
    if id in d:
        d[id][2] = desc
    else:
        d[id] = [None, None, desc]

for id in d:
    if all(d[id]): 
       print ','.join([id]+d[id])
    else: # for this id you have not complete info,
          # so you have to decide on your own what you want, I have to
        pass

如果您确定不想进一步处理不完整的记录，可以简化上面的代码

d = {id:[name] for id, name in [line.strip().split(',') for line in open(fn1)]}
for line in open(fn2):
    id, cost = line.strip().split(',')
    if id in d: d[id].append(name)
for line in open(fn3):
    id, desc = line.strip().split(',')
    if id in d: d[id].append(desc)

for id in d:
    if len(d[id])==3: print ','.join([id]+d[id])