使用多列解析 CSV 文件并聚合值答案

【问题标题】：Parse CSV file with and aggregate values, multiple columns使用多列解析 CSV 文件并聚合值
【发布时间】：2013-07-02 16:12:51
【问题描述】：

我想修改这里的帖子 (Parse CSV file and aggregate the values) 来汇总多列而不是一列。

所以对于这些数据：

CITY,AMOUNT,AMOUNT2,AMOUNTn
London,20,21,22
Tokyo,45,46,47
London,55,56,57
New York,25,26,27

我怎样才能得到这个：

CITY,AMOUNT,AMOUNT2,AMOUNTn
London,75,77,79
Tokyo,45,46,47
New York,25,26,27

我最终会有几千列，不幸的是我不能使用 pandas 包来完成这个任务。这是我刚刚将所有三个 AMOUNT cols 聚合为一个的代码，这不是我想要的

from __future__ import division
import csv
from collections import defaultdict

def default_factory():
    return [0, None, None, 0]

reader = csv.DictReader(open('test_in.txt'))
cities = defaultdict(default_factory)
for row in reader:
    headers = [r for r in row.keys()]
    headers.remove('CITY')
    for i in headers:
        amount = int(row[i])
        cities[row["CITY"]][0] += amount
        max = cities[row["CITY"]][1]
        cities[row["CITY"]][1] = amount if max is None else amount if amount > max else max
        min = cities[row["CITY"]][2]
        cities[row["CITY"]][2] = amount if min is None else amount if amount < min else min
        cities[row["CITY"]][3] += 1


for city in cities:
    cities[city][3] = cities[city][0]/cities[city][3] # calculate mean

with open('test_out.txt', 'wb') as myfile:
    writer = csv.writer(myfile, delimiter="\t")
    writer.writerow(["CITY", "AMOUNT", "AMOUNT2", "AMOUNTn ,"max", "min", "mean"])
    writer.writerows([city] + cities[city] for city in cities)

感谢您的帮助

【问题讨论】：

标签： python csv dictionary aggregate

【解决方案1】：

这是使用itertools.groupby 的一种方式。

import StringIO
import csv
import itertools

data = """CITY,AMOUNT,AMOUNT2,AMOUNTn
London,20,21,22
Tokyo,45,46,47
London,55,56,57
New York,25,26,27"""

# I use StringIO to create a file like object for demo purposes
f = StringIO.StringIO(data) 
fieldnames = f.readline().strip().split(',')
key = lambda x: x[0] # the first column will be a grouping key
# rows must be sorted by city before passing to itertools.groupby
rows_sorted = sorted(csv.reader(f), key=key)
outfile = StringIO.StringIO('')
writer = csv.DictWriter(outfile, fieldnames=fieldnames, lineterminator='\n')
writer.writeheader()
for city, rows in itertools.groupby(rows_sorted, key=key):
    # remove city column for aggregation, convert to ints
    rows = [[int(x) for x in row[1:]] for row in rows] 
    agg = [sum(column) for column in zip(*rows)]
    writer.writerow(dict(zip(fieldnames, [city] + agg)))

print outfile.getvalue()

# CITY,AMOUNT,AMOUNT2,AMOUNTn
# London,75,77,79
# New York,25,26,27
# Tokyo,45,46,47

【讨论】：

这是一个有趣的方法，不幸的是我对 itertools 不是很熟悉。是否可以在每列中包含基本的汇总统计信息，例如平均值、最大值和最小值？
您可以在 agg 行更改统计信息。将 sum 替换为 max、min 或您的平均函数。
这些方法看起来很简单，感谢您指出这一点。当我进行以下更改时，您知道为什么我的输出文件是空白的吗？ outfile = open('C:/path/to/file.txt' 'wb')
最后忘记关闭了吗？

【解决方案2】：

我会这样做。

import csv
from StringIO import StringIO

data = '''CITY,AMOUNT,AMOUNT2,AMOUNTn
London,20,21,22
Tokyo,45,46,47
London,55,56,57,99
New York,25,26,27'''

file_ = StringIO(data)

reader = csv.reader(file_)
headers = next(reader)

rows = {}

def add(col1, col2):
    l = len(col1)
    for i, n in enumerate(col2):
        if i >= l:
            col1.extend(col2[i:])
            break
        col1[i] += n
    return col1

for row in reader:
    key = row[0]
    nums = map(int, row[1:])
    if key in rows:
        rows[key] = add(rows[key], nums)
    else:
        rows[key] = map(int, nums)

【讨论】：