Python：如何最好地解析 csv 并仅计算一个子集的值

Question

我有一个 CSV 文件，其中包含 3 列和 11 行的以下内容，第一行是标题。我自己创建了这个，以便有一个简单的文件可供学习。每个订单项都是一个水果订单。

OrderNo      Fruit     Origin
1           Apple        NY
2           Orange       FL      
3           Banana       CA
4           Pear         NJ
5           Grapes       VA
6           Grapes       VA
7           Grapes       MD
8           Grapes       MA
9           Pineapple    HI
10          Grapes       GA

我正在尝试在 Python 中解析这些数据，以执行以下操作：

(1) 确定为每种水果生成最多订单的状态，(2) 确定每种水果任何单个状态的最高订单数，(3) 按字母顺序输出此结果，如下所示：

Apple NY 1
Banana CA 1
Grapes VA 2
Orange FL 1
Pear NJ 1
Pineapple HI 1

使用 csv.reader 读取 csv 文件后，我尝试使用 Counter 和 for 循环完成计数：

import csv
from collections import Counter 

cnt = Counter()
f = open("/test.csv")
reader = csv.reader(f, delimiter=",")
header = next(f) 

for row in reader:   
    cnt[row[2]] += 1 

但是有更好的方法吗？

Answer 1

我实际上会使用 pandas，它是列表/字典/电子表格/数据库的组合。它专为以这种方式处理数据而设计。

import pandas as pd
from collections import defaultdict

path_to_file = "/test.csv"
df = pd.read_csv(path_to_file)

groups = df.groupby(['Fruit', 'Origin'])
max_for_fruit = defaultdict(int) #first pass through the groups, store the maximum for each fruit to handle ties

for g in groups:
    fruit, count = g[0][0], len(g[1])
    max_for_fruit[ fruit ] = max( max_for_fruit[fruit], count )

for g in groups:
    fruit, state, count = g[0][0], g[0][1], len(g[1])
    if count == max_for_fruit[ fruit ]:
        print( "{} {} {}".format(fruit, state, count ) )

这是输出。

Apple NY 1
Banana CA 1
Grapes VA 2
Orange FL 1
Pear NJ 1
Pineapple HI 1

http://pandas.pydata.org/pandas-docs/stable/groupby.html

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.parsers.read_csv.html

http://pandas.pydata.org/pandas-docs/stable/tutorials.html

Answer 2

您的方法很好，并且效果很好，但是您需要两级嵌套（按水果和按状态计数。为了清晰起见，另一个小改进是使用命名元组：

import csv
from collections import Counter, namedtuple, defaultdict
from itertools import imap

reader = csv.reader(data)
Order = namedtuple('Order', next(reader))

state_orders = defaultdict(Counter)
for order in imap(Order._make, reader):
    state_orders[order.fruit][order.origin] += 1

for fruit, counts_by_state in sorted(state_orders.items()):
    state, cnt = counts_by_state.most_common(1)[0]
    print '%s is ordered most by %s with %s orders' % (fruit, state, cnt)

虽然 dicts 和 counters 可以轻松处理此类问题，但使用 sqlite3 模块可能会更好。 SQL 就是为了解决这些问题而诞生的：

import csv
import sqlite3

reader = csv.reader(data)
header = next(reader)

conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute('CREATE TABLE Orders (order_no integer, fruit text, origin text)')
c.executemany('INSERT INTO Orders VALUES (?,?,?)', reader)
c.execute('''CREATE VIEW StateOrders AS
             SELECT fruit, origin, COUNT(*) as cnt
             FROM Orders GROUP BY fruit, origin ''')

for fruit, state, cnt in c.execute('''
    SELECT fruit, origin, cnt
    FROM StateOrders AS Outer
    WHERE cnt = (SELECT MAX(cnt) FROM StateOrders WHERE Outer.fruit = fruit)
    ORDER BY FRUIT '''):
        print '%s is ordered most by %s with %s orders' % (fruit, state, cnt)