Python/ Pandas CSV 解析

Question

我使用 JotForm 可配置列表小部件来收集数据，但无法正确解析结果数据。当我使用

testdf = pd.read_csv ("TestLoad.csv")

数据作为两条记录读入，详细信息存储在“信息”列中。我理解为什么会这样解析，但我想将详细信息分解为多条记录，如下所述。

任何帮助将不胜感激。

CSV 示例

"Date","Information","Type"
"2015-12-06","First: Tom, Last: Smith, School: MCAA; First: Tammy, Last: Smith, School: MCAA;","New"
"2015-12-06","First: Jim, Last: Jones, School: MCAA; First: Jane, Last: Jones,  School: MCAA;","New" 

当前结果

Date        Information                                                                      Type
2015-12-06  First: Tom, Last: Smith, School: MCAA; First: Tammy, Last: Smith, School: MCAA;  New
2015-12-06  First: Jim, Last: Jones, School: MCAA; First: Jane, Last: Jones,  School: MCAA;  New

期望的结果

Date        First  Last   School Type
2015-12-06  Tom    Smith  MCAA   New
2015-12-06  Tammy  Smith  MCAA   New
2015-12-06  Jim    Jones  MCAA   New
2015-12-06  Jane   Jones  MCAA   New

Answer 1

这是防止答案被版主否决的无用文本。这是我使用的数据：

"Date","Information","Type"
"2015-12-07","First: Jim, Last: Jones, School: MCAA; First: Jane, Last: Jones,  School: MCAA;","Old"
"2015-12-06","First: Tom, Last: Smith, School: MCAA; First: Tammy, Last: Smith, School: MCAA;","New"

---

import pandas as pd
import numpy as np
import csv
import re
import itertools as it
import pprint
import datetime as dt

records = [] #Construct a complete record for each person

colon_pairs = r"""
    (\w+)   #Match a 'word' character, one or more times, captured in group 1, followed by..
    :       #A colon, followed by...
    \s*     #Whitespace, 0 or more times, followed by...
    (\w+)   #A 'word' character, one or more times, captured in group 2.
"""

colon_pairs_per_person = 3

with open("csv1.csv", encoding='utf-8') as f:
    next(f) #skip header line
    record = {}

    for date, info, the_type in csv.reader(f):
        info_parser = re.finditer(colon_pairs, info, flags=re.X)

        for i, match_obj in enumerate(info_parser):
            key, val = match_obj.groups()
            record[key] = val

            if (i+1) % colon_pairs_per_person == 0: #then done with info for a person
                record['Date'] = dt.datetime.strptime(date, '%Y-%m-%d') #So that you can sort the DataFrame rows by date.
                record['Type'] = the_type

                records.append(record)
                record = {}

pprint.pprint(records)
df = pd.DataFrame(
        sorted(records, key=lambda record: record['Date'])
)
print(df)
df.set_index('Date', inplace=True)
print(df)

--output:--
[{'Date': datetime.datetime(2015, 12, 7, 0, 0),
  'First': 'Jim',
  'Last': 'Jones',
  'School': 'MCAA',
  'Type': 'Old'},
 {'Date': datetime.datetime(2015, 12, 7, 0, 0),
  'First': 'Jane',
  'Last': 'Jones',
  'School': 'MCAA',
  'Type': 'Old'},
 {'Date': datetime.datetime(2015, 12, 6, 0, 0),
  'First': 'Tom',
  'Last': 'Smith',
  'School': 'MCAA',
  'Type': 'New'},
 {'Date': datetime.datetime(2015, 12, 6, 0, 0),
  'First': 'Tammy',
  'Last': 'Smith',
  'School': 'MCAA',
  'Type': 'New'}]

        Date  First   Last School Type
0 2015-12-06    Tom  Smith   MCAA  New
1 2015-12-06  Tammy  Smith   MCAA  New
2 2015-12-07    Jim  Jones   MCAA  Old
3 2015-12-07   Jane  Jones   MCAA  Old

            First   Last School Type
Date                                
2015-12-06    Tom  Smith   MCAA  New
2015-12-06  Tammy  Smith   MCAA  New
2015-12-07    Jim  Jones   MCAA  Old
2015-12-07   Jane  Jones   MCAA  Old

Answer 2

我在 python 引擎中使用了正则表达式分隔符，因此我可以指定多个分隔符。然后，我使用usecols 参数指定您希望数据框中包含的 csv 文件中的哪些列。标题不会从文件中读取，我跳过了第一行，因为它没有任何数据。我将第一组和第二组记录读入 2 个数据帧，然后将这 2 个数据帧连接起来。

a = pd.read_csv('sample.csv', sep=',|:|;', skiprows = 1, usecols = (0,2,4,6, 14), header = None, engine='python')
b = pd.read_csv('sample.csv', sep=',|:|;', skiprows = 1, usecols = (0,8,10,12,14), header = None, engine='python')
a.columns = ['Date', 'First', "Last", 'School', 'Type']
b.columns = ['Date', 'First', "Last", 'School', 'Type']
final_data = pd.concat([a,b], axis = 0)

如果您需要保留顺序，以便第二个名称出现在第一个名称的正下方，您可以使用索引进行排序。我使用归并排序，因为它是一种稳定的排序，这样可以确保第一个信息记录（右侧的记录）将位于左侧的信息记录之上。

final_data.sort_index(kind='mergesort', inplace = True)
>>>final_data
        Date        First  Last     School  Type
0   "2015-12-06"    Tom    Smith    MCAA    "New"
0   "2015-12-06"    Tammy  Smith    MCAA    "New"
1   "2015-12-06"    Jim    Jones    MCAA    "New"
1   "2015-12-06"    Jane   Jones    MCAA    "New"

编辑：将第二组记录包含到数据中。将轴更改为 0。