在python csv中处理带有标题的数据[关闭]

Question

我有一个 csv 文件，第一行包含产品名称，第二行包含数据标题，从第三行开始包含每个用户状态的实际数据。

而 csv 文件如下所示：

adidas,,
USER_ID,USER_NAME
b012345,zaihan,Process
b212345,nurhanani,Check
b843432,nasirah,Call
b712345,ibrahim,Check
nike,,
USER_ID,USER_NAME
b842134,khalee,Call
h123455,shabree,Process
b777345,ibrahim,Process
b012345,zaihan,Check
b843432,nasirah,Call
b312451,nurhanani,Process

我想明智地拆分数据产品并重新排列标题和数据，如下所示：

来自这样的标题

   adidas,,
   USER_ID,USER_NAME
   b012345,zaihan,Process

这样的标题

  USER_ID,USER_NAME,adidas
  b012345,zaihan,Process

并像这样为每个产品创建DataFrame 和merge：

我已经编写代码一段时间了，我认为我必须对标题进行硬编码（例如，“adidas”和“nike”），因为我从阅读 SO 答案中了解到，我需要唯一的标题名称并且以下代码没有得到我想要的：

我的python代码是：

import csvkit
import sys
import os
from csvkit import convert

with open('/tmp/csvdata.csv', 'rb') as q:
    reader = csvkit.reader(q)
    with open('/tmp/csvdata2.csv', 'wb') as s:
        data = csvkit.writer(s)
        data.writerow(['Name', 'Userid', 'adidas', 'nike'])
        for row in reader:
            row_data = [row[0], row[1], row[2], '']
            data = csvkit.writer(s)
            data.writerow(row_data)

编辑

所以我从@piRSquared 那里得到了一个解决方案，如果产品有唯一的记录集，这是正确的，但同一产品的每个用户可能有多个状态。并且解决方案给了ValueError: Index contains duplicate entries, cannot reshape

输入 CSV 数据具有多个状态并会导致此问题的示例：

adidas,,
USER_ID,USER_NAME
b012345,zaihan,Process
h003455,shabree,Check
b212345,nurhanani,Check
b843432,nasirah,Call
b712345,ibrahim,Check
b712345,ibrahim,Process
nike,,
USER_ID,USER_NAME
b842134,khalee,Call
h123455,shabree,Process
b777345,ibrahim,Process
b012345,zaihan,Check
b843432,nasirah,Call
b312451,nurhanani,Process

我希望能达到这样的结果，貌似同一个品牌类别的用户可以有相同的id，name，同时Process和Check。

USER_ID,USER_NAME,adidas,nike
b012345,zaihan,Process
h003455,shabree,Check,Process
b212345,nurhanani,Check,Process
b843432,nasirah,Call,Call
b712345,ibrahim,Check
b712345,ibrahim,Process 
b777345,ibrahim,,Process
b842134,khalee,,Call

对于在同一品牌中同时具有 Check 和 Process 的用户（在本例中为 耐克品牌的用户 ibrahim)

Answer 1

好的，这很复杂。

解决方案

from StringIO import StringIO
import re
import pandas as pd

text = """adidas,,
USER_ID,USER_NAME
b012345,zaihan,Process
b212345,nurhanani,Check
b451234,nasirah,Call
c234567,ibrahim,Check
nike,,
USER_ID,USER_NAME
b842134,khalee,Call
h123455,shabree,Process
c234567,ibrahim,Process
c143322,zaihan,Check
b451234,nasirah,Call
"""

m = re.findall(r'(.*,,\n(.*([^,]|,[^,])\n)*)', text)

dfs = range(len(m))
keys = range(len(m))
for i, f in enumerate(m):
    lines = f[0].split('\n')
    lines[1] += ','
    keys[i] = lines[0].split(',')[0]
    dfs[i] = pd.read_csv(StringIO('\n'.join(lines[1:])))

df = pd.concat(dfs, keys=keys)
df = df.set_index(['USER_ID', 'USER_NAME'], append=True).unstack(0)

df.index = df.index.droplevel(0)
df.columns = df.columns.droplevel(0)

df = df.stack().unstack()

演示

print df.to_csv()

USER_ID,USER_NAME,adidas,nike
b012345,zaihan,Process,
b212345,nurhanani,Check,
b451234,nasirah,Call,Call
b842134,khalee,,Call
c143322,zaihan,,Check
c234567,ibrahim,Check,Process
h123455,shabree,,Process

说明

# regular expression to match line with a single value identified
# by having two commas at the end of the line.
# This grabs nike and adidas.
# It also grabs all lines after that until the next single valued line.
m = re.findall(r'(.*,,\n(.*([^,]|,[^,])\n)*)', text)

# place holder for list of sub dataframes
dfs = range(len(m))
# place holder for list of keys.  In this example this will be nike and adidas
keys = range(len(m))

# Loop through each regex match.  This example will only have 2.
for i, f in enumerate(m):
    # split on new line so I can grab and fix stuff
    lines = f[0].split('\n')
    # Fix that header row only has 2 columns and data has 3
    lines[1] += ','
    # Grab nike or adidas or other single value
    keys[i] = lines[0].split(',')[0]
    # Create dataframe by reading in rest of lines
    dfs[i] = pd.read_csv(StringIO('\n'.join(lines[1:])))

# Concat dataframes with appropriate keys and pivot stuff
df = pd.concat(dfs, keys=keys)
df = df.set_index(['USER_ID', 'USER_NAME'], append=True).unstack(0)

df.index = df.index.droplevel(0)
df.columns = df.columns.droplevel(0)

df = df.stack().unstack()

Answer 2

首先，Ctrl+C 你的示例数据并尝试在下面运行。

import pandas as pd
import numpy as np

df = pd.read_clipboard(header=None)

i = np.where(~df[0].str.contains(','))[0].astype(int).tolist()+[len(df)]

frames = []
for n in range(len(i))[:-1]:
    part = df.iloc[i[n]:i[n+1]]
    part_df = part.iloc[2:, 0].str.extract('(.+),(.+),(.+)')
    part_df.columns = ['USER_ID', 'USER_NAME', '{}'.format(part.iloc[0, 0])]
    frames.append(part_df.set_index(['USER_ID', 'USER_NAME']))

final = pd.concat(frames, axis=1).fillna('')
final.to_csv('result.csv')

结果是，

USER_ID,USER_NAME,adidas,nike
b012345,zaihan,Process,
b212345,nurhanani,Check,
b451234,nasirah,Call,
b712345,ibrahim,,Process
b842134,khalee,,Call
b843432,nasirah,,Call
c143322,zaihan,,Check
c234567,ibrahim,Check,
h123455,shabree,,Process

Answer 3

也许这会有所帮助，您可以使用 Pandas 合并您的 2 个数据集。

import pandas as pd
df1 = pd.read_csv("csvdata.csv")
df2 = pd.read_csv("csvdata2.csv")

df3 = df1.merge(df2, on='USER_ID', how='left')
df3 = df3[['USER_ID', 'USER_NAME', 'NIKE', 'ADIDAS']]

print df3

您应该更改您的数据，使其包含 Nike/Adidas 的标题，去掉 of 中的所有标题并使用 Pandas 编写标题，就像您在原始代码中所做的那样：

df1 = pd.read_csv("csvdata.csv", names = ['USER_ID', 'USER_NAME', 'NIKE'])

或

重命名标题：

USER_ID,USERNAME,NIKE

b842134,khalee,Call

h123455,shabree,Process

b712345,ibrahim,Process

c143322,zaihan,Check

b843432,nasirah,Call

编辑： 如果您的数据在一个文件中，您可以尝试将其拆分为 2 个数据框，如下所示：

index = df1.index[df1['adidas'] == 'nike'].tolist()[0]
df2 = df1[index:]
df1 = df1[:index]

这有点草率，但它应该可以工作......