旋转表格并添加功能答案

【问题标题】：Pivoting a table and adding features旋转表格并添加功能
【发布时间】：2020-07-07 11:02:16
【问题描述】：

我创建了以下数据框，其中给定的作业 work_id 由学生 s_id 在日期 work_date 执行。一个work_id 通常由三个测试组成，由 test_code 指定，并且在数据帧中，每个执行的测试都有一个 score 和一个相对阈值thresh。如果执行test1，也会出现注释comment1。

In [5]: import pandas as pd
   ...: import numpy as np
   ...:
   ...: df = pd.DataFrame(columns=['work_id', 'test_code', 's_id', 'score','thresh','work_date','comment1'],
   ...:                   data =[['a1','test1', 'p01',5, 1,'2020-06-15','score was 5'],
   ...:                          ['a2','test1', 'p01',10,1,'2020-06-10','score was 10'],
   ...:                          ['a2','test3', 'p01',7, 3,'2020-06-10',np.nan],
   ...:                          ['a3','test2', 'p01',6, 2,'2020-05-01',np.nan],
   ...:                          ['a4','test1', 'p02',4,1,'2020-06-20','score was 4'],
   ...:                          ['a4','test2', 'p02',5,2,'2020-06-20',np.nan],
   ...:                          ['a5','test3', 'p02',2,3,'2019-10-10',np.nan],
   ...:                          ['a6','test3', 'p02',7,3,'2020-04-01',np.nan]])
   ...: df
Out[5]:
  work_id test_code s_id  score  thresh   work_date      comment1
0      a1     test1  p01      5       1  2020-06-15   score was 5
1      a2     test1  p01     10       1  2020-06-10  score was 10
2      a2     test3  p01      7       3  2020-06-10           NaN
3      a3     test2  p01      6       2  2020-05-01           NaN
4      a4     test1  p02      4       1  2020-06-20   score was 4
5      a4     test2  p02      5       2  2020-06-20           NaN
6      a5     test3  p02      2       3  2019-10-10           NaN
7      a6     test3  p02      7       3  2020-04-01           NaN

我想旋转以work_id 作为索引的数据框。一个简单的数据透视表是这样的：

work_id s_id  score_1  score_2  score_3  thresh_1  thresh_2  thresh_3        date      comment1
0      a1  p01      5.0      NaN      NaN       1.0       NaN       NaN  2020-06-15   score was 5
1      a2  p01     10.0      NaN      7.0       1.0       NaN       3.0  2020-06-10  score was 10
2      a3  p01      NaN      6.0      NaN       NaN       2.0       NaN  2020-05-01           NaN
3      a4  p02      4.0      5.0      NaN       1.0       2.0       NaN  2020-06-20   score was 4
4      a5  p02      NaN      NaN      2.0       NaN       NaN       3.0  2019-10-10           NaN
5      a6  p02      NaN      NaN      7.0       NaN       NaN       3.0  2020-04-01           NaN

使用

df_p = pd.pivot_table(df,  index=['work_id','s_id','work_date','comment1'], columns='test_code', values=['score','thresh']).reset_index()

不包括所有缺少 cmets 的行。

此外，对于每个work_id，我都愿意

将缺失的test_code 值替换为由相同s_id,
添加一列Delta1，显示test1 相对于同一s_id 执行的先前work_id 的得分变化，以及同一@ 执行的所有测试1 中的平均Mean1 987654340@,

最终得到这个：

work_id s_id  score_1  score_2  score_3  thresh_1  thresh_2  thresh_3        date      comment1  Delta1  Mean1
0      a1  p01      5.0      6.0      7.0       1.0       2.0       3.0  2020-06-15   score was 5    -5.0    7.5
1      a2  p01     10.0      6.0      7.0       1.0       2.0       3.0  2020-06-10  score was 10     NaN    NaN
2      a3  p01      NaN      6.0      NaN       NaN       2.0       NaN  2020-05-01           NaN     NaN    NaN
3      a4  p02      4.0      5.0      7.0       1.0       2.0       3.0  2020-06-20   score was 4     NaN    4.0
4      a5  p02      NaN      NaN      2.0       NaN       NaN       3.0  2019-10-10           NaN     NaN    NaN
5      a6  p02      NaN      NaN      7.0       NaN       NaN       3.0  2020-04-01           NaN     NaN    NaN

【问题讨论】：

你能解释一下填充Delta1和Mean1列的逻辑吗？
@ShubhamSharma，Delta1 计算一个学生在 work_id 中取得的 test1 分数与同一学生在前一个 work_id 中取得的分数之间的差异。在这种情况下：学生 p01 在 2020 年 6 月 15 日执行的 work_id a1 的 test1 (score_1) 中获得 5.0，然后他在 2020 年 6 月 10 日执行的过去 work_id a2 的相同测试中获得 10.0，但有所不同-5。 Mean1 计算同一学生在所有 test1 中取得的分数的平均值，即 15/2。
那么，之前的工作id对应的是之前的日期，对吧？
@ShubhamSharma，是的。
@ShubhamSharma，是的，我相应地编辑了最终数据框。

标签： python pandas dataframe pivot pivot-table

【解决方案1】：

用途：

# step-1
df1 = df.astype({'comment1': 'str'}).set_index(
    ['work_id', 's_id', 'comment1', 'work_date', 'test_code']).unstack()
df1.columns = df1.columns.map(lambda s: '_'.join([s[0], s[1].strip('test')]))

# step-2
df1 = df1.reset_index()
df1['comment1'] = df1['comment1'].replace('nan', np.nan)
df1 = df1.groupby(['work_id', 's_id'], as_index=False).first().sort_values('work_date')

# step-3
g = df1['score_1'].groupby(df1['s_id'])
df1['Delta1'] = g.diff()
df1['Mean1'] = g.transform('mean').mask(lambda x: x.duplicated(keep='last'))

# step-4
cols = df1.columns.str.contains(r'_\d+$')
df1.loc[:, cols] = df1.loc[:, cols].groupby(df1['s_id']).ffill()
df1 = df1.sort_index()

详情：

第 1 步：使用 DataFrame.unstack 旋转数据框，并使用 map 和 join 平展 MultiLevel 列。

# step-1
                                      score_1  score_2  score_3  thresh_1  thresh_2  thresh_3
work_id s_id comment1     work_date                                                          
a1      p01  score was 5  2020-06-15      5.0      NaN      NaN       1.0       NaN       NaN
a2      p01  nan          2020-06-10      NaN      NaN      7.0       NaN       NaN       3.0
             score was 10 2020-06-10     10.0      NaN      NaN       1.0       NaN       NaN
a3      p01  nan          2020-05-01      NaN      6.0      NaN       NaN       2.0       NaN
a4      p02  nan          2020-06-20      NaN      5.0      NaN       NaN       2.0       NaN
             score was 4  2020-06-20      4.0      NaN      NaN       1.0       NaN       NaN
a5      p02  nan          2019-10-10      NaN      NaN      2.0       NaN       NaN       3.0
a6      p02  nan          2020-04-01      NaN      NaN      7.0       NaN       NaN       3.0

步骤 2：在 work_id 和 s_id 上使用 groupby 并使用聚合函数 first 并在 work_date 上对数据帧进行排序

# step-2
  work_id s_id      comment1   work_date  score_1  score_2  score_3  thresh_1  thresh_2  thresh_3
4      a5  p02           NaN  2019-10-10      NaN      NaN      2.0       NaN       NaN       3.0
5      a6  p02           NaN  2020-04-01      NaN      NaN      7.0       NaN       NaN       3.0
2      a3  p01           NaN  2020-05-01      NaN      6.0      NaN       NaN       2.0       NaN
1      a2  p01  score was 10  2020-06-10     10.0      NaN      7.0       1.0       NaN       3.0
0      a1  p01   score was 5  2020-06-15      5.0      NaN      NaN       1.0       NaN       NaN
3      a4  p02   score was 4  2020-06-20      4.0      5.0      NaN       1.0       2.0       NaN

第 3 步：通过将 score_1 分组到 s_id 并使用适当的转换函数来计算 Delta1 和 Mean1。

# step-3
      work_id s_id      comment1   work_date  score_1  score_2  score_3  thresh_1  thresh_2  thresh_3  Delta1  Mean1
    4      a5  p02           NaN  2019-10-10      NaN      NaN      2.0       NaN       NaN       3.0     NaN    NaN
    5      a6  p02           NaN  2020-04-01      NaN      NaN      7.0       NaN       NaN       3.0     NaN    NaN
    2      a3  p01           NaN  2020-05-01      NaN      6.0      NaN       NaN       2.0       NaN     NaN    NaN
    1      a2  p01  score was 10  2020-06-10     10.0      NaN      7.0       1.0       NaN       3.0     NaN    NaN
    0      a1  p01   score was 5  2020-06-15      5.0      NaN      NaN       1.0       NaN       NaN    -5.0    7.5
    3      a4  p02   score was 4  2020-06-20      4.0      5.0      NaN       1.0       2.0       NaN     NaN    4.0

第 4 步：将缺失的 test_code 值替换为之前的 work_id 中的值。

# step-4
  work_id s_id      comment1   work_date  score_1  score_2  score_3  thresh_1  thresh_2  thresh_3  Delta1  Mean1
0      a1  p01   score was 5  2020-06-15      5.0      6.0      7.0       1.0       2.0       3.0    -5.0    7.5
1      a2  p01  score was 10  2020-06-10     10.0      6.0      7.0       1.0       2.0       3.0     NaN    NaN
2      a3  p01           NaN  2020-05-01      NaN      6.0      NaN       NaN       2.0       NaN     NaN    NaN
3      a4  p02   score was 4  2020-06-20      4.0      5.0      7.0       1.0       2.0       3.0     NaN    4.0
4      a5  p02           NaN  2019-10-10      NaN      NaN      2.0       NaN       NaN       3.0     NaN    NaN
5      a6  p02           NaN  2020-04-01      NaN      NaN      7.0       NaN       NaN       3.0     NaN    NaN

【讨论】：