Pandas：获取分组和条件最后一个值答案

【问题标题】：Pandas: Get Grouped and Conditioned Last ValuePandas：获取分组和条件最后一个值
【发布时间】：2021-08-15 16:07:55
【问题描述】：

在以下数据集中（作为字典 - 使用 pd.DataFrame.from_dict）

{'customer_id': {0: 33179018, 1: 33179018, 2: 33179018, 3: 33179018, 4: 33179018, 5: 33179018, 6: 33179018, 7: 33179018, 8: 33179018, 9: 33179018, 10: 33179018, 11: 33179018, 12: 33179018, 13: 33179018, 14: 33179018, 15: 33179018, 16: 33179018, 17: 33179018, 18: 33179018, 19: 33179018, 20: 33179018, 21: 33179018, 22: 33179018, 23: 33179018, 24: 33179018, 25: 33179018, 26: 33179018, 27: 33179018, 28: 33179018, 29: 33179018}, 'binned_due_date': {0: Timestamp('2020-01-01 00:00:00'), 1: Timestamp('2020-01-01 00:00:00'), 2: Timestamp('2020-01-01 00:00:00'), 3: Timestamp('2020-01-01 00:00:00'), 4: Timestamp('2020-01-01 00:00:00'), 5: Timestamp('2020-01-01 00:00:00'), 6: Timestamp('2020-01-01 00:00:00'), 7: Timestamp('2020-01-01 00:00:00'), 8: Timestamp('2020-01-01 00:00:00'), 9: Timestamp('2020-01-01 00:00:00'), 10: Timestamp('2020-02-01 00:00:00'), 11: Timestamp('2020-02-01 00:00:00'), 12: Timestamp('2020-02-01 00:00:00'), 13: Timestamp('2020-02-01 00:00:00'), 14: Timestamp('2020-02-01 00:00:00'), 15: Timestamp('2020-02-01 00:00:00'), 16: Timestamp('2020-02-01 00:00:00'), 17: Timestamp('2020-02-01 00:00:00'), 18: Timestamp('2020-02-01 00:00:00'), 19: Timestamp('2020-02-01 00:00:00'), 20: Timestamp('2020-03-01 00:00:00'), 21: Timestamp('2020-03-01 00:00:00'), 22: Timestamp('2020-03-01 00:00:00'), 23: Timestamp('2020-03-01 00:00:00'), 24: Timestamp('2020-03-01 00:00:00'), 25: Timestamp('2020-03-01 00:00:00'), 26: Timestamp('2020-03-01 00:00:00'), 27: Timestamp('2020-03-01 00:00:00'), 28: Timestamp('2020-03-01 00:00:00'), 29: Timestamp('2020-03-01 00:00:00')}, 'days_after_due_date': {0: 0, 1: 7, 2: 15, 3: 30, 4: 45, 5: 60, 6: 90, 7: 120, 8: 150, 9: 180, 10: 0, 11: 7, 12: 15, 13: 30, 14: 45, 15: 60, 16: 90, 17: 120, 18: 150, 19: 180, 20: 0, 21: 7, 22: 15, 23: 30, 24: 45, 25: 60, 26: 90, 27: 120, 28: 150, 29: 180}, 'delinquency': {0: 0.2867237699667474, 1: 0.2111735340275364, 2: 0.1403712350605344, 3: 0.0159991170314348, 4: 0.0114093576839494, 5: 0.0114093576839494, 6: 0.0114093576839494, 7: 0.0096316009774459, 8: 0.0078538442709424, 9: 0.0078538442709424, 10: 0.0941250733801591, 11: 0.0823426447122737, 12: 0.0659927025855154, 13: 0.0415580671739743, 14: 0.0415580671739743, 15: 0.0415580671739743, 16: 0.038385027182922006, 17: 0.0352119871918695, 18: 0.0352119871918695, 19: 0.0280579446625502, 20: 0.0895907764209953, 21: 0.0854471519423207, 22: 0.0718793634721559, 23: 0.0663738935651319, 24: 0.0613301602549993, 25: 0.04804977925841, 26: 0.042425938521263, 27: 0.042425938521263, 28: 0.0355907007772587, 29: 0.0355907007772587}}

对于每个客户 (customer_id)，获取上个月 delinquency 的值 (t - 1) 其中days_after_due_date == 7。我还需要获取delinquency 2 个月前(t - 2) 的值，其中days_after_due == 30，对于我的数据集中的每一行。

customer_id          binned_due_date days_after_due_date  delinquency
0      33179018      2020-01-01                    0     0.286724
1      33179018      2020-01-01                    7     0.211174
2      33179018      2020-01-01                   15     0.140371
3      33179018      2020-01-01                   30     0.015999
4      33179018      2020-01-01                   45     0.011409
5      33179018      2020-01-01                   60     0.011409
6      33179018      2020-01-01                   90     0.011409
7      33179018      2020-01-01                  120     0.009632
8      33179018      2020-01-01                  150     0.007854
9      33179018      2020-01-01                  180     0.007854
10     33179018      2020-02-01                    0     0.094125
11     33179018      2020-02-01                    7     0.082343
12     33179018      2020-02-01                   15     0.065993
13     33179018      2020-02-01                   30     0.041558
14     33179018      2020-02-01                   45     0.041558
15     33179018      2020-02-01                   60     0.041558
16     33179018      2020-02-01                   90     0.038385
17     33179018      2020-02-01                  120     0.035212
18     33179018      2020-02-01                  150     0.035212
19     33179018      2020-02-01                  180     0.028058
20     33179018      2020-03-01                    0     0.089591
21     33179018      2020-03-01                    7     0.085447
22     33179018      2020-03-01                   15     0.071879
23     33179018      2020-03-01                   30     0.066374
24     33179018      2020-03-01                   45     0.061330
25     33179018      2020-03-01                   60     0.048050
26     33179018      2020-03-01                   90     0.042426
27     33179018      2020-03-01                  120     0.042426
28     33179018      2020-03-01                  150     0.035591
29     33179018      2020-03-01                  180     0.035591

在 SQL 中尝试了以下操作，但与我描述的方式不完全一样，对我来说最好使用 Pandas 来完成。

LAST_VALUE(IF (days_after_due_date = 7, delinquency, NULL) IGNORE NULLS) OVER (
        PARTITION BY observed.customer_id ORDER BY observed.binned_due_date, observed.days_after_due_date ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING) last_i7,

所需的输出将是（由@Umar.h 请求）：

【问题讨论】：

你能添加你的输出吗？
这应该可以为您回答大部分问题：Pandas accessing last non-null value （对于执行完整逻辑的实际代码，您应该包含您想要的结果，而不仅仅是您开始使用的数据)
对不起，刚刚更新，因为我忘了添加它@MatBailie

标签： sql pandas pandas-groupby

【解决方案1】：

方法

c = ['customer_id', 'binned_due_date']

t1 = df[df['days_after_due_date'] == 7].copy()
t1['binned_due_date'] += pd.DateOffset(months=1)

t2 = df[df['days_after_due_date'] == 30].copy()
t2['binned_due_date'] += pd.DateOffset(months=2)

df['d7 t-1']  = df.set_index(c).index.map(t1.set_index(c)['delinquency'])
df['d30 t-2'] = df.set_index(c).index.map(t2.set_index(c)['delinquency'])

说明

查询数据框以选择days_after_due_date 为7 的行，让我们将此数据框称为t1
将1 月份的日期偏移量添加到binned_due_date 列，以便我们能够将当前月份的delinquency 值映射到下个月
以类似的方式，生成另一个数据框t2，以便我们能够将当月的拖欠值映射到之后的两个月
将t1 和t2 中的delinquency 值映射到基于公共'customer_id' 和'binned_due_date' 的给定数据帧

结果

    customer_id binned_due_date  days_after_due_date  delinquency    d7 t-1   d30 t-2
0      33179018      2020-01-01                    0     0.286724       NaN       NaN
1      33179018      2020-01-01                    7     0.211174       NaN       NaN
2      33179018      2020-01-01                   15     0.140371       NaN       NaN
3      33179018      2020-01-01                   30     0.015999       NaN       NaN
4      33179018      2020-01-01                   45     0.011409       NaN       NaN
5      33179018      2020-01-01                   60     0.011409       NaN       NaN
6      33179018      2020-01-01                   90     0.011409       NaN       NaN
7      33179018      2020-01-01                  120     0.009632       NaN       NaN
8      33179018      2020-01-01                  150     0.007854       NaN       NaN
9      33179018      2020-01-01                  180     0.007854       NaN       NaN
10     33179018      2020-02-01                    0     0.094125  0.211174       NaN
11     33179018      2020-02-01                    7     0.082343  0.211174       NaN
12     33179018      2020-02-01                   15     0.065993  0.211174       NaN
13     33179018      2020-02-01                   30     0.041558  0.211174       NaN
14     33179018      2020-02-01                   45     0.041558  0.211174       NaN
15     33179018      2020-02-01                   60     0.041558  0.211174       NaN
16     33179018      2020-02-01                   90     0.038385  0.211174       NaN
17     33179018      2020-02-01                  120     0.035212  0.211174       NaN
18     33179018      2020-02-01                  150     0.035212  0.211174       NaN
19     33179018      2020-02-01                  180     0.028058  0.211174       NaN
20     33179018      2020-03-01                    0     0.089591  0.082343  0.015999
21     33179018      2020-03-01                    7     0.085447  0.082343  0.015999
22     33179018      2020-03-01                   15     0.071879  0.082343  0.015999
23     33179018      2020-03-01                   30     0.066374  0.082343  0.015999
24     33179018      2020-03-01                   45     0.061330  0.082343  0.015999
25     33179018      2020-03-01                   60     0.048050  0.082343  0.015999
26     33179018      2020-03-01                   90     0.042426  0.082343  0.015999
27     33179018      2020-03-01                  120     0.042426  0.082343  0.015999
28     33179018      2020-03-01                  150     0.035591  0.082343  0.015999
29     33179018      2020-03-01                  180     0.035591  0.082343  0.015999

【讨论】：

【解决方案2】：

# One row per customer/due_date, with columns for 7 and 30 days
df_pivot = df[df['days_after_due_date'].isin([7, 30])].pivot(index=['customer_id','binned_due_date'],columns='days_after_due_date',values='delinquency').reset_index()

# Offset by one or two rows (only within the same customer)
df_pivot['7_lag_1'] = df_pivot.groupby('customer_id')[7].shift(1)
df_pivot['30_lag_2'] = df_pivot.groupby('customer_id')[30].shift(2)

# Merge back on to original set
df.merge(df_pivot, on=['customer_id','binned_due_date'], how='left')

注意：数据必须已经按 Customer、DueDate、DaysAfterDueDate 顺序。

【讨论】：

【解决方案3】：

不确定是否有更好的方法，但这里有一个：

>>> df['d7 t-1'] = df.query('days_after_due_date == [7]')[['delinquency']].shift(1)
>>> df['d30 t-2'] = df.query('days_after_due_date == [30]')[['delinquency']].shift(1)
>>> df.ffill().fillna('-')

    customer_id binned_due_date  days_after_due_date  delinquency     d7 t-1    d30 t-2
0      33179018      2020-01-01                    0     0.286724          -          -
1      33179018      2020-01-01                    7     0.211174          -          -
2      33179018      2020-01-01                   15     0.140371          -          -
3      33179018      2020-01-01                   30     0.015999          -          -
4      33179018      2020-01-01                   45     0.011409          -          -
5      33179018      2020-01-01                   60     0.011409          -          -
6      33179018      2020-01-01                   90     0.011409          -          -
7      33179018      2020-01-01                  120     0.009632          -          -
8      33179018      2020-01-01                  150     0.007854          -          -
9      33179018      2020-01-01                  180     0.007854          -          -
10     33179018      2020-02-01                    0     0.094125          -          -
11     33179018      2020-02-01                    7     0.082343   0.211174          -
12     33179018      2020-02-01                   15     0.065993   0.211174          -
13     33179018      2020-02-01                   30     0.041558   0.211174  0.0159991
14     33179018      2020-02-01                   45     0.041558   0.211174  0.0159991
15     33179018      2020-02-01                   60     0.041558   0.211174  0.0159991
16     33179018      2020-02-01                   90     0.038385   0.211174  0.0159991
17     33179018      2020-02-01                  120     0.035212   0.211174  0.0159991
18     33179018      2020-02-01                  150     0.035212   0.211174  0.0159991
19     33179018      2020-02-01                  180     0.028058   0.211174  0.0159991
20     33179018      2020-03-01                    0     0.089591   0.211174  0.0159991
21     33179018      2020-03-01                    7     0.085447  0.0823426  0.0159991
22     33179018      2020-03-01                   15     0.071879  0.0823426  0.0159991
23     33179018      2020-03-01                   30     0.066374  0.0823426  0.0415581
24     33179018      2020-03-01                   45     0.061330  0.0823426  0.0415581
25     33179018      2020-03-01                   60     0.048050  0.0823426  0.0415581
26     33179018      2020-03-01                   90     0.042426  0.0823426  0.0415581
27     33179018      2020-03-01                  120     0.042426  0.0823426  0.0415581
28     33179018      2020-03-01                  150     0.035591  0.0823426  0.0415581
29     33179018      2020-03-01                  180     0.035591  0.0823426  0.0415581

【讨论】：

但是这种方法不适用于groupby('customer_id')
对不起，我一定是误解了...提供的解决方案与您想要的输出完全匹配。我不知道有不同的客户 ID（至少在您的示例中没有提供）。不过，您可以使用groupby 轻松地做类似的事情。现在我需要去，但稍后会提供一些东西。
这些结果显然与 OP 发布的内容不符。在 OP 中，d7_t-1 应该在第 10 行有一个值，d30 t-2 应该从第 20 行开始，而不是第 13 行。
啊，你是对的。我匆匆忙忙，因为我不得不出去把它出版。我看到现在提供了其他解决方案，所以我可能会删除我的答案。