【问题标题】:median imputation by groups in pandas (handling group medians that are NaN)pandas 中组的中位数插补(处理组中位数为 NaN)
【发布时间】:2021-09-27 23:24:11
【问题描述】:

我有以下 DataFrame 火车:

train = {'NAME_EDUCATION_TYPE': {5: 'Secondary / secondary special',
  6: 'Higher education',
  7: 'Higher education',
  8: 'Secondary / secondary special',
  9: 'Secondary / secondary special',
  10: 'Higher education',
  11: 'Secondary / secondary special',
  12: 'Secondary / secondary special',
  13: 'Secondary / secondary special',
  14: 'Secondary / secondary special'},
 'OCCUPATION_TYPE': {5: 'Laborers',
  6: 'Accountants',
  7: 'Managers',
  8: nan,
  9: 'Laborers',
  10: 'Core staff',
  11: nan,
  12: 'Laborers',
  13: 'Drivers',
  14: 'Laborers'},
 'AGE_GROUP': {5: '45-60',
  6: '21-45',
  7: '45-60',
  8: '45-60',
  9: '21-45',
  10: '21-45',
  11: '45-60',
  12: '21-45',
  13: '21-45',
  14: '21-45'},
 'DAYS_EMPLOYED': {5: -1588.0,
  6: -3130.0,
  7: -449.0,
  8: nan,
  9: -2019.0,
  10: -679.0,
  11: nan,
  12: -2717.0,
  13: -3028.0,
  14: -203.0},
 'DAYS_EMPLOYED_ANOM': {5: False,
  6: False,
  7: False,
  8: True,
  9: False,
  10: False,
  11: True,
  12: False,
  13: False,
  14: False},
 'DAYS_LAST_PHONE_CHANGE': {5: -2536.0,
  6: -1562.0,
  7: -1070.0,
  8: 0.0,
  9: -1673.0,
  10: -844.0,
  11: -2396.0,
  12: -2370.0,
  13: -4.0,
  14: -188.0}}

我在 DAYS_EMPLOYED 列中有几个 NaN。它们在 DAYS_EMPLOYED_ANOM 列中被标记为“真”。 我想通过以下一组列使用 DAYS_EMPLOYED 的中位数来估算这些 NaN:NAME_EDUCATION_TYPE、OCCUPATION_TYPE 和 AGE_GROUP

我相信这可以在 pandas 中用几行代码完成,但我想不通。我已尝试应用以下代码,该代码在系列的均值插补示例中找到,但 NaN 值未得到插补。

fill_median = lambda g: g.fillna(g.median())
train.loc[train['DAYS_EMPLOYED_ANOM'] == True,'DAYS_EMPLOYED'] = train.groupby(['NAME_EDUCATION_TYPE', 'OCCUPATION_TYPE', 'AGE_GROUP'])['DAYS_EMPLOYED'].apply(fill_median)`

我也尝试应用这篇文章中的代码但没有成功: How can I impute values to outlier cells based on groups?

【问题讨论】:

    标签: pandas group-by imputation


    【解决方案1】:

    你可以这样做:

    train['DAYS_EMPLOYED'] =  (train.groupby(['NAME_EDUCATION_TYPE', 'OCCUPATION_TYPE', 'AGE_GROUP'],
                                             dropna=False)
                                    ['DAYS_EMPLOYED']
                                    .apply(lambda x: x.fillna(x.median()))
                              )
    

    但是,请注意,这不适用于您的特定数据集,因为您需要每组至少有一个非 NaN 值才能计算中位数。

    您可以改用人口中位数:

    train['DAYS_EMPLOYED'] =  (train.groupby(['NAME_EDUCATION_TYPE', 'OCCUPATION_TYPE', 'AGE_GROUP'],
                                             dropna=False)
                                    ['DAYS_EMPLOYED']
                                    .apply(lambda x: x.fillna(train['DAYS_EMPLOYED'].median()))
                              )
    

    这是一种尝试计算组中位数的混合方法,否则会退回到总体一:

    def median(s):
        m = s.median()
        if np.isnan(m):
            m = train['DAYS_EMPLOYED'].median()
        return m
    
    train['DAYS_EMPLOYED'] =  (train.groupby(['NAME_EDUCATION_TYPE', 'OCCUPATION_TYPE', 'AGE_GROUP'],
                                             dropna=False
                                            )
                                    ['DAYS_EMPLOYED'].apply(lambda x: x.fillna(median(s)))
                              )
    

    【讨论】:

    • 感谢 mozway !我在 OCCUPATION_TYPE 列中有太多 NaN 来使用它来计算组中位数,如果我删除它它工作正常。我一定会使用你的混合功能。
    • 嗨@mozway,我一直在尝试将您的混合方法与模式而不是中值一起应用来估算分类数据。但看起来 np.isnan 无法将组的某些模式作为输入处理,我收到以下错误:TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' 以下是 jupyter 输出中表示的不同组的模式:Stone, NaN, [] 我可以如果有帮助,请使用可重复的示例发布新帖子
    • 如果你想看看 :) new post
    猜你喜欢
    • 2021-10-27
    • 1970-01-01
    • 2016-05-17
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-05-27
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多