【问题标题】:mode imputation by groups in pandas (handling group modes that are NaN)pandas 中组的模式插补(处理 NaN 的组模式)
【发布时间】:2021-10-27 14:35:27
【问题描述】:

我有一个包含 NaN 的分类列“WALLSMATERIAL_MODE”,我想使用以下组 ['NAME_EDUCATION_TYPE', 'AGE_GROUP'] 的模式进行估算:

    NAME_EDUCATION_TYPE             AGE_GROUP   WALLSMATERIAL_MODE
20  Secondary / secondary special   45-60       Stone, brick
21  Secondary / secondary special   21-45       NaN
22  Secondary / secondary special   21-45       Panel
23  Secondary / secondary special   60-70       Mixed
24  Secondary / secondary special   21-45       Panel
25  Secondary / secondary special   45-60       Stone, brick
26  Secondary / secondary special   45-60       Wooden
27  Secondary / secondary special   21-45       NaN
28  Higher education                21-45       NaN
29  Higher education                21-45       Panel

重现性代码

df = pd.DataFrame({'NAME_EDUCATION_TYPE': {20: 'Secondary / secondary special',
  21: 'Secondary / secondary special',
  22: 'Secondary / secondary special',
  23: 'Secondary / secondary special',
  24: 'Secondary / secondary special',
  25: 'Secondary / secondary special',
  26: 'Secondary / secondary special',
  27: 'Secondary / secondary special',
  28: 'Higher education',
  29: 'Higher education'},
 'AGE_GROUP': {20: '45-60',
  21: '21-45',
  22: '21-45',
  23: '60-70',
  24: '21-45',
  25: '45-60',
  26: '45-60',
  27: '21-45',
  28: '21-45',
  29: '21-45'},
 'WALLSMATERIAL_MODE': {20: 'Stone, brick',
  21: np.nan,
  22: 'Panel',
  23: 'Mixed',
  24: 'Panel',
  25: 'Stone, brick',
  26: 'Wooden',
  27: np.nan,
  28: np.nan,
  29: 'Panel'}})

我尝试从 post 调整以下函数,该函数适用于中值插补并处理为 NaN 的组中值

在:

def mode(s):
    if pd.isnull(s.mode()):
        return df['WALLSMATERIAL_MODE'].mode()
    return s.mode()
        
df['WALLSMATERIAL_MODE'] = df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']], dropna=False).apply(lambda x: x.fillna(mode(x)))

OUT:调用 pd.isnull 时出现以下错误

The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

我不明白,我尝试将 pd.isnull 应用于所有组模式,但它不会引发此错误。请参阅下面的组模式

在:

df['WALLSMATERIAL_MODE'].groupby([df['NAME_EDUCATION_TYPE'], df['AGE_GROUP']]).agg(pd.Series.mode).to_dict()

输出:

{('Higher education', '60-70'): nan,
 ('Higher education', '45-60'): nan,
 ('Higher education', '21-45'): 'Panel',
 ('Higher education', '0-21'): nan,
 ('Secondary / secondary special', '60-70'): 'Mixed',
 ('Secondary / secondary special', '45-60'): 'Stone, brick',
 ('Secondary / secondary special', '21-45'): 'Panel',
 ('Secondary / secondary special', '0-21'): nan}

如果有人能说出错误在哪里,或者是否有有效的方法来按组估算此列,我将不胜感激!

【问题讨论】:

  • 使用median(提供的链接)和使用mode 的帖子之间的主要区别在于mode 返回Series 而median 返回单个值。因此,执行if ...s.mode() 返回将if 与系列一起使用的公共值错误。每次使用 mode 时,您的函数中都会有一个快速修复,使用 ...mode().iloc[0]
  • 感谢 Ben,我尝试使用 pd.isnull(s.mode().iloc[0]),但收到以下错误 IndexError: single positional indexer is out-of-bounds。

标签: pandas pandas-groupby nan categorical-data imputation


【解决方案1】:

下面的代码似乎可以使用 try except 来解决问题。我宁愿避免使用 try except,但我想不出更简洁的方法。

def mode_cats(s):
        try:
            if pd.isnull(s.mode().any()): # check if the mode of the subgroup is NaN or contains NaN 
                                          # (mode() may indeed return a list of several modes)
                m = app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0] # returns the mode of the column
            else:
                m = s.mode().iloc[0]  # returns the mode of the subgroup
            return m
        except IndexError: # mode returns an empty series if the subgroup consists of a single NaN value
                           # this causes s.mode().iloc[0] to raise an index error
            return app_train_dash['WALLSMATERIAL_MODE'].mode().iloc[0]

正如@Ben.T 指出的那样,我必须将.iloc[0] 与.mode() 一起使用 但是当.mode().iloc[0] 有一个空数组作为输入时,我得到IndexError: single positional indexer is out-of-bounds。 错误回溯:

  1. mode() 在 value = NaN 的一行的子组上调用。 .mode() 为单个 NaN 的这个子组返回一个空数组
  2. 对传入的空数组调用pd.isnull,返回一个空数组
  3. 对空数组调用 .iloc[0] 会引发索引错误

【讨论】:

    猜你喜欢
    • 2021-09-27
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2021-07-11
    • 2017-11-25
    • 2017-05-27
    • 2017-04-04
    相关资源
    最近更新 更多