从前一行和特定列值有效地更新熊猫数据框中的 NaN

Question

我有一个pandas'DataFrame，它看起来像这样：

# Output 
#        A     B     C     D
# 0    3.0   6.0   7.0   4.0
# 1   42.0  44.0   1.0   3.0
# 2    4.0   2.0   3.0  62.0
# 3   90.0  83.0  53.0  23.0
# 4   22.0  23.0  24.0   NaN
# 5    5.0   2.0   5.0  34.0
# 6    NaN   NaN   NaN   NaN
# 7    NaN   NaN   NaN   NaN
# 8    2.0  12.0  65.0   1.0
# 9    5.0   7.0  32.0   7.0
# 10   2.0  13.0   6.0  12.0
# 11   NaN   NaN   NaN   NaN
# 12  23.0   NaN  23.0  34.0
# 13  61.0   NaN  63.0   3.0
# 14  32.0  43.0  12.0  76.0
# 15  24.0   2.0  34.0   2.0

我想做的是用最早的前一行的B 值填充 NaN。除了D 列之外，在这一行上，我希望将 NaN 替换为零。

我已经研究过 ffill，fillna.. 似乎两者都无法完成这项工作。

到目前为止我的解决方案：

def fix_abc(row, column, df):

    # If the row/column value is null/nan 
    if pd.isnull( row[column] ):

        # Get the value of row[column] from the row before
        prior = row.name
        value = df[prior-1:prior]['B'].values[0]

        # If that values empty, go to the row before that
        while pd.isnull( value ) and prior >= 1 :
            prior = prior - 1
            value = df[prior-1:prior]['B'].values[0]

    else:
        value = row[column]

    return value 

df['A'] = df.apply( lambda x: fix_abc(x,'A',df), axis=1 )
df['B'] = df.apply( lambda x: fix_abc(x,'B',df), axis=1 )
df['C'] = df.apply( lambda x: fix_abc(x,'C',df), axis=1 )


def fix_d(x):
    if pd.isnull(x['D']):
        return 0
    return x

df['D'] = df.apply( lambda x: fix_d(x), axis=1 )

感觉这样效率很低，而且很慢。所以我想知道是否有更快、更有效的方法来做到这一点。

示例输出；

#        A     B     C     D
# 0    3.0   6.0   7.0   3.0
# 1   42.0  44.0   1.0  42.0
# 2    4.0   2.0   3.0   4.0
# 3   90.0  83.0  53.0  90.0
# 4   22.0  23.0  24.0   0.0
# 5    5.0   2.0   5.0   5.0
# 6    2.0   2.0   2.0   0.0
# 7    2.0   2.0   2.0   0.0
# 8    2.0  12.0  65.0   2.0
# 9    5.0   7.0  32.0   5.0
# 10   2.0  13.0   6.0   2.0
# 11  13.0  13.0  13.0   0.0
# 12  23.0  13.0  23.0  23.0
# 13  61.0  13.0  63.0  61.0
# 14  32.0  43.0  12.0  32.0
# 15  24.0   2.0  34.0  24.0

我已将包含数据帧数据的代码转储到可用的 python 小提琴中 (here)

Answer 1

fillna 允许以多种方式进行填充。在这种情况下，D 列可以填入0。列B 可以通过pad 填写。然后A 和C 列可以从B 列填充，例如：

代码：

df['D'] = df.D.fillna(0)
df['B'] = df.B.fillna(method='pad')
df['A'] = df.A.fillna(df['B'])
df['C'] = df.C.fillna(df['B'])

测试代码：

df = pd.read_fwf(StringIO(u"""
       A     B     C     D
     3.0   6.0   7.0   4.0
    42.0  44.0   1.0   3.0
     4.0   2.0   3.0  62.0
    90.0  83.0  53.0  23.0
    22.0  23.0  24.0   NaN
     5.0   2.0   5.0  34.0
     NaN   NaN   NaN   NaN
     NaN   NaN   NaN   NaN
     2.0  12.0  65.0   1.0
     5.0   7.0  32.0   7.0
     2.0  13.0   6.0  12.0
     NaN   NaN   NaN   NaN
    23.0   NaN  23.0  34.0
    61.0   NaN  63.0   3.0
    32.0  43.0  12.0  76.0
    24.0   2.0  34.0   2.0"""), header=1)

print(df)

df['D'] = df.D.fillna(0)
df['B'] = df.B.fillna(method='pad')
df['A'] = df.A.fillna(df['B'])
df['C'] = df.C.fillna(df['B'])
print(df)

结果：

       A     B     C     D
0    3.0   6.0   7.0   4.0
1   42.0  44.0   1.0   3.0
2    4.0   2.0   3.0  62.0
3   90.0  83.0  53.0  23.0
4   22.0  23.0  24.0   NaN
5    5.0   2.0   5.0  34.0
6    NaN   NaN   NaN   NaN
7    NaN   NaN   NaN   NaN
8    2.0  12.0  65.0   1.0
9    5.0   7.0  32.0   7.0
10   2.0  13.0   6.0  12.0
11   NaN   NaN   NaN   NaN
12  23.0   NaN  23.0  34.0
13  61.0   NaN  63.0   3.0
14  32.0  43.0  12.0  76.0
15  24.0   2.0  34.0   2.0

       A     B     C     D
0    3.0   6.0   7.0   4.0
1   42.0  44.0   1.0   3.0
2    4.0   2.0   3.0  62.0
3   90.0  83.0  53.0  23.0
4   22.0  23.0  24.0   0.0
5    5.0   2.0   5.0  34.0
6    2.0   2.0   2.0   0.0
7    2.0   2.0   2.0   0.0
8    2.0  12.0  65.0   1.0
9    5.0   7.0  32.0   7.0
10   2.0  13.0   6.0  12.0
11  13.0  13.0  13.0   0.0
12  23.0  13.0  23.0  34.0
13  61.0  13.0  63.0   3.0
14  32.0  43.0  12.0  76.0
15  24.0   2.0  34.0   2.0