在 groupby 之后将组与一个数据框合并

Question

我试图通过组级合并来回答this question。下面是同一个问题的略微修改版本，但我需要通过组级合并输出。

这里是输入数据帧：

df = pd.DataFrame({ "group":[1,1,1 ,2,2],
                   "cat": ['a', 'b', 'c', 'a', 'c'] ,
                   "value": range(5),
                   "value2": np.array(range(5))* 2})

df

cat group   value value2
a   1         0   0
b   1         1    2
c   1         2    4
a   2         3    6
c   2         4    8

categories = ['a', 'b', 'c', 'd']
categories =  pd.DataFrame(['a', 'b', 'c', 'd'], columns=['cat'])
print(categories)

    cat
0   a
1   b
2   c
3   d

这是预期的输出：

cat group   value  value2
a   1         0    0
b   1         1    2
c   1         2    4
d   NA        NA   NA
a   2         3    6
c   2         4    8
b   NA        NA   NA
d   NA        NA   NA

问题：

我可以通过 for 循环实现我想要的。不过有熊猫的方法吗？

（我需要在categories和df.groupby('group')的groupby结果的每一组之间进行外连接）

grouped = df.groupby('group')

merged_list = []
for g in grouped:
    merged = pd.merge(categories, g[1], how = 'outer', on='cat')
    merged_list.append(merged)

out = pd.concat(merged_list)

Answer 1

我认为groupby + merge 这只是过于复杂的方式。

使用reindex by MultiIndex 会更快：

mux = pd.MultiIndex.from_product([df['group'].unique(), categories], names=('group','cat'))
df = df.set_index(['group','cat']).reindex(mux).swaplevel(0,1).reset_index()
#add missing values to group column
df['group'] = df['group'].mask(df['value'].isnull())
print (df)
  cat  group  value  value2
0   a    1.0    0.0     0.0
1   b    1.0    1.0     2.0
2   c    1.0    2.0     4.0
3   d    NaN    NaN     NaN
4   a    2.0    3.0     6.0
5   b    NaN    NaN     NaN
6   c    2.0    4.0     8.0
7   d    NaN    NaN     NaN

---

可能的解决方案：

df = df.groupby('group', group_keys=False)
       .apply(lambda x: pd.merge(categories, x, how = 'outer', on='cat'))
  cat  group  value  value2
0   a    1.0    0.0     0.0
1   b    1.0    1.0     2.0
2   c    1.0    2.0     4.0
3   d    NaN    NaN     NaN
0   a    2.0    3.0     6.0
1   b    NaN    NaN     NaN
2   c    2.0    4.0     8.0
3   d    NaN    NaN     NaN

时间安排：

np.random.seed(123)
N = 1000000
L = list('abcd') #235,94.1,156ms

df = pd.DataFrame({'cat': np.random.choice(L, N, p=(0.002,0.002,0.005, 0.991)),
                   'group':np.random.randint(10000,size=N),
                   'value':np.random.randint(1000,size=N),
                   'value2':np.random.randint(5000,size=N)})
df = df.sort_values(['group','cat']).drop_duplicates(['group','cat']).reset_index(drop=True)
print (df.head(10))

categories = ['a', 'b', 'c', 'd']

---

def jez1(df):
    mux = pd.MultiIndex.from_product([df['group'].unique(), categories], names=('group','cat'))
    df = df.set_index(['group','cat']).reindex(mux, fill_value=0).swaplevel(0,1).reset_index()
    df['group'] = df['group'].mask(df['value'].isnull())
    return df

def jez2(df):
    grouped = df.groupby('group')
    categories =  pd.DataFrame(['a', 'b', 'c', 'd'], columns=['cat'])
    return grouped.apply(lambda x: pd.merge(categories, x, how = 'outer', on='cat'))



def coldspeed(df):
    grouped = df.groupby('group')
    categories =  pd.DataFrame(['a', 'b', 'c', 'd'], columns=['cat'])
    return pd.concat([g[1].merge(categories, how='outer', on='cat') for g in grouped])

def akilat90(df):
    grouped = df.groupby('group')
    categories =  pd.DataFrame(['a', 'b', 'c', 'd'], columns=['cat'])
    merged_list = []

    for g in grouped:
        merged = pd.merge(categories, g[1], how = 'outer', on='cat')
        merged['group'].fillna(merged['group'].mode()[0],inplace=True) # replace the `group` column's `NA`s by mode
        merged.fillna(0, inplace=True)
        merged_list.append(merged)

    return pd.concat(merged_list)

---

In [471]: %timeit jez1(df)
100 loops, best of 3: 12 ms per loop

In [472]: %timeit jez2(df)
1 loop, best of 3: 14.5 s per loop

In [473]: %timeit coldspeed(df)
1 loop, best of 3: 19.4 s per loop

In [474]: %timeit akilat90(df)
1 loop, best of 3: 22.3 s per loop

Answer 2

实际上回答您的问题，不 - 您一次只能合并 2 个数据帧（我不知道 pandas 中的多路合并）。你无法避免循环，但你当然可以让你的代码更整洁。

pd.concat([g[1].merge(categories, how='outer', on='cat') for g in grouped])

  cat  group  value  value2
0   a    1.0    0.0     0.0
1   b    1.0    1.0     2.0
2   c    1.0    2.0     4.0
3   d    NaN    NaN     NaN
0   a    2.0    3.0     6.0
1   c    2.0    4.0     8.0
2   b    NaN    NaN     NaN
3   d    NaN    NaN     NaN