如何根据来自另一个 DataFrame 的条件从 Multi-Index Dataframe 中选择一个子集

Question

我有一个如下的数据框：

                     dates         0
numbers letters               
0       a       2013-01-01  0.261092
                2013-01-02 -1.267770
                2013-01-03  0.008230
        b       2013-01-01 -1.515866
                2013-01-02  0.351942
                2013-01-03 -0.245463
        c       2013-01-01 -0.253103
                2013-01-02 -0.385411
                2013-01-03 -1.740821
1       a       2013-01-01 -0.108325
                2013-01-02 -0.212350
                2013-01-03  0.021097
        b       2013-01-01 -1.922214
                2013-01-02 -1.769003
                2013-01-03 -0.594216
        c       2013-01-01 -0.419775
                2013-01-02  1.511700
                2013-01-03  0.994332
2       a       2013-01-01 -0.020299
                2013-01-02 -0.749474
                2013-01-03 -1.478558
        b       2013-01-01 -1.357671
                2013-01-02  0.161185
                2013-01-03 -0.658246
        c       2013-01-01 -0.564796
                2013-01-02 -0.333106
                2013-01-03 -2.814611

现在我得到了一个类似的列表：

   numbers letters
0        0       b
1        1       c

我需要选择索引满足列表的数据。答案是这样的：

                     dates         0
numbers letters               
0       b       2013-01-01 -1.515866
                2013-01-02  0.351942
                2013-01-03 -0.245463
1       c       2013-01-01 -0.419775
                2013-01-02  1.511700
                2013-01-03  0.994332

如何从 MultiIndex 的 Dataframe 中选择特定的数据？

Answer 1

你也可以使用索引交集：

In [39]: l
Out[39]:
   numbers letters
0        0       b
1        1       c


In [40]: df.loc[df.index.intersection(l.set_index(['numbers','letters']).index)]
Out[40]:
                      dates         0
numbers letters
0       b        2013-01-01 -1.515866
        b        2013-01-02  0.351942
        b        2013-01-03 -0.245463
1       c        2013-01-01 -0.108325
        c        2013-01-02 -0.212350
        c        2013-01-03  0.021097
        c        2013-01-01 -0.419775
        c        2013-01-02  1.511700
        c        2013-01-03  0.994332

或more straightforward and faster solution from @Javier:

In [155]: df.loc[l.set_index(['numbers','letters']).index]
Out[155]:
                      dates         0
numbers letters
0       b        2013-01-01 -1.515866
        b        2013-01-02  0.351942
        b        2013-01-03 -0.245463
1       c        2013-01-01 -0.108325
        c        2013-01-02 -0.212350
        c        2013-01-03  0.021097
        c        2013-01-01 -0.419775
        c        2013-01-02  1.511700
        c        2013-01-03  0.994332

时间：

对于 27.000 行多索引 DF

In [156]: df = pd.concat([df.reset_index()] * 10**3, ignore_index=True).set_index(['numbers','letters'])

In [157]: df.shape
Out[157]: (27000, 2)

In [158]: %%timeit
     ...: q = l.apply(lambda r: "(numbers == {} and letters == '{}')".format(r.numbers, r.letters),
     ...:             axis=1) \
     ...:      .str.cat(sep=' or ')
     ...: df.query(q)
     ...:
10 loops, best of 3: 21.3 ms per loop

In [159]: %%timeit
     ...: df.loc[l.set_index(['numbers','letters']).index]
     ...:
10 loops, best of 3: 20.2 ms per loop

In [160]: %%timeit
     ...: df.loc[df.index.intersection(l.set_index(['numbers','letters']).index)]
     ...:
10 loops, best of 3: 27.2 ms per loop

对于 270.000 行多索引 DF

In [163]: %%timeit
     ...: q = l.apply(lambda r: "(numbers == {} and letters == '{}')".format(r.numbers, r.letters),
     ...:             axis=1) \
     ...:      .str.cat(sep=' or ')
     ...: df.query(q)
     ...:
10 loops, best of 3: 117 ms per loop

In [164]: %%timeit
     ...: df.loc[l.set_index(['numbers','letters']).index]
     ...:
1 loop, best of 3: 142 ms per loop

In [165]: %%timeit
     ...: df.loc[df.index.intersection(l.set_index(['numbers','letters']).index)]
     ...:
10 loops, best of 3: 185 ms per loop

结论： 内部使用numexpr 模块的df.query() 方法对于更大的DF 似乎更快

Answer 2

假设您有以下 DF 以及您想要获取的值：

In [28]: l
Out[28]:
   numbers letters
0        0       b
1        1       c

如果您需要选择numbers 为0 或1 且letters 在['b','c'] 中的所有行，您可以使用df.query() 方法，如下所示：

In [29]: df.query("numbers in @l.numbers and letters in @l.letters")
Out[29]:
                      dates         0
numbers letters
0       b        2013-01-01 -1.515866
        b        2013-01-02  0.351942
        b        2013-01-03 -0.245463
        c        2013-01-01 -0.253103
        c        2013-01-02 -0.385411
        c        2013-01-03 -1.740821
1       c        2013-01-01 -0.108325
        c        2013-01-02 -0.212350
        c        2013-01-03  0.021097
        b        2013-01-01 -1.922214
        b        2013-01-02 -1.769003
        b        2013-01-03 -0.594216
        c        2013-01-01 -0.419775
        c        2013-01-02  1.511700
        c        2013-01-03  0.994332

或者简单地说：

df.query("numbers in [0,1] and letters in ['b','c']")

更新：如果必须完全匹配，例如(0, 'b') 和(1, 'c')：

In [14]: q = l.apply(lambda r: "(numbers == {} and letters == '{}')".format(r.numbers, r.letters),
    ...:             axis=1) \
    ...:      .str.cat(sep=' or ')
    ...:

In [15]: q
Out[15]: "(numbers == 0 and letters == 'b') or (numbers == 1 and letters == 'c')"

In [16]: df.query(q)
Out[16]:
                      dates         0
numbers letters
0       b        2013-01-01 -1.515866
        b        2013-01-02  0.351942
        b        2013-01-03 -0.245463
1       c        2013-01-01 -0.108325
        c        2013-01-02 -0.212350
        c        2013-01-03  0.021097
        c        2013-01-01 -0.419775
        c        2013-01-02  1.511700
        c        2013-01-03  0.994332