仅 NaN 列上的 groupby 给出 IndexError答案

【问题标题】：groupby on NaN-only column gives IndexError仅 NaN 列上的 groupby 给出 IndexError
【发布时间】：2017-02-26 13:24:17
【问题描述】：

以下给出IndexError: index out of bounds：

import pandas as pd
from numpy import nan

df1 = pd.DataFrame({'Date': {0: '2016-10-11', 1: '2016-10-11', 2: '2016-10-11', 3: '2016-10-11', 4: '2016-10-11',5: '2016-10-11'}, 'Stock': {0: 'ABC', 1: 'ABC', 2: 'ABC', 3: 'ABC', 4: 'ABC', 5: 'XYZ'}, 'StartTime': {0: '08:00:00.241', 1: '08:00:00.243', 2: '12:34:23.563', 3: '08:14.05.908', 4: '18:54:50.100', 5: '10:08:36.657'}, 'EndTime': {0: nan,1: nan, 2: nan, 3: nan, 4: nan, 5: nan}})

df1.groupby(['Stock','EndTime']).head(1)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/users/.../egg_cache/p/pandas-0.16.2-py2.7-linux-x86_64.egg/pandas/core/groupby.py", line 994, in head
   in_head = self._cumcount_array() < n
File "/users/.../egg_cache/p/pandas-0.16.2-py2.7-linux-x86_64.egg/pandas/core/groupby.py", line 1034, in _cumcount_array
   arr = np.arange(self.grouper._max_groupsize, dtype='int64')
File "pandas/src/properties.pyx", line 34, in pandas.lib.cache_readonly.__get__ (pandas/lib.c:41917)
File "/users/.../egg_cache/p/pandas-0.16.2-py2.7-linux-x86_64.egg/pandas/core/groupby.py", line 1343, in _max_groupsize
   if self.indices:
File "pandas/src/properties.pyx", line 34, in pandas.lib.cache_readonly.__get__ (pandas/lib.c:41917)
File "/users/.../egg_cache/p/pandas-0.16.2-py2.7-linux-x86_64.egg/pandas/core/groupby.py", line 1309, in indices
    return _get_indices_dict(label_list, keys)
File "/users/.../egg_cache/p/pandas-0.16.2-py2.7-linux-x86_64.egg/pandas/core/groupby.py", line 3767, in _get_indices_dict
    return lib.indices_fast(sorter, group_index, keys, sorted_labels)
File "pandas/lib.pyx", line 1385, in pandas.lib.indices_fast (pandas/lib.c:23875)
File "pandas/src/util.pxd", line 41, in util.get_value_at (pandas/lib.c:62901)
IndexError: index out of bounds

但是，如果我排除所有 NaN 列，它可以正常工作，如下所示：

df1.groupby(['Stock','Date']).head(1)
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
5  2016-10-11      NaN  10:08:36.657   XYZ

任何想法，如果这是 Pandas 中的错误，或者我在这里遗漏了什么。我正在阅读以下内容：https://github.com/pandas-dev/pandas/issues/11016
如果它是一个错误，任何解决方法的建议，假设摆脱所有 Nan 列不是一个选项。

一些更有趣的观察：

df1 = pd.DataFrame({'Date': {0: '2016-10-11', 1: '2016-10-11', 2: '2016-10-11', 3: '2016-10-11', 4: '2016-10-11',5: '2016-10-11'}, 'Stock': {0: 'ABC', 1: 'ABC', 2: 'ABC', 3: 'ABC', 4: 'ABC', 5: 'XYZ'}, 'StartTime': {0: '08:00:00.241', 1: '08:00:00.243', 2: '12:34:23.563', 3: '08:14.05.908', 4: '18:54:50.100', 5: '10:08:36.657'}, 'EndTime': {0: nan,1: nan, 2: 1, 3: nan, 4: nan, 5: nan}})

print df1
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
1  2016-10-11      NaN  08:00:00.243   ABC
2  2016-10-11        1  12:34:23.563   ABC
3  2016-10-11      NaN  08:14.05.908   ABC
4  2016-10-11      NaN  18:54:50.100   ABC
5  2016-10-11      NaN  10:08:36.657   XYZ

df1.groupby(['Stock','EndTime']).head(1)
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
2  2016-10-11        1  12:34:23.563   ABC

上面的输出对我来说看起来不正确。不应该是：

         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
2  2016-10-11        1  12:34:23.563   ABC
5  2016-10-11      NaN  10:08:36.657   XYZ

现在针对以下情况：

df1 = pd.DataFrame({'Date': {0: '2016-10-11', 1: '2016-10-11', 2: '2016-10-11', 3: '2016-10-11', 4: '2016-10-11',5: '2016-10-11'}, 'Stock': {0: 'ABC', 1: 'ABC', 2: 'ABC', 3: 'ABC', 4: 'ABC', 5: 'XYZ'}, 'StartTime': {0: '08:00:00.241', 1: '08:00:00.243', 2: '12:34:23.563', 3: '08:14.05.908', 4: '18:54:50.100', 5: '10:08:36.657'}, 'EndTime': {0: nan,1: nan, 2: nan, 3: nan, 4: nan, 5: 1}})

print df1
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
1  2016-10-11      NaN  08:00:00.243   ABC
2  2016-10-11      NaN  12:34:23.563   ABC
3  2016-10-11      NaN  08:14.05.908   ABC
4  2016-10-11      NaN  18:54:50.100   ABC
5  2016-10-11        1  10:08:36.657   XYZ

df1.groupby(['Stock','EndTime']).head(1)
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
5  2016-10-11        1  10:08:36.657   XYZ

这个不错。

【问题讨论】：

df1.groupby(['Stock','EndTime']).head(1)（对于第一个 df1）对我来说很好（熊猫 0.19.0）
我的是 0.16.2。不幸的是，在企业环境中更新版本可能需要更长的时间。所以寻求解决方法。
@MaxU 您能否在使用 0.19.0 的所有三种情况下共享输出。谢谢

标签： python pandas dataframe group-by nan

【解决方案1】：

@Rahul，这是您使用 Pandas 0.19.0 时的代码输出：

In [5]: df1
Out[5]:
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
1  2016-10-11      NaN  08:00:00.243   ABC
2  2016-10-11      NaN  12:34:23.563   ABC
3  2016-10-11      NaN  08:14.05.908   ABC
4  2016-10-11      NaN  18:54:50.100   ABC
5  2016-10-11      NaN  10:08:36.657   XYZ

In [6]: df1.groupby(['Stock','EndTime']).head(1)
Out[6]:
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC

In [7]: df1.groupby(['Stock','Date']).head(1)
Out[7]:
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
5  2016-10-11      NaN  10:08:36.657   XYZ

In [8]: df1 = pd.DataFrame({'Date': {0: '2016-10-11', 1: '2016-10-11', 2: '2016-10-11', 3: '2016-10-11', 4: '2016-10-11',5: '2016-10-11'}, 'Stock': {
   ...: 0: 'ABC', 1: 'ABC', 2: 'ABC', 3: 'ABC', 4: 'ABC', 5: 'XYZ'}, 'StartTime': {0: '08:00:00.241', 1: '08:00:00.243', 2: '12:34:23.563', 3: '08:14
   ...: .05.908', 4: '18:54:50.100', 5: '10:08:36.657'}, 'EndTime': {0: nan,1: nan, 2: 1, 3: nan, 4: nan, 5: nan}})
   ...:

In [9]: df1.groupby(['Stock','EndTime']).head(1)
Out[9]:
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
2  2016-10-11      1.0  12:34:23.563   ABC

In [10]: df1 = pd.DataFrame({'Date': {0: '2016-10-11', 1: '2016-10-11', 2: '2016-10-11', 3: '2016-10-11', 4: '2016-10-11',5: '2016-10-11'}, 'Stock':
    ...: {0: 'ABC', 1: 'ABC', 2: 'ABC', 3: 'ABC', 4: 'ABC', 5: 'XYZ'}, 'StartTime': {0: '08:00:00.241', 1: '08:00:00.243', 2: '12:34:23.563', 3: '08:
    ...: 14.05.908', 4: '18:54:50.100', 5: '10:08:36.657'}, 'EndTime': {0: nan,1: nan, 2: nan, 3: nan, 4: nan, 5: 1}})
    ...:

In [11]: df1.groupby(['Stock','EndTime']).head(1)
Out[11]:
         Date  EndTime     StartTime Stock
0  2016-10-11      NaN  08:00:00.241   ABC
5  2016-10-11      1.0  10:08:36.657   XYZ

【讨论】：

谢谢@MaxU。我认为在 Out[6] 中的 Stock XYZ 应该再多一行？
@Rahul，你为什么这么认为？
同样在 Out[9]。我错过了什么吗？
groupby 应该是 'Stock' 和 EndTime 的每个唯一组合。对于Out[6]，它们是'ABC' - NaN 和'XYZ'-NaN。然后head(1) 从这些分组中获取第一行。