Pandas groupby->resample 删除列答案

【问题标题】：Pandas groupby->resample deletes columnsPandas groupby->resample 删除列
【发布时间】：2019-09-24 04:30:47
【问题描述】：

我有一个包含 [key, datetime, receiver, score] 属性的 DataFrame。我想在 5 分钟内按日期和接收者重新采样数据。增量。我的方法如下。首先，我将“日期时间”设置为适当的“日期”和“时间”类型。然后我将“日期”和“接收者”分组，并使用应用函数对每个组进行重采样。

如果我删除“resample”并只返回“x”，我可以看到数据已正确分组并传递给“process”函数（见下文）。

       key             datetime receiver         score        date      time
0   9IIWNCEZD  2017-01-03 08:36:09        A -2.013896e+08  2017-01-03  08:36:09
5   ZEU7GZP47  2017-01-03 08:36:23        A -2.013668e+08  2017-01-03  08:36:23
6   ZEYSUQEI1  2017-01-03 08:36:27        A -2.013640e+08  2017-01-03  08:36:27
10  KW5FYJPIT  2017-01-03 08:36:38        A -2.013632e+08  2017-01-03  08:36:38
17  CE9RZFN5S  2017-01-03 08:36:49        A -2.013631e+08  2017-01-03  08:36:49
21  YQ7KSTNSC  2017-01-03 09:09:32        A -2.029635e+08  2017-01-03  09:09:32
         key             datetime receiver        score        date      time
1  10E1WQXUI  2017-01-03 08:36:11        B -50020185.32  2017-01-03  08:36:11
         key             datetime receiver         score        date      time
2  EHB0FM863  2017-01-03 08:36:12        C -1.008293e+08  2017-01-03  08:36:12
8  KW0UKT04Y  2017-01-03 08:36:35        C -1.007854e+08  2017-01-03  08:36:35
          key             datetime receiver        score        date      time
3   EHFLTCXJX  2017-01-03 08:36:14        D -90002925.25  2017-01-03  08:36:14
12  YD2EHEZUE  2017-01-03 08:36:39        D -90001925.25  2017-01-03  08:36:39
18  KWJ83RTOH  2017-01-03 08:36:50        D -90001725.25  2017-01-03  08:36:50
          key             datetime receiver         score        date      time
4   VHYI21ALA  2017-01-03 08:36:15        E -1.006858e+08  2017-01-03  08:36:15
9   YCXT3OAGJ  2017-01-03 08:36:36        E -1.006308e+08  2017-01-03  08:36:36
11  PUSYD2TBQ  2017-01-03 08:36:38        E -1.006268e+08  2017-01-03  08:36:38
13  3VR53M1VB  2017-01-03 08:36:40        E -1.006264e+08  2017-01-03  08:36:40
16  PV254K83I  2017-01-03 08:36:47        E -1.006258e+08  2017-01-03  08:36:47
19  3W4X8U610  2017-01-03 08:36:53        E -1.005406e+08  2017-01-03  08:36:53
20  DS1EUQNUE  2017-01-03 09:07:34        E -1.005189e+08  2017-01-03  09:07:34
25  T5ZOVXHGW  2017-01-03 10:17:53        E -1.005244e+08  2017-01-03  10:17:53
          key             datetime receiver         score        date      time
7   IRBW5Z94D  2017-01-03 08:36:31        F -1.001900e+08  2017-01-03  08:36:31
14  CE0L7Y8E0  2017-01-03 08:36:40        F -1.001320e+08  2017-01-03  08:36:40
15  YD6ZV5P8A  2017-01-03 08:36:43        F -1.001270e+08  2017-01-03  08:36:43
29  PUXJQTNW2  2017-01-03 10:28:35        F -1.012220e+08  2017-01-03  10:28:35
          key             datetime receiver         score        date      time
24  L0VF2ZUFX  2017-01-04 09:14:37        A -2.026835e+08  2017-01-04  09:14:37
30  OCTPWAQOH  2017-01-04 10:51:29        A -2.025107e+08  2017-01-04  10:51:29
          key             datetime receiver       score        date      time
23  FBJRWFDKB  2017-01-04 09:12:43        B -44649416.6  2017-01-04  09:12:43
          key             datetime receiver        score        date      time
22  JVEE0WOVC  2017-01-04 09:10:32        D -88645751.82  2017-01-04  09:10:32
          key             datetime receiver         score        date      time
28  KWA1CAK36  2017-01-04 10:28:35        E -1.005225e+08  2017-01-04  10:28:35
          key             datetime receiver         score        date      time
26  8IO0DWFDA  2017-01-04 10:22:38        F -1.012222e+08  2017-01-04  10:22:38
27  RK21L5E69  2017-01-04 10:27:46        F -1.012221e+08  2017-01-04  10:27:46

但是，如果我包括重新采样，则行为很奇怪（请参阅底部的打印输出）。看起来列被逐渐删除，直到一个空的 DataFrame 被传递给'process'函数，然后出错。我知道“应用”实际上在第一行/列上调用了两次函数，但由于我认为我没有改变任何数据，所以我不明白发生了什么。注意：我不只是在寻找解决方案，我也在尝试理解这种行为。

s = pd.DataFrame([["9IIWNCEZD","2017-01-03 08:36:09","A",-201389609],["10E1WQXUI","2017-01-03 08:36:11","B",-50020185.32],["EHB0FM863","2017-01-03 08:36:12","C",-100829267.43],["EHFLTCXJX","2017-01-03 08:36:14","D",-90002925.25],["VHYI21ALA","2017-01-03 08:36:15","E",-100685818.41],["ZEU7GZP47","2017-01-03 08:36:23","A",-201366792.15],["ZEYSUQEI1","2017-01-03 08:36:27","A",-201363981.95999998],["IRBW5Z94D","2017-01-03 08:36:31","F",-100190030.42],["KW0UKT04Y","2017-01-03 08:36:35","C",-100785367.43],["YCXT3OAGJ","2017-01-03 08:36:36","E",-100630818.41],["KW5FYJPIT","2017-01-03 08:36:38","A",-201363181.95999998],["PUSYD2TBQ","2017-01-03 08:36:38","E",-100626818.41],["YD2EHEZUE","2017-01-03 08:36:39","D",-90001925.25],["3VR53M1VB","2017-01-03 08:36:40","E",-100626418.41],["CE0L7Y8E0","2017-01-03 08:36:40","F",-100132011.16],["YD6ZV5P8A","2017-01-03 08:36:43","F",-100127011.16],["PV254K83I","2017-01-03 08:36:47","E",-100625778.41],["CE9RZFN5S","2017-01-03 08:36:49","A",-201363081.95999998],["KWJ83RTOH","2017-01-03 08:36:50","D",-90001725.25],["3W4X8U610","2017-01-03 08:36:53","E",-100540645.57],["DS1EUQNUE","2017-01-03 09:07:34","E",-100518856.89999999],["YQ7KSTNSC","2017-01-03 09:09:32","A",-202963512.17000002],["JVEE0WOVC","2017-01-03 09:10:32","D",-88645751.82],["FBJRWFDKB","2017-01-03 09:12:43","B",-44649416.6],["L0VF2ZUFX","2017-01-03 09:14:37","A",-202683512.17000002],["T5ZOVXHGW","2017-01-03 10:17:53","E",-100524437.18999998],["8IO0DWFDA","2017-01-03 10:22:38","F",-101222150.92999999],["RK21L5E69","2017-01-03 10:27:46","F",-101222144.03999999],["KWA1CAK36","2017-01-03 10:28:35","E",-100522494.62],["PUXJQTNW2","2017-01-03 10:28:35","F",-101221964.32],["OCTPWAQOH","2017-01-03 10:51:29","A",-202510655.58]],columns=["key","datetime","receiver","score"])

s["date"] = pd.to_datetime(s["datetime"]).dt.date
s["time"] = pd.to_datetime(s["datetime"]).dt.time

data_YMD = s.copy()
i=0
def process(x):
global i
if i<=6:
 print(x)
y = x.resample("5T", on="time").max()
return y

data15 = data_YMD.groupby(by=["date","receiver"]).apply(lambda x: process(x))

这会产生以下输出（来自“打印”语句）和错误：

          key             datetime receiver         score        date      time
0   9IIWNCEZD  2017-01-03 08:36:09        A -2.013896e+08  2017-01-03  08:36:09
5   ZEU7GZP47  2017-01-03 08:36:23        A -2.013668e+08  2017-01-03  08:36:23
6   ZEYSUQEI1  2017-01-03 08:36:27        A -2.013640e+08  2017-01-03  08:36:27
10  KW5FYJPIT  2017-01-03 08:36:38        A -2.013632e+08  2017-01-03  08:36:38
17  CE9RZFN5S  2017-01-03 08:36:49        A -2.013631e+08  2017-01-03  08:36:49
21  YQ7KSTNSC  2017-01-03 09:09:32        A -2.029635e+08  2017-01-03  09:09:32
          key             datetime receiver         score        date      time
0   9IIWNCEZD  2017-01-03 08:36:09        A -2.013896e+08  2017-01-03  08:36:09
5   ZEU7GZP47  2017-01-03 08:36:23        A -2.013668e+08  2017-01-03  08:36:23
6   ZEYSUQEI1  2017-01-03 08:36:27        A -2.013640e+08  2017-01-03  08:36:27
10  KW5FYJPIT  2017-01-03 08:36:38        A -2.013632e+08  2017-01-03  08:36:38
17  CE9RZFN5S  2017-01-03 08:36:49        A -2.013631e+08  2017-01-03  08:36:49
21  YQ7KSTNSC  2017-01-03 09:09:32        A -2.029635e+08  2017-01-03  09:09:32
          key             datetime         score      time
0   9IIWNCEZD  2017-01-03 08:36:09 -2.013896e+08  08:36:09
5   ZEU7GZP47  2017-01-03 08:36:23 -2.013668e+08  08:36:23
6   ZEYSUQEI1  2017-01-03 08:36:27 -2.013640e+08  08:36:27
10  KW5FYJPIT  2017-01-03 08:36:38 -2.013632e+08  08:36:38
17  CE9RZFN5S  2017-01-03 08:36:49 -2.013631e+08  08:36:49
21  YQ7KSTNSC  2017-01-03 09:09:32 -2.029635e+08  09:09:32
          key             datetime         score      time
0   9IIWNCEZD  2017-01-03 08:36:09 -2.013896e+08  08:36:09
5   ZEU7GZP47  2017-01-03 08:36:23 -2.013668e+08  08:36:23
6   ZEYSUQEI1  2017-01-03 08:36:27 -2.013640e+08  08:36:27
10  KW5FYJPIT  2017-01-03 08:36:38 -2.013632e+08  08:36:38
17  CE9RZFN5S  2017-01-03 08:36:49 -2.013631e+08  08:36:49
21  YQ7KSTNSC  2017-01-03 09:09:32 -2.029635e+08  09:09:32
Traceback (most recent call last):
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 725, in apply
    result = self._python_apply_general(f)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 742, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 237, in apply
    res = f(group)
  File "main.py", line 112, in <lambda>
    data15 = data_YMD.groupby(by=["date","receiver"]).apply(lambda x: process(x))
  File "main.py", line 109, in process
    y = x.resample("5T", on="time").max()
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/generic.py", line 8449, in resample
    level=level,
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/resample.py", line 1306, in resample
    return tg._get_resampler(obj, kind=kind)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/resample.py", line 1443, in _get_resampler
    "but got an instance of %r" % type(ax).__name__
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 112, in <module>
    data15 = data_YMD.groupby(by=["date","receiver"]).apply(lambda x: process(x))
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 737, in apply
    return self._python_apply_general(f)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 742, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 237, in apply
    res = f(group)
  File "main.py", line 112, in <lambda>
    data15 = data_YMD.groupby(by=["date","receiver"]).apply(lambda x: process(x))
  File "main.py", line 109, in process
    y = x.resample("5T", on="time").max()
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/generic.py", line 8449, in resample
    level=level,
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/resample.py", line 1306, in resample
    return tg._get_resampler(obj, kind=kind)
  File "/home/runner/.local/share/virtualenvs/python3/lib/python3.7/site-packages/pandas/core/resample.py", line 1443, in _get_resampler
    "but got an instance of %r" % type(ax).__name__
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'

后续注意事项：

1) 问题可能与“groupby”有关。如果我不调用“groupby('date','receiver')”而不是调用 groupby('date') 然后调用应用执行 groupby('receiver') 等的函数，则它可以工作，如下所示。但这似乎很笨拙，我想知道为什么它有效？

def process(x):
  def scoop(y):
      return y.set_index(pd.DatetimeIndex(y["datetime"])) \
              .resample("5T").max()
  x = x.groupby("receiver").apply(lambda y: scoop(y))
  return x

data15 = data_YMD.groupby(by=["date"]).apply(lambda x: process(x))

【问题讨论】：

我不确定您所说的“标题”行是什么意思。在 pd.DataFrame() 调用中正确定义了列，并且在传递给“进程”函数时正确形成了“组”。
抱歉，我看到了问题。将来，当事情换行时（由于数据宽），您可以改为 print(df.to_string())，这可能看起来像纯文本更混乱，但在发布时会呈现得更好。

标签： python-3.x pandas pandas-groupby

【解决方案1】：

您的代码有两个不同的问题：

resample 需要一个日期时间列，而您的 'date' 不需要。这就是您在回溯结束时收到错误TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index' 的原因。看看这个：
```
s["date"] = pd.to_datetime(s["datetime"]).dt.date
s["time"] = pd.to_datetime(s["datetime"]).dt.time
```
dt.date 和 dt.time 正在提取表示日期和时间的字符串。但是是一个字符串，而不是一个日期时间对象。所以你不能在这些列中的任何一个上resample。
groupby().apply() 的组合尝试将使用 apply 生成的结果数据帧连接到一个多索引数据帧中。但是，apply 使用的函数正在使用resample，这似乎很难重新加入数据框。如果你只修复第 1 点，那么你会得到一个像这样的KeyError：ValueError: Key 2017-01-03 00:00:00 not in level Index([2017-01-03], dtype='object', name='date')。

如何解决这一切？首先，将"datetime" 设为真正的日期时间列（如果还没有的话）：

s["datetime"] = pd.to_datetime(s["datetime"])

要解决问题2，您可以使用pandas Grouper，它提供了有用的重采样操作，因此无需使用resample。

data_YMD = s.copy()
data15 = data_YMD.groupby(by=[pd.Grouper(key="receiver"), pd.Grouper(key="datetime", freq="5T")]).max()

data15 输出，使用您的样本数据，是：

                                    key         score        date
receiver datetime                                                
A        2017-01-03 08:35:00  ZEYSUQEI1 -2.013631e+08  2017-01-03
         2017-01-03 09:05:00  YQ7KSTNSC -2.029635e+08  2017-01-03
         2017-01-03 09:10:00  L0VF2ZUFX -2.026835e+08  2017-01-03
         2017-01-03 10:50:00  OCTPWAQOH -2.025107e+08  2017-01-03
B        2017-01-03 08:35:00  10E1WQXUI -5.002019e+07  2017-01-03
         2017-01-03 09:10:00  FBJRWFDKB -4.464942e+07  2017-01-03
C        2017-01-03 08:35:00  KW0UKT04Y -1.007854e+08  2017-01-03
D        2017-01-03 08:35:00  YD2EHEZUE -9.000173e+07  2017-01-03
         2017-01-03 09:10:00  JVEE0WOVC -8.864575e+07  2017-01-03
E        2017-01-03 08:35:00  YCXT3OAGJ -1.005406e+08  2017-01-03
         2017-01-03 09:05:00  DS1EUQNUE -1.005189e+08  2017-01-03
         2017-01-03 10:15:00  T5ZOVXHGW -1.005244e+08  2017-01-03
         2017-01-03 10:25:00  KWA1CAK36 -1.005225e+08  2017-01-03
F        2017-01-03 08:35:00  YD6ZV5P8A -1.001270e+08  2017-01-03
         2017-01-03 10:20:00  8IO0DWFDA -1.012222e+08  2017-01-03
         2017-01-03 10:25:00  RK21L5E69 -1.012220e+08  2017-01-03

【讨论】：

Tnx。我没有过多关注 pd.Grouper，它比我重复的“应用”方法优雅得多。

【解决方案2】：

您得到的错误是说明您是时候对非数据时间 dtype 列重新采样了。

将 s['datetime'] 转换为 datetime dtype 并在 s['datetime'] 上重新采样，如下所示：

s["date"] = pd.to_datetime(s["datetime"]).dt.date
s["time"] = pd.to_datetime(s["datetime"]).dt.time
s['datetime'] = pd.to_datetime(s['datetime'])
data_YMD = s.copy()
i=0
def process(x):
    global i 
    if i<=6:
        print(x)
    y = x.resample("5T", on="datetime").max()
    return y

data15 = data_YMD.groupby(by=["date","receiver"]).apply(lambda x: process(x))

【讨论】：