【问题标题】:How to avoid loops for specifying limits for Pandas Dataframe filtering?如何避免指定 Pandas Dataframe 过滤限制的循环?
【发布时间】:2021-03-03 09:31:30
【问题描述】:

以下示例代码包含三个for 循环:

import numpy as np 
import pandas as pd

#Generating a sample (ndarray) of 25 particles with 3 random coordinates in the range between 0 and 3. 
#Maybe think of the particles as contained in a cube of 3 x 3 x 3 units.
sample_data = np.random.uniform(0, 3, (25,3))

#Converting the narray into a dataframe
df = pd.DataFrame(data = sample_data, columns = ['A', 'B', 'C'])
print(df)

#Generating another narray which will store the number of particles in each cell of the cube
#Each cell has dimentions 1 x 1 x 1; total cells = 27
counts_in_cells = np.empty((3, 3, 3))
counts_in_cells[:] = np.NaN

#Three nested loops to count the number of particles in each cell
for i, x_low in enumerate(np.arange(0, 3, 1)):
    for j, y_low in enumerate(np.arange(0, 3, 1)):
        for k, z_low in enumerate(np.arange(0, 3, 1)):
            
            #Specifying filtering conditions for three dimentions of the cells
            x_condition = (df['A'] >= x_low) & (df['A'] < (x_low + 1))
            y_condition = (df['B'] >= y_low) & (df['B'] < (y_low + 1))
            z_condition = (df['C'] >= z_low) & (df['C'] < (z_low + 1))
            
            #Applying the filtering conditions
            df_select = df[x_condition & y_condition & z_condition]
            
            #Counting the particles in cells (desired outcome)
            counts_in_cells[i][j][k] = len(df_select)

#Paricles in each cell 
print(counts_in_cells)

示例输入

期望的结果

快速运行

此示例代码可以立即在 Kaggle 上运行:https://www.kaggle.com/awaismirza/counting-particles-in-each-cell-of-a-cube

问题

我想避免三个循环,因为此代码的实际版本需要几分钟才能运行。 (它有 90k 个粒子和一个更大的立方体。)此外,我必须运行实际代码 6k 次,这需要很多天。

有没有办法(Pandas 功能或 NumPy 掩码之类的)我可以避免循环并更快地运行代码?

原始代码

代码的实际版本可用here 但上面的例子 代码应该足以理解问题。

【问题讨论】:

  • 请查看How to make good pandas examples,并在您的问题文本中提供示例输入和预期输出,而不是作为图像或外部链接,以制作minimal reproducible example,以便我们更好地理解如何提供帮助
  • 另外,由于您的代码有效,但您正在寻求改进方面的帮助,您可能希望改为在 Code Review 发帖
  • @G.Anderson,我已经编辑了我的问题,以便在一个小示例代码的上下文中提供示例输入和预期输出。如果需要更多信息,请告诉我。感谢您让我了解代码审查。我也会在那里发帖。

标签: python pandas performance dataframe filtering


【解决方案1】:

由于您的真实数据与此示例大相径庭,您需要进行更多处理。请对此代码持保留态度,并自行检查结果。

import numpy as np
import pandas as pd

# Reading a data file 
df_gal = pd.read_csv('massive_galaxies.csv')  # modified for my purposes

 
def density_field_calc(clus_x, clus_y, clus_z): 
    #Converting strings into floats
    clus_x = float(clus_x)
    clus_y = float(clus_y)
    clus_z = float(clus_z)

    # Filtering the input dataframe using the arguments of the function
    df_gal_selected = df_gal[(df_gal['x[kpc/h]'] >= (clus_x - 120000)) & (df_gal['x[kpc/h]'] <= (clus_x + 120000))
                            & (df_gal['y[kpc/h]'] >= (clus_y - 120000)) & (df_gal['y[kpc/h]'] <= (clus_y + 120000)) 
                             & (df_gal['z[kpc/h]'] >= (clus_z - 120000)) & (df_gal['z[kpc/h]'] <= (clus_z + 120000))]

    # copy the filtered value and normalize - subtract the lower bound so we start at 0 
    # max upper value of filtered data are just shy of 240000
    dfs = df_gal_selected.copy()
    dfs['x[kpc/h]'] -= clus_x-120000
    dfs['y[kpc/h]'] -= clus_y-120000
    dfs['z[kpc/h]'] -= clus_z-120000
    
    # now divide by 5000 (integer-div) so we get bin-numbers

    dfs['x[kpc/h]'] = dfs['x[kpc/h]'] // 5000
    dfs['y[kpc/h]'] = dfs['y[kpc/h]'] // 5000
    dfs['z[kpc/h]'] = dfs['z[kpc/h]'] // 5000
    
    # same trick as ealier, make tuples, convert tuples to running bin numbers
    dfs["cell"] = list(zip(dfs['x[kpc/h]'].astype(int), dfs['y[kpc/h]'].astype(int), dfs['z[kpc/h]'].astype(int)))
    
    lu = {(x,y,z):z*49*49 + y*49 + z for x in range(48) for y in range(48) for z in range(48)} 
    
    dfs["idx"] = dfs["cell"].map(lu)
    # print(dfs)

    # occurences by tuples grouped
    print(dfs.groupby(["cell"]).count()["idx"])

    # Creating and initiating an array containing NaN values
    counts_in_cells = np.empty((48, 48, 48))
    counts_in_cells[:] = 0

    for cell in dfs["cell"]:
        x, y, z = cell
        counts_in_cells[x, y, z] += 1

    np.set_printoptions(precision=1, suppress=True )
    print(counts_in_cells)

density_field_calc('416658.59', '455771.69', '72710.742')

输出:

# from groupby and count()
cell
(0, 6, 10)      1
(0, 6, 24)      1
(0, 6, 25)      1
(0, 6, 40)      1
(0, 8, 12)      1
               ..
(47, 44, 14)    1
(47, 44, 15)    3
(47, 45, 33)    1
(47, 45, 43)    1
(47, 47, 44)    1
Name: idx, Length: 3407, dtype: int64

# the np counted - mostly 0's
[[[0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 1. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]]

 [[0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 1.]
  [0. 0. 0. ... 0. 0. 0.]]

 [[0. 0. 0. ... 0. 0. 1.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]]

 ...

 [[0. 0. 0. ... 0. 0. 1.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]]

 [[0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 1. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]]

 [[0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  ...
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]
  [0. 0. 0. ... 0. 0. 0.]]] 

选择包含 4401 行,numpy 总和 (sum(sum(sum(counts_in_cells))) 也是 4401.0 - 所以它可能会起作用。几秒钟后就结束了。

【讨论】:

  • 非常感谢您向我推荐如此优雅的解决方案!代码现在只需不到一秒钟,而不是大约四分钟。只是一个小观察:不需要lu dict 和结果dfs['idx'] 列中的键值,因为仅在'cell' column 的基础上完成了粒子的分组和计数。
【解决方案2】:

您所做的是使用循环的普通 python 的混合, 和使用索引的熊猫。 结果确实是性能下降,因为您在每次循环迭代时都在访问和过滤 Dataframe。

据我所知,有两种方法可以继续:
1 - 删除循环并通过 apply() 使用 pandas
2 - 您使用没有 Dataframe 的循环在普通 python 中实现它。

这是您的代码,封装为函数:

import numpy as np
import pandas as pd
import time


def loop_count_hybrid(df):
    # Generating another narray which will store the number of particles in each cell of the cube
    # Each cell has dimentions 1 x 1 x 1; total cells = 27
    counts_in_cells = np.empty((3, 3, 3))
    counts_in_cells[:] = np.NaN

    # Three nested loops to count the number of particles in each cell
    for i, x_low in enumerate(np.arange(0, 3, 1)):
        for j, y_low in enumerate(np.arange(0, 3, 1)):
            for k, z_low in enumerate(np.arange(0, 3, 1)):
                # Specifying filtering conditions for three dimentions of the cells
                x_condition = (df['A'] >= x_low) & (df['A'] < (x_low + 1))
                y_condition = (df['B'] >= y_low) & (df['B'] < (y_low + 1))
                z_condition = (df['C'] >= z_low) & (df['C'] < (z_low + 1))

                # Applying the filtering conditions
                df_select = df[x_condition & y_condition & z_condition]

                # Counting the particles in cells (desired outcome)
                counts_in_cells[i][j][k] = len(df_select)
    return counts_in_cells

这里有一个用普通 python 实现的函数。

def loop_plain_python(sample_data):

    def find_discrete_position(row):
        for i, x_low in enumerate(range(0, 3, 1)):
            for j, y_low in enumerate(range(0, 3, 1)):
                for k, z_low in enumerate(range(0, 3, 1)):
                    if (x_low < row[0] < x_low + 1) \
                            and (y_low < row[1] < y_low + 1) \
                            and (z_low < row[2] < z_low + 1):
                        return i, j, k

    counts_in_cells = np.zeros([3, 3, 3])
    for row in sample_data:
        i, j, k = find_discrete_position(row)
        counts_in_cells[i][j][k] += 1
    return counts_in_cells

最后是两种方法的时间对比:

# Generating a sample (ndarray) of 25 particles with 3 random coordinates in the range between 0 and 3.
# Maybe think of the particles as contained in a cube of 3 x 3 x 3 units.
sample_data = np.random.uniform(0, 3, (25, 3))
df = pd.DataFrame(data=sample_data, columns=['A', 'B', 'C'])

start = time.time()
counts_in_cells = loop_count_hybrid(df)
elapsed1 = time.time() - start

start = time.time()
counts_in_cells = loop_plain_python(sample_data)
elapsed2 = time.time() - start

print("time of hybrid implementation", elapsed1)
print("time of plain python:", elapsed2)

输出:

time of hybrid implementation 0.05029702186584473 s
time of plain python: 0.0006160736083984375 s

显然,使用普通 python,您可以获得多达 2 个数量级! 而且我猜你也可以通过 pandas 成功获得性能...... ..但这足以解决您的问题吗?

-- 编辑:使用 apply 添加解决方案

如果你真的想使用apply,代码如下:

def find_discrete_position(row):
    for i, x_low in enumerate(range(0, 3, 1)):
        for j, y_low in enumerate(range(0, 3, 1)):
            for k, z_low in enumerate(range(0, 3, 1)):
                if (x_low < row[0] < x_low + 1) \
                        and (y_low < row[1] < y_low + 1) \
                        and (z_low < row[2] < z_low + 1):
                    return i, j, k


# sample data
sample_data = np.random.uniform(0, 3, (25, 3))
df = pd.DataFrame(data=sample_data, columns=['A', 'B', 'C'])

start = time.time()

# apply function to each row
df_pos = df[["A", "B", "C"]].apply(find_discrete_position, axis=1).\
    value_counts().reset_index().rename(columns={0: "position"})

# transfer into ndarray
counts_in_cells = np.zeros([3, 3, 3])
for i, row in df_pos.iterrows():
    counts_in_cells[row["index"][0], row["index"][1], row["index"][2]] = row["position"]

elapsed3 = time.time() - start

与其他方法的时间比较:

time of hybrid implementation 0.04173994064331055
time of plain python: 0.0004730224609375
time of apply and transfer: 0.007564067840576172

普通蟒蛇似乎赢了..

【讨论】:

  • 谢谢@pguardati!知道如何实现方法 1:'您删除循环并使用带有 apply() 的 pandas'?
  • 我将它添加到帖子中,它仍然是逐行解决方案 - 但不是使用 python 循环遍历行,而是使用 apply 循环。但是,您需要将结果从 Dataframe 移动到 ndarray 中。 (总而言之,我还是会选择普通的python)
  • 嗨@pguardati,感谢您使用Pandas 的apply() 函数实施解决方案。但是,当应用于在 48 x 48 x 48 的盒子中有 90k 个粒子的原始问题时,您的所有方法都需要大量时间:混合(433s)、普通(3297s)和应用(16138s)。这是我测试它们的笔记本:https://www.kaggle.com/awaismirza/speed-test-of-available-solutions
  • 我认为这是因为每种方法都使用了应该避免的 3 或 4 级循环。 Patrick Artner 建议的补救措施更好,因为它避免了嵌套循环并且需要 0.25 秒才能解决原始问题。
  • 没错,测试有偏差,因为我使用了一个小样本。此外,您的代码在更大的集合上运行得更快,因为它找到了 pandas 的离散位置 - 在木头下使用 numpy - 因此 ~C 级速度!
【解决方案3】:

对于这个特定问题,您需要更改框架。您想知道这是在哪个子单元格中 - 基于您获得该单元格的浮点数的整数值:

import numpy as np 
import pandas as pd
from math import floor

np.random.seed(42)

# Generating a sample (ndarray) of 25 particles with 3 random coordinates 
# in the range between 0 and 3 - inside a cube of 3 x 3 x 3 units.
sample_data = np.random.uniform(0, 3, (25,3))

# Converting the narray into a dataframe
df = pd.DataFrame(data = sample_data, columns = ['A', 'B', 'C'])

df["cell"] = list(zip(df["A"].astype(int),df["B"].astype(int),df["C"].astype(int)))

# map `(0,0,0)` to 0, `(0,0,1)` to 1, ... to get indexes
lu = {(a, b, c): (a * 9 + b * 3 + c) for a in range(3) for b in range(3) for c in range(3)}
df["idx"] = df["cell"].map(lu)

print(df)

输出:

           A         B         C       cell  idx
0   1.123620  2.852143  2.195982  (1, 2, 2)   17
1   1.795975  0.468056  0.467984  (1, 0, 0)    9
2   0.174251  2.598528  1.803345  (0, 2, 1)    7
3   2.124218  0.061753  2.909730  (2, 0, 2)   20
4   2.497328  0.637017  0.545475  (2, 0, 0)   18
5   0.550214  0.912727  1.574269  (0, 0, 1)    1
6   1.295835  0.873687  1.835559  (1, 0, 1)   10
7   0.418482  0.876434  1.099086  (0, 0, 1)    1
8   1.368210  2.355528  0.599021  (1, 2, 0)   15
9   1.542703  1.777244  0.139351  (1, 1, 0)   12
10  1.822635  0.511572  0.195155  (1, 0, 0)    9
11  2.846657  2.896896  2.425192  (2, 2, 2)   26
12  0.913841  0.293016  2.052699  (0, 0, 2)    2
13  1.320457  0.366115  1.485531  (1, 0, 1)   10
14  0.103166  2.727961  0.776340  (0, 2, 0)    6
15  1.987567  0.935133  1.560204  (1, 0, 1)   10
16  1.640131  0.554563  2.908754  (1, 0, 2)   11
17  2.325398  2.818497  2.684482  (2, 2, 2)   26
18  1.793700  2.765623  0.265478  (1, 2, 0)   15
19  0.587949  0.135682  0.975991  (0, 0, 0)    0
20  1.166032  0.814047  2.486213  (1, 0, 2)   11
21  1.070260  0.842804  1.628088  (1, 0, 1)   10
22  0.422773  2.406591  0.223652  (0, 2, 0)    6
23  2.960661  2.316734  0.596147  (2, 2, 0)   24
24  0.016566  2.446384  2.120572  (0, 2, 2)    8

根据该单元格列,您可以构建计数:

counts_in_cells = np.empty((3, 3, 3))
for cell in df["cell"]:
    x, y, z = cell
    counts_in_cells[x, y, z] += 1

np.set_printoptions(precision=1, suppress=True)
print(counts_in_cells)

输出:

[[[1. 2. 1.]
  [0. 1. 0.]
  [2. 2. 1.]]

[[2. 4. 0.]
  [0. 0. 0.]
  [1. 0. 0.]]

[[1. 2. 1.]    # 1 times index 0, 2 times index 1, 1 times index 2, 
  [0. 0. 0.]   # 0 times 3,4,5
  [1. 1. 2.]]] # 2 times index 6, 1 times index 7, 1 times index 8, etc

【讨论】:

  • 谢谢@Patrick Artner!实际上,我不能使用浮点数的整数值,即 astype(int),因为代码的实际版本的步长为 5000(不是 1),并且单元格的边界(此处为 0、1、2、3)是浮动(不是整数)。这是最初的问题:stackoverflow.com/revisions/64921467/1 对此有什么建议吗?顺便说一句,我喜欢你使用地图功能和字典lu 的方式。我现在也在考虑类似的思路来解决原来的问题。
猜你喜欢
  • 1970-01-01
  • 2019-05-22
  • 2019-05-01
  • 2017-09-15
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2021-11-25
  • 2018-07-31
相关资源
最近更新 更多