【问题标题】:Python numpy code optimization with enumerationPython numpy 代码优化与枚举
【发布时间】:2022-10-06 02:49:00
【问题描述】:

我一直在尝试优化这段代码,但我从来没有真正做过这样的优化,我不太确定如何开始或如何分析哪部分代码运行缓慢。我真的一直坚持这一点,并且有一段时间没有设法进一步发展。欢迎任何提示或建议。

q = np.array([0]*10 + [1] + [0]*10)
W = np.array([[0, 1], [1, 0]])
discreteA = {\'Prior\': 0.6153846153846154,
             \'Prob\': np.array([0.0125, 0., 0., 0.0125, 0.025, 0.0125, 0.025, 0.0375, 0.075, 0.1, 0.2125, 0.1375, 0.15, 0.1, 0.0875, 0.0125, 0., 0., 0., 0., 0.])}
discreteC = {\'Prior\': 0.38461538461538464,
             \'Prob\': np.array([0., 0., 0., 0.02, 0.02, 0.22, 0.46, 0.16, 0.1, 0.02, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])}

float(np.sum([np.dot(W.T[x,:], (discrete_A[\'Prob\'][i]*discrete_A[\'Prior\'], discrete_B[\'Prob\'][i]*discrete_B[\'Prior\'])) for i,x in enumerate(q)]))

我正在为qs = np.array(list(product((0,1), repeat=21))) 的 q 值运行此代码以找到最佳 q,它的数量级太慢了。

  • 如果您提供一些示例数据和所需的输出,它可能会有所帮助
  • 您不需要float。如果您将dict 替换为单独的变量,也会更容易阅读。需要滚动的代码行更难阅读。使计算简单明了。

标签: python numpy performance optimization


【解决方案1】:

我在下面附加了一些代码,对您提供的代码的 sn-p 进行了一些基本优化。它大约快 1.4 倍,不是您正在寻找的数量级,但希望它能给您一些想法。

请注意,您提供的代码不起作用,因为您使用了 undefined discrete_A discrete_B 但我认为这些应该是您上面定义的 discreteAdiscreteC

有几个原则:

  • 使用 numpy 时,如果您可以对数组执行向量操作,它通常比索引到数组并在 python 中执行计算要快。 numpy 是一个 C 库,库内的计算将受益于编译时优化。对数组执行操作也受益于处理器预测缓存(您的 CPU 期望您使用相邻的内存向前并预加载它;如果您零散地访问数据,您可能会失去这种好处)。
  • 通过缓存中间结果,避免在不需要时多次执行操作。一个很好的例子是矩阵转置,当您只需要执行一次相同的操作时,您执行了 20 次。

听起来代码中可能有一些范围调用 this 来应用这些原则。如果您经常调用此代码,则可能值得在更高级别进行缓存,并可能查看具有更大数据块的缓存操作。我不确定您的 product 方法是什么,所以我无法真正发表评论。

除此之外,熟悉 python see here 中的分析器是非常值得的。探查器是了解代码中花费时间的关键。

以下代码中的改进步骤是(以秒为单位的时间):

v1: 9.801263800123706
v2: 8.354220200097188
v3: 7.2868248000741005
v4: 7.5897450998891145
v5: 6.721231299918145

代码更改:

import timeit
import numpy as np

q = np.array([0]*10 + [1] + [0]*10)
W = np.array([[0, 1], [1, 0]])
discreteA = {'Prior': 0.6153846153846154,
             'Prob': np.array([0.0125, 0., 0., 0.0125, 0.025, 0.0125, 0.025, 0.0375, 0.075, 0.1, 0.2125, 0.1375, 0.15, 0.1, 0.0875, 0.0125, 0., 0., 0., 0., 0.])}
discreteC = {'Prior': 0.38461538461538464,
             'Prob': np.array([0., 0., 0., 0.02, 0.02, 0.22, 0.46, 0.16, 0.1, 0.02, 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])}

def v1():
    return float(np.sum([np.dot(W.T[x,:], (discreteA['Prob'][i]*discreteA['Prior'], discreteC['Prob'][i]*discreteC['Prior'])) for i,x in enumerate(q)]))

# instead of doing the Prob * Prior calculation one at a time on each loop, do it as a vector operation
# first, then index into the resultant vector. The vector multiply is faster as it happens in C without
# having to come back to python
def v2():
    prodA = discreteA['Prob'] * discreteA['Prior']
    prodC = discreteC['Prob'] * discreteC['Prior']
    return float(np.sum([np.dot(W.T[x,:], (prodA[i], prodC[i])) for i,x in enumerate(q)]))

# there are only two possible transposed matrices so don't recalulate every time
def v3():
    prodA = discreteA['Prob'] * discreteA['Prior']
    prodC = discreteC['Prob'] * discreteC['Prior']
    trans = (W.T[0,:], W.T[1,:])
    return float(np.sum([np.dot(trans[x], (prodA[i], prodC[i])) for i,x in enumerate(q)]))

# there's no need to enumerate, you can just index directly into q
def v4():
    prodA = discreteA['Prob'] * discreteA['Prior']
    prodC = discreteC['Prob'] * discreteC['Prior']
    trans = (W.T[0,:], W.T[1,:])
    return np.sum([np.dot(trans[q[i]], (prodA[i], prodC[i])) for i in range(len(q))])

# sum from a generator rather than creating a list. note np.sum(generator) is depreciated and np.sum(np.fromiter(generator)) or built-in sum is preferred
# note this changes result from 0.5153846153846154 to 0.5153846153846153 due to differences in python and numpy sum
def v5():
    prodA = discreteA['Prob'] * discreteA['Prior']
    prodC = discreteC['Prob'] * discreteC['Prior']
    trans = (W.T[0,:], W.T[1,:])
    return sum((np.dot(trans[q[i]], (prodA[i], prodC[i])) for i in range(len(q))))

if (res := v1()) == v2() and res == v3() and res == v4() and abs(res - v5()) < 0.000000000000001:
    print(f'Results match.')

print(f'v1: {timeit.timeit(v1, number=100000)}')
print(f'v2: {timeit.timeit(v2, number=100000)}')
print(f'v3: {timeit.timeit(v3, number=100000)}')
print(f'v4: {timeit.timeit(v4, number=100000)}')
print(f'v5: {timeit.timeit(v5, number=100000)}')

【讨论】:

    猜你喜欢
    • 2014-06-24
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-06-24
    • 2012-06-25
    • 1970-01-01
    • 1970-01-01
    • 2012-11-14
    相关资源
    最近更新 更多