python-3 并行化我的函数问题答案

【问题标题】：python-3 parallelizing my functions issuepython-3 并行化我的函数问题
【发布时间】：2020-10-03 15:30:40
【问题描述】：

我正在使用 python-3.x，我想通过使用多处理并行化我的函数来加速我的代码，我应用了多处理但由于某种原因，它可能不起作用，我不确定在哪里有问题吗？

所以下面是我所做的一个小例子。任何建议表示赞赏

import numpy as np
import math
import multiprocessing as mp

lower_bound = -500
upper_bound =500 
dimension =1000
Base_Value = 10
Popula_size = 3000
MinResolution = 8

population_in = np.random.choice ( np.linspace ( lower_bound , upper_bound , Base_Value ** MinResolution ) , size = ( Popula_size , dimension ) , replace = True )
resolution = np.random.randint(1, 8, size = (1, dimension))

def Discretiz(lower_bound, upper_bound, DiscPopulation, resolution):
        
    pop_size = int(len(DiscPopulation))
    the_new_population = np.zeros ((pop_size, dimension))
    for i in range (pop_size) :
        for ii in range (dimension):          
            decimal = int(np.round((DiscPopulation[i][ii] - lower_bound) / ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1))))
            the_new_population[i, ii]  = (lower_bound + decimal *  ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1)))
    return the_new_population


# without_parallelizing
# the_new_population = Discretiz(lower_bound, upper_bound, population_in, resolution)


# wit_parallelizing
pool = mp.Pool(mp.cpu_count())
the_new_population = [pool.apply(Discretiz, args=(lower_bound, upper_bound, population_in, resolution))]


print (the_new_population)

【问题讨论】：

当numpy 可以在整个数组上做同样的事情时，你为什么还要使用math 函数。 math.power 需要标量输入，迫使您使用慢速双 for 循环。在加入多处理潮流之前，您应该最大限度地利用 numpy。
@hpaulj 感谢您的帮助，但我如何将 numpy 的力量应用到我的工作中，例如数学力量 math.pow(Base_Value,resolution[:,ii])-1)
您能否也添加什么不起作用以及您是如何确定的？
@atru 你能看看我的回答吗
我仔细查看了您的代码：您需要自己指定要并行化什么以及如何并行化。它可能像并行 map 和共享数组一样简单，但我需要刷新更多细节。

标签： python-3.x numpy parallel-processing python-multiprocessing

【解决方案1】：

与：

population_in = np.random.choice ( np.linspace ( lower_bound , upper_bound , Base_Value ** MinResolution ) , size = ( Popula_size , dimension ) , replace = True )

您制作了一个二维数组(Popula_size, dimension) 形状。这是作为DiscPopulation 传递的。

resolution = np.random.randint(1, 8, size = (1, dimension))

双迭代函数可以替换为对整个数组进行操作而没有缓慢迭代的函数：

def Discretiz(lower_bound, upper_bound, DiscPopulation, resolution):
    pop_size = DiscPopulation[0]  # no need for the 'int'
    num = DiscPopulation - lower_bound
    divisor = (upper_bound-lower_bound)/(Base_value**resolution-1)
    decimal = num/divisor
    # this divide does (pop,dimension)/(1,dimension); ok by broadcasting)
    decimal = np.round(decimal)  # no need for int
    the_new_population = lower_bound + decimal * divisor
    return the_new_population

我在这里就地写了这个。它在语法上是正确的，但我没有尝试运行它。

【讨论】：

这对我来说是最好的答案，你知道吗，我改变了我的大部分代码，因为大部分 for loops 都被删除了。编程关乎我们的思维方式。再次感谢你，我从中学到了很多:)

【解决方案2】：

我现在修复了代码，但仍然没有比旧代码快，它需要更多时间不知道为什么？没有并行化：25.831339597702026 秒并行化：44.12706518173218 秒？？？！！！

import numpy as np
import math
import multiprocessing as mp
import time

from multiprocessing import Process, Value, Array, Manager, Pool, cpu_count
import time

lower_bound = -500
upper_bound =500 
dimension =1000
Base_Value = 10
Popula_size = 2000
MinResolution = 8

population_in = np.random.choice ( np.linspace ( lower_bound , upper_bound , Base_Value ** MinResolution ) , size = ( Popula_size , dimension ) , replace = True )
resolution = np.random.randint(1, 8, size = (1, dimension))

start_time = time.time()
def Discretiz1(DiscPopulation, resolution):
# def Discretiz1(DiscPopulation):
    DiscPopulation = np.reshape(DiscPopulation, (Popula_size, dimension))
    resolution = np.reshape(resolution, (1,dimension))
    the_new_population = np.zeros ((Popula_size, dimension))
    for i in range (Popula_size) :
        for ii in range (dimension):          
            decimal = int(np.round((DiscPopulation[i][ii] - lower_bound) / ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1))))
            the_new_population[i, ii]  = (lower_bound + decimal *  ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1)))
    # print(the_new_population)

if __name__ == '__main__':

    num_cores = cpu_count()
    Pool(processes=num_cores)
    population_in = np.reshape(population_in, (1,Popula_size * dimension))[0]
    resolution = np.reshape(resolution, (1,dimension))[0]
    arr1 = Array('d', population_in)
    arr2 = Array('i', resolution)
    
    start_time = time.time()
    p = Process(target=Discretiz1, args=(arr1, arr2))
    p.start()
    p.join()
    print('--- %s seconds ---'%(time.time() - start_time))
    
print("--- %s seconds ---3" % (time.time() - start_time))

这是旧的或没有并行化的：

import numpy as np
import math
import multiprocessing as mp
import time

from multiprocessing import Process, Value, Array, Manager, Pool, cpu_count
import time

lower_bound = -500
upper_bound =500 
dimension =1000
Base_Value = 10
Popula_size = 2000
MinResolution = 8

population_in = np.random.choice ( np.linspace ( lower_bound , upper_bound , Base_Value ** MinResolution ) , size = ( Popula_size , dimension ) , replace = True )
resolution = np.random.randint(1, 8, size = (1, dimension))

start_time = time.time()
def Discretiz(lower_bound, upper_bound, DiscPopulation, resolution):
        
    pop_size = int(len(DiscPopulation))
    the_new_population = np.zeros ((pop_size, dimension))
    for i in range (pop_size) :
        for ii in range (dimension):          
            decimal = int(np.round((DiscPopulation[i][ii] - lower_bound) / ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1))))
            the_new_population[i, ii]  = (lower_bound + decimal *  ((upper_bound-lower_bound)/(math.pow(Base_Value,resolution[:,ii])-1)))
    return the_new_population

# without_parallelizing
the_new_population = Discretiz(lower_bound, upper_bound, population_in, resolution)

print("--- %s seconds ---" % (time.time() - start_time))

【讨论】：