python中类中的多处理池将数组更改为列表答案

【问题标题】：Multiprocessing pool inside a class in python changing array to listpython中类中的多处理池将数组更改为列表
【发布时间】：2021-07-18 02:26:44
【问题描述】：

我正在尝试在类中的 python 3 中使用 multiprocessing.pool。这是原来的功能：

stress_matrix, compliance = self.problem.compute_objs(self.xphys)
avg_sm = np.zeros(self.nel)
for i2 in range(self.nel):
    avg_sm = avg_stress_calc(self.xphys, self.nelx, self.nely,
                            self.rmin, stress_matrix, avg_sm, i2)

这给我留下了一个形状为 (16,) 的数组，是这样的：

array([0.81814754, 0.64561319, 0.62517261, 0.78422925, 0.6962134 ,
   0.65993462, 0.63970099, 0.68776093, 0.49890513, 0.60900864,
   0.71575952, 0.73120825, 0.32964378, 0.53196899, 0.80481781,
   0.99930964])

我试图通过使用多处理池来加快速度，因为我的 NEL 大于 10,000（通常），如下所示：

avgsm = np.zeros(self.nel)
pool = multiprocessing.Pool()
func = partial(avg_stress_calc, self.xphys, self.nelx,
                       self.nely, self.rmin, stress_matrix, avgsm)
avg_sm = pool.map(func, range(self.nel))

由于某种原因，当我这样做时，我得到一个属性错误：'list' 对象没有属性'shape'，我将它转换为一个数组以获得它的形状 (16, 16)。多进程版本的输出如下所示：

[array([0.81814754, 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.64561319, 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.62517261, 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.78422925, 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.       , 0.       , 0.       , 0.       , 0.6962134, 0.       ,
   0.       , 0.       , 0.       , 0.       , 0.       , 0.       ,
   0.       , 0.       , 0.       , 0.       ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.65993462, 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.63970099, 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.68776093, 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.49890513, 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.60900864,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.71575952, 0.        , 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.73120825, 0.        , 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.32964378, 0.        , 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.53196899, 0.        ,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.80481781,
   0.        ]), array([0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.        , 0.        , 0.        , 0.        , 0.        ,
   0.99930964])]

我希望使用多处理来加速 for 循环，因为这是我的代码中最耗时的。任何关于我做错了什么的建议将不胜感激。谢谢！

【问题讨论】：

你能提供实际的异常输出吗？解释一下你想做什么并说出什么是部分的（...）。另外，您是否要使 func 成为 lambda 方法？
编辑了原始帖子以包含来自 for 循环的两个不同输出
你以前知道数组的维度吗？
我用它来启动它：avg_sm = np.zeros(self.nel) 我只是不明白它是如何切换到列表的......
我写了一个使用示例作为答案，我将尝试使用更大的数组，并将工作时间分配给每个线程并让您知道结果

标签： python arrays python-3.x multiprocessing python-multiprocessing

【解决方案1】：

Pool().map() 很好，但如果您需要在不同实例之间共享对象（内存空间），您最好使用这样的线程：

import numpy as np
from threading import  Thread
import random
import time

def partial(args, process_id , avg_sm):
    # do some stuff
    print("Th%d - Starting working time from 1 to 3 seconds"%process_id)
    # do more stuff
    working_time = random.uniform(1,3)
    time.sleep( working_time )
    # write the result
    avg_sm[process_id] = random.uniform(0.0,1.0)
    print("Th%d - Finishing working time from 1 to 3 seconds"%process_id)

if __name__ == "__main__":
    MAX_THREADS = 1000

    nel = 10000

    args = ["some args you define"]
    
    avg_sm = np.zeros(nel) #[0]*nel
    
    process_l = []

    t0 = time.time()

    for pid in range(nel):
        if pid%MAX_THREADS == 0:
            for p in process_l:
                p.join()
        process_l.append( Thread( target=partial, args=( args, pid, avg_sm ) ) )
        process_l[pid].start()
    
    for p in process_l:
        p.join()

    t1 = time.time() - t0
    
    print(avg_sm)

    print("Total computation time: %d s"%t1)

PS：你可以放任意多的参数，在这个例子中我使用了一个名为args的列表

编辑

由于内存问题需要MAX_THREADS 限制，您必须自己定义。当达到创建的线程数（if pid%MAX_THREADS == 0:）时，您将等到它们完成（也许您可以在一个或多个完成后创建一个新线程，我认为这会更有效率）。我假设您的方法延迟 1 到 3 秒才能完成计算时间。

结果：

Th9889 - Starting working time from 1 to 3 seconds
Th9988 - Starting working time from 1 to 3 seconds
Th9919 - Starting working time from 1 to 3 seconds
Th9986 - Starting working time from 1 to 3 seconds
Th9866 - Starting working time from 1 to 3 seconds
Th9951 - Starting working time from 1 to 3 seconds
Th9918 - Starting working time from 1 to 3 seconds
Th9991 - Starting working time from 1 to 3 seconds
Th9886 - Starting working time from 1 to 3 seconds
Th9915 - Starting working time from 1 to 3 seconds
Th9996 - Starting working time from 1 to 3 seconds
Th9963 - Starting working time from 1 to 3 seconds
Th9978 - Starting working time from 1 to 3 seconds
[0.01340808 0.0567745  0.31191508 ... 0.91127015 0.95141791 0.60075809]
Total computation time: 42 s

【讨论】：