比生成、使用和关闭多个 python 多处理池更好的做法？答案

【问题标题】：A better practice than spawning, using, and closing multiple python multiprocessing pools?比生成、使用和关闭多个 python 多处理池更好的做法？
【发布时间】：2014-05-09 03:53:15
【问题描述】：

我有一个如下形式的算法：

V = {}
for i in range(N): 
    V[i] = {}
    for j in range(10):
        if should_skip(i,j):
            continue
        V[i][j] = do_something(V)

我已成功并行化如下：

V = {}
for i in range(N):
    V[i] = {}
    p = Pool(4) # On each iteration, start a new pool
    results = []
    for j in range(10):
        if should_skip(i,j):
            continue
        results.append(p.apply_async(do_something, (V,)))
    p.close() 
    p.join() # Then close and join the pool after forking the jobs
    for result in results:
        r = result.get() # do_something modified to return 'j' as its second arg
        V[i][r[1]] = r[0]

我想知道，这是正确的方法吗？保持打开和关闭池不是很昂贵吗？有没有更好的方法来避免每次迭代都创建一个新池？

【问题讨论】：

标签： python parallel-processing multiprocessing threadpool pool

【解决方案1】：

没有必要在池上调用close 和join，因为您正在对所有单独的AsyncResult 对象调用get，这将是block until the result is ready。所以，你可以这样做：

V = {}
p = Pool(4)
for i in range(N):
    V[i] = {}
    results = []
    for j in range(10):
        if should_skip(i,j):
            continue
        results.append(p.apply_async(do_something, (V,)))
    for result in results:
        r = result.get() # This will block until `result` is available.
        V[i][r[1]] = r[0]

【讨论】：