给定大量工作时，Python 多处理线程永远不会加入答案

【问题标题】：Python multiprocessing threads never join when given large amounts of work给定大量工作时，Python 多处理线程永远不会加入
【发布时间】：2014-04-22 00:47:36
【问题描述】：

我不认为这是 this 的重复，因为他的问题似乎是由使用 multiprocessing.pool 引起的，我没有这样做。

这个程序：

import multiprocessing
import time

def task_a(procrange,result):
    "Naively identify prime numbers in an iterator of integers. Procrange may not contain negative numbers, 0, or 1. Result should be a multiprocessing.queue."

    for i in procrange: #For every number in our given iterator...
        for t in range (2,(i//2)+1): #Take every number up to half of it...
            if (i % t == 0): #And see if that number goes evenly into it.
                break   #If it does, it ain't prime.
        else:
            #print(i)
            result.put(i) #If the loop never broke, it's prime.




if __name__ == '__main__':
    #We seem to get the best times with 4 processes, which makes some sense since my machine has 4 cores (apparently hyperthreading doesn't do shit)
    #Time taken more or less halves for every process up to 4, then very slowly climbs back up again as overhead eclipses the benifit from concurrency
    processcount=4
    procs=[]
    #Will search up to this number.
    searchto=11000
    step=searchto//processcount
    results=multiprocessing.Queue(searchto)
    for t in range(processcount):
        procrange=range(step * t, step * (t+1) )
        print("Process",t,"will search from",step*t,"to",step*(t+1))
        procs.append(
                     multiprocessing.Process(target=task_a, name="Thread "+str(t),args=(procrange,results))
                     )
    starttime=time.time()
    for theproc in procs:
        theproc.start()
    print("Processing has begun.")

    for theproc in procs:
        theproc.join()
        print(theproc.name,"has terminated and joined.")
    print("Processing finished!")
    timetook=time.time()-starttime

    print("Compiling results...")

    resultlist=[]
    try:
        while True:
            resultlist.append(results.get(False))
    except multiprocessing.queues.Empty:
        pass

    print(resultlist)
    print("Took",timetook,"seconds to find",len(resultlist),"primes from 0 to",searchto,"with",processcount,"concurrent executions.")

...完美运行，给出结果：

Process 0 will search from 0 to 2750
Process 1 will search from 2750 to 5500
Process 2 will search from 5500 to 8250
Process 3 will search from 8250 to 11000
Processing has begun.
Thread 0 has terminated and joined.
Thread 1 has terminated and joined.
Thread 2 has terminated and joined.
Thread 3 has terminated and joined.
Processing finished!
Compiling results...
[Many Primes]
Took 0.3321540355682373 seconds to find 1337** primes from 0 to 11000 with 4 concurrent executions.

但是，如果search_to 增加了 500...

Processing has begun.
Thread 0 has terminated and joined.
Thread 1 has terminated and joined.
Thread 2 has terminated and joined.

...剩下的就是沉默。 Process Hacker 显示每个 Python 线程消耗 12% 的 CPU，一个一个地逐渐减少……并且没有终止。它们只是挂起，直到我手动终止它们。

为什么？

** 显然，无论是上帝还是圭多都有一种残酷的幽默感。

【问题讨论】：

无法在 2.7.5 上复制。成功测试多达 30,000 个。
不适用于我的 2.7.3 安装。你用的是什么操作系统？我在 Windows 上。
OSX。也许是 Windows 上的线程不一致？
这似乎很有可能。
我有一个非常相似的问题，请参阅我的解决方案：stackoverflow.com/questions/28807023/…

标签： python multithreading python-3.x multiprocessing

【解决方案1】：

看来result.put(i)有问题，因为当我提交时，脚本开始运行良好。所以我建议你不要使用multiprocessing.Queue来保存结果。相反，您可以使用数据库：MySQL、MongoDB 等。注意：您不能使用 SQLite，因为 SQLite 只有一个进程可以随时更改数据库（来自docs）。

【讨论】：