Python多处理：如果返回时间过长，则终止进程

Question

下面是一个简单的例子，它因为子进程退出而没有返回任何东西而父进程一直等待而冻结。如果需要的时间太长，有没有办法让进程超时，让其余的继续？我是 python 多处理的初学者，我发现文档不是很有启发性。

import multiprocessing as mp
import time

def foo(x):
    if x == 3:
        sys.exit()
    #some heavy computation here
    return result

if __name__ == '__main__':  
    pool = mp.Pool(mp.cpu_count)
    results = pool.map(foo, [1, 2, 3])

Answer 1

我遇到了同样的问题，我就是这样解决的。也许有更好的解决方案，但是，它也解决了未提及的问题。例如。如果进程占用了很多资源，则可能会发生正常终止需要一段时间才能完成进程的情况——因此我使用强制终止 (kill -9)。这部分可能只适用于 Linux，因此如果您使用其他操作系统，您可能需要调整终止。

这是我自己的代码的一部分，所以它可能不可复制粘贴。

from multiprocessing import Process, Queue
import os 
import time 

timeout_s = 5000 # seconds after which you want to kill the process

queue = Queue()  # results can be written in here, if you have return objects

p = Process(target=INTENSIVE_FUNCTION, args=(ARGS_TO_INTENSIVE_FUNCTION, queue))
p.start()

start_time = time.time()
check_interval_s = 5  # regularly check what the process is doing

kill_process = False
finished_work = False

while not kill_process and not finished_work:
    time.sleep(check_interval_s)  
    now = time.time()
    runtime = now - start_time

    if not p.is_alive():
        print("finished work")
        finished_work = True

    if runtime > timeout_s and not finished_work:
        print("prepare killing process")
        kill_process = True

if kill_process:
    while p.is_alive():
        # forcefully kill the process, because often (during heavvy computations) a graceful termination
        # can be ignored by a process.
        print(f"send SIGKILL signal to process because exceeding {timeout_s} seconds.")
        os.system(f"kill -9 {p.pid}")

        if p.is_alive():
            time.sleep(check_interval_s)
else:
    try:
        p.join(60)  # wait 60 seconds to join the process
        RETURN_VALS = queue.get(timeout=60)
    except Exception:
        # This can happen if a process was killed for other reasons (such as out of memory)
        print("Joining the process and receiving results failed, results are set as invalid.")