使用 atexit 终止线程时脚本卡在退出时

Question

我正在使用 python 3.7.4 上的线程，我想使用 atexit 注册一个将（干净地）终止线程的清理函数。

例如：

# example.py
import threading
import queue
import atexit
import sys

Terminate = object()

class Worker(threading.Thread):
    def __init__(self):
        super().__init__()
        self.queue = queue.Queue()

    def send_message(self, m):
        self.queue.put_nowait(m)

    def run(self):
        while True:
            m = self.queue.get()
            if m is Terminate:
                break
            else:
                print("Received message: ", m)


def shutdown_threads(threads):
    for t in threads:
        print(f"Terminating thread {t}")
        t.send_message(Terminate)
    for t in threads:
        print(f"Joining on thread {t}")
        t.join()
    else:
        print("All threads terminated")

if __name__ == "__main__":
    threads = [
        Worker()
        for _ in range(5)
    ]
    atexit.register(shutdown_threads, threads)

    for t in threads:
        t.start()

    for t in threads:
        t.send_message("Hello")
        #t.send_message(Terminate)

    sys.exit(0)

但是，似乎与 atexit 回调中的线程和队列交互会通过一些内部关闭例程创建死锁：

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
^CException ignored in: <module 'threading' from '/usr/lib64/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 1308, in _shutdown
    lock.acquire()
KeyboardInterrupt
Terminating thread <Worker(Thread-1, started 140612492904192)>
Terminating thread <Worker(Thread-2, started 140612484511488)>
Terminating thread <Worker(Thread-3, started 140612476118784)>
Terminating thread <Worker(Thread-4, started 140612263212800)>
Terminating thread <Worker(Thread-5, started 140612254820096)>
Joining on thread <Worker(Thread-1, stopped 140612492904192)>
Joining on thread <Worker(Thread-2, stopped 140612484511488)>
Joining on thread <Worker(Thread-3, stopped 140612476118784)>
Joining on thread <Worker(Thread-4, stopped 140612263212800)>
Joining on thread <Worker(Thread-5, stopped 140612254820096)>
All threads terminated

（KeyboardInterrupt 是我使用的ctrl-c，因为该过程似乎无限期挂起）。

但是，如果我在退出之前发送Terminate 消息（取消注释t.send_message("Hello") 之后的行），程序不会挂起并正常终止：

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Terminating thread <Worker(Thread-1, stopped 140516051592960)>
Terminating thread <Worker(Thread-2, stopped 140516043200256)>
Terminating thread <Worker(Thread-3, stopped 140515961992960)>
Terminating thread <Worker(Thread-4, stopped 140515953600256)>
Terminating thread <Worker(Thread-5, stopped 140515945207552)>
Joining on thread <Worker(Thread-1, stopped 140516051592960)>
Joining on thread <Worker(Thread-2, stopped 140516043200256)>
Joining on thread <Worker(Thread-3, stopped 140515961992960)>
Joining on thread <Worker(Thread-4, stopped 140515953600256)>
Joining on thread <Worker(Thread-5, stopped 140515945207552)>
All threads terminated

这引出了一个问题，相对于atexit 处理程序，这个threading._shutdown 例程何时执行？ 与atexit 处理程序中的线程交互是否有意义？

Answer 1

您可以使用一个守护线程来要求您的非守护线程优雅地进行清理。举个必要的例子，如果您使用的是启动非守护线程的第三方库，则必须更改该库或执行以下操作：

import threading

def monitor_thread():
    main_thread = threading.main_thread()
    main_thread.join()
    send_signal_to_non_daemon_thread_to_gracefully_shutdown()


monitor = threading.Thread(target=monitor_thread)
monitor.daemon = True
monitor.start()

start_non_daemon_thread()

将它放在原始海报代码的上下文中（注意我们不需要 atexit 函数，因为在所有非守护线程停止之前不会调用它）：

if __name__ == "__main__":
    threads = [
        Worker()
        for _ in range(5)
    ]
    
    for t in threads:
        t.start()

    for t in threads:
        t.send_message("Hello")
        #t.send_message(Terminate)

    def monitor_thread():
        main_thread = threading.main_thread()
        main_thread.join()
        shutdown_threads(threads)

    monitor = threading.Thread(target=monitor_thread)
    monitor.daemon = True
    monitor.start()

Answer 2

atexit.register(func) 将func 注册为要在终止时执行的函数。

在主线程中执行最后一行代码（上例中为sys.exit(0)）后，（由解释器）调用threading._shutdown等待所有非守护线程（上例中创建的工人）退出

当没有存活的非守护线程时，整个 Python 程序退出。

所以在输入CTRL+C后，主线程被SIGINT信号终止，然后atexit注册的函数被解释器调用。

顺便说一句，如果您将daemon=True 传递给Thread.__init__，程序将直接运行，无需任何人工交互。