Python 多处理 log_to_stderr 在 Windows 上重复答案

【问题标题】：Python multiprocessing log_to_stderr duplicated on WindowsPython 多处理 log_to_stderr 在 Windows 上重复
【发布时间】：2018-10-30 04:36:30
【问题描述】：

我正在关注documentation 进行多处理日志记录，但我看到工作人员在每个子进程中创建了两个日志。我在某个地方犯了一个愚蠢的错误吗？

环境：

Python 3.6.1 |Anaconda 自定义（64 位）| (默认, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32

代码（编辑以修复@georgexsh 推荐的范围问题）：

import logging
import multiprocessing


logger = multiprocessing.log_to_stderr(logging.INFO)


def test(i):
    logger.info(f'worker processing {i}')


if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        metrics = pool.map(test, range(20))

日志输出：

[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 4
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 5
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] worker processing 6
[INFO/SpawnPoolWorker-3] worker processing 4
...
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-2] worker processing 12
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-6] worker processing 13
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-8] worker processing 14
[INFO/SpawnPoolWorker-4] worker processing 15
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-2] process shutting down
[INFO/SpawnPoolWorker-6] process shutting down
[INFO/MainProcess] process shutting down

【问题讨论】：

标签： python windows logging duplicates multiprocessing

【解决方案1】：

将logger = multiprocessing.log_to_stderr() 移动到全局范围，而不是在工作函数内部。确保它只调用一次。因为每次log_to_stderr 被调用时，it will add a new handler to the logger：

def test(i):
    logger.info('worker processing %s', i)

if __name__ == '__main__':
    logger = multiprocessing.log_to_stderr(logging.INFO)

注意在windows下，由于没有fork()，the whole module get executed again创建子进程重构上下文时，可以用Pool's initializer初始化logger，它只运行一次前子进程：

logger = None

def test(i):
    logger.info('worker processing %s', i)

def initializer(level):
    global logger
    logger = multiprocessing.log_to_stderr(level)

if __name__ == '__main__':
    pool = multiprocessing.Pool(4, initializer=initializer, initargs=(logging.INFO,))
    metrics = pool.map(test, range(20))

【讨论】：

感谢您发现范围问题！当我在 Linux 上运行代码时，这似乎有效，但我仍然在 Windows 上看到重复的日志（现在每个子进程只有两个）。
@atm 知道了，python 在 windows 中使用了不同的 fork 方法，我稍后会修改。