使用 Python 2.7 引发属性错误的多处理

Question

from itertools import product
from multiprocessing import Pool

with Pool(4) as pool:
            pool.map(lambda x: run_test_function(x, arg2, arg3, arg4), arg1)
    

执行上述代码后出现以下错误。还有一些其他的代码，我不能在这里写。但实际问题仅来自这段代码。

Traceback (most recent call last):
  File "modProfileChange_test.py", line 347, in <module>
    main(sys.argv[1:])
  File "modProfileChange_test.py", line 336, in main
    test_run_code(arg1, arg2, arg3, arg4, arg5, arg6)
  File "modProfileChange_test.py", line 23, in test_run_code
    with Pool(4) as pool:
AttributeError: __exit__

Answer 1

1. 在 Python 2.7 中，multiprocessing.Pool 不是上下文管理器，因此不能在 with 语句中使用
   1. 解决方案 - 使用对变量的常规分配创建池：

      my_pool = Pool(4)
      my_pool.map(...)
      

2. lambda functions don't work with multiprocessing.Pool，即使在 Python 3 中也是如此。
   1. 解决方案 - 使用上面链接中的解决方案模拟闭包：

      from functors import partial
      
      def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
          # your code here
          ...
      
      process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)
      

把这些放在一起：

from multiprocessing import Pool
from functools import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # this is an example
    print x, fun_arg2, fun_arg3, fun_arg4

if __name__ == "__main__":
    arg1 = 1,2,3,4
    arg2 = "hello"
    arg3 = "world"
    arg4 = "!"

    process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

    my_pool = Pool(4)
    my_pool.map(process_func, arg1)

输出：

~/test $ python2.7 so10.py
1 hello world !
2 hello world !
3 hello world !
4 hello world !