在 python 中使用多处理时无法使用所有处理器

Question

我正在研究 python 并尝试学习多处理。当我尝试以下代码时，它应该并行运行，但它只使用一个处理器。我无法理解原因。可能是什么问题，为什么它没有使用我电脑的所有 4 个内核。

我试过的代码如下：

import multiprocessing
import time
start = time.perf_counter()

def do_something():
    print("hello")
    time.sleep(1)
    print("done")

p1 = multiprocessing.Process(target=do_something())
p2 = multiprocessing.Process(target=do_something())
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'finished in {round(finish-start,1)} sec')

---

结果：

hello
done
hello
done
finished in 2.1 sec

---

它应该在 1 秒内执行（大约）

#

我用来查核数的代码：

#

import multiprocessing
print("Number of cpu : ", multiprocessing.cpu_count())

---

结果：

Number of cpu :  4

---

#

我尝试的另外一个代码是：

#

from multiprocessing import Lock, Process, Queue, current_process
import time
import queue # imported for using queue.Empty exception


def do_job(tasks_to_accomplish, tasks_that_are_done):
    while True:
        try:
            '''
                try to get task from the queue. get_nowait() function will 
                raise queue.Empty exception if the queue is empty. 
                queue(False) function would do the same task also.
            '''
            task = tasks_to_accomplish.get_nowait()
        except queue.Empty:

            break
        else:
            '''
                if no exception has been raised, add the task completion 
                message to task_that_are_done queue
            '''
            print(task)
            tasks_that_are_done.put(task + ' is done by ' + current_process().name)
            time.sleep(.5)
    return True


def main():
    number_of_task = 10
    number_of_processes = 4
    tasks_to_accomplish = Queue()
    tasks_that_are_done = Queue()
    processes = []

    for i in range(number_of_task):
        tasks_to_accomplish.put("Task no " + str(i))

    # creating processes
    for w in range(number_of_processes):
        p = Process(target=do_job(tasks_to_accomplish, tasks_that_are_done))
        processes.append(p)
        p.start()

    # completing process
    for p in processes:
        p.join()

    # print the output
    while not tasks_that_are_done.empty():
        print(tasks_that_are_done.get())

    return True


if __name__ == '__main__':
    main()

---

结果：

Task no 0
Task no 1
Task no 2
Task no 3
Task no 4
Task no 5
Task no 6
Task no 7
Task no 8
Task no 9
Task no 0 is done by MainProcess
Task no 1 is done by MainProcess
Task no 2 is done by MainProcess
Task no 3 is done by MainProcess
Task no 4 is done by MainProcess
Task no 5 is done by MainProcess
Task no 6 is done by MainProcess
Task no 7 is done by MainProcess
Task no 8 is done by MainProcess
Task no 9 is done by MainProcess

---

#

根据建议，我做了如下改动：

import multiprocessing
import time
start = time.perf_counter()

def do_something():
    print("hello")
    time.sleep(1)
    print("done")

p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
p1.join()
p2.join()
finish = time.perf_counter()
print(f'finished in {round(finish-start,1)} sec')

我得到的结果是：

finished in 0.2 sec

Answer 1

目标必须是可调用对象。

p1 = multiprocessing.Process(target=do_something)

按照你的方式，你在主进程中调用方法并将结果传递给多进程。

Answer 2

您应该使用partial 将函数绑定到参数。目前您只是在调用该函数：

from functools import partial
...
for w in range(number_of_processes):
        p = Process(target=partial(do_job, tasks_to_accomplish, tasks_that_are_done))
        processes.append(p)
        p.start()

正如@DanielRoseman 建议的那样，在Process 调用中传递参数会更好：

Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done))