Python 多处理映射返回无序结果

Question

我有一个简单的多进程池脚本，它使用 map 来按顺序取回结果，就像按顺序执行一样。

注意 - 请忽略“(1-3A-D)”。它们不是输出的一部分，仅用于说明目的。假设它是 multiprocessing 执行的序列。

import base64
import getpass
import os

from ssh.ssh_module import SSH
from multiprocessing import Pool

username = ''
password = ''
cmd = 'uptime'

def runcommand (server):

    print ('Proccess ID: %d' % (os.getpid ()))
    ssh = SSH (server, username, password)
    if ssh.connect ():
        print (ssh.command (cmd))

    ssh.close ()


if __name__ == '__main__':

    username = input ('Username: ')
    # Encode password for SSH
    password = getpass.getpass ('Password: ')
    password = base64.b64encode (password.encode('utf-8'))

    servers = ['192.168.100.1', '192.168.100.2', '192.168.100.3']
    with Pool (processes = 2) as pool:
        pool.map (runcommand, servers)

输出：

(virtual) [user@centos python]$ ./multiprocess.py 
Username: spongebob
Password: 
(1A) Proccess ID: 24528
(1B) SSH into 192.168.100.1
(2A) Proccess ID: 24529
(2B) SSH into 192.168.100.2
(1C) Successfully logged in!
     00:15:24 up 4 days,  8:16,  0 users,  load average: 0.12, 0.10, 0.20

(1D) SSH session closed.

(3A) Proccess ID: 24529
(3B) SSH into 192.168.100.3
(2C) Successfully logged in!
     00:15:26 up 7 days,  6:17,  0 users,  load average: 0.10, 0.11, 0.09

(2D) SSH session closed.

(3C) Successfully logged in!
     00:15:47 up 5 days,  6:41,  0 users,  load average: 0.14, 0.10, 0.17

(3D) SSH session closed.

如您所见，结果是随机返回的，我认为使用 'pool.map' 应该是有序的。首先执行 IP/系统的顺序并不重要，但输出必须以正确的顺序进行，这样才有意义。关于如何修复它的任何想法，以便我可以在下面获得所需的输出？

(1A) Proccess ID: 24528
(1B) SSH into 192.168.100.1
(1C) Successfully logged in!
     00:15:24 up 4 days,  8:16,  0 users,  load average: 0.12, 0.10, 0.20

(1D) SSH session closed.

(2A) Proccess ID: 24529
(2B) SSH into 192.168.100.2
(2C) Successfully logged in!
     00:15:26 up 7 days,  6:17,  0 users,  load average: 0.10, 0.11, 0.09

(2D) SSH session closed.

(3A) Proccess ID: 24529
(3B) SSH into 192.168.100.3
(3C) Successfully logged in!
     00:15:47 up 5 days,  6:41,  0 users,  load average: 0.14, 0.10, 0.17

(3D) SSH session closed.

Answer 1

Pool.map 以与计划相同的顺序返回结果，但不确保它们以相同的顺序执行。

顺序执行会破坏使用multiprocessing 的目的。如果您需要按顺序执行，只需使用map。

Answer 2

我设法通过删除 SSH 类中的消息并将其移至调用者来实现所需的输出。这是一个使用 .join() 来构造消息序列的问题。这不是最漂亮的，但现在已经订购了。

import base64
import getpass
import os

from cet.ssh_module import SSH
from multiprocessing import Pool

username = ''
password = ''
cmd = 'uptime'


def runcommand (server):

    print ('Proccess ID: %d' % (os.getpid ()))
    ssh = SSH (server, username, password)
    s1 = 'SSH into %s' % server
    if ssh.connect ():
        s2 = 'Successfully logged in!'
        s3 = ssh.command (cmd)

        ssh.close ()
        s4 = 'SSH session closed.\n'

        output = '\n'.join ([s1, s2, s3, s4])

        return output
    else:
        pass


if __name__ == '__main__':

    servers = ['192.168.100.1', '192.168.100.2', '192.168.100.3']

    username = input ('Username: ')
    # Encode password for SSH
    password = getpass.getpass ('Password: ')
    password = base64.b64encode (password.encode('utf-8'))

    with Pool (processes = 4) as pool:
        results = pool.map (runcommand, servers)

    for i in results:
        print (i)

输出

(virtual) [user@centos python]$ ./multiprocess.py 
Username: spongebob
Password: 
Proccess ID: 40764
Proccess ID: 40763
Proccess ID: 40766
SSH into 192.168.100.1
Successfully logged in!
 17:25:33 up 8 days, 23:51,  0 users,  load average: 3.04, 2.05, 1.43

SSH session closed.

SSH into 192.168.100.2
Successfully logged in!
 17:25:17 up 2 days,  1:26,  0 users,  load average: 0.44, 0.42, 0.72

SSH session closed.

SSH into 192.168.100.3
Successfully logged in!
 17:25:15 up 8 days, 23:27,  0 users,  load average: 2.37, 2.35, 2.03

SSH session closed.