锁定进程直到子进程到达某个点

Question

我正在并行运行多个子进程，但我需要锁定每个进程，直到子进程给出输出（通过打印功能）。子进程正在运行已打包为可执行文件的 python 脚本。

代码如下：

import multiprocessing as mp
import subprocess
import os


def main(args):

    l,inpath = args
    l.acquire()
    print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\\')[-1] 

    #Run KNN executable as a subprocess
    subprocess.call(os.path.join(os.getcwd(), "KNN.exe"))
    #This is where I want to wait for any output from the subprocess before releasing the lock
    l.release()

    #Here I would like to wait until subprocess is done then print that it is done
    l.acquire()
    print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
    l.release()


if __name__ == "__main__":
    #Set working directory path containing input text file
    os.chdir("C:\Users\Patrick\Google Drive\KNN")
    #Get folder names in directory containing GCM input
    manager = mp.Manager()
    l = manager.Lock()
    gcm_dir = "F:\FIDS_GCM_Data_CMIP5\UTRB\UTRB KNN-CAD\Input"
    paths = [(l, os.path.join(gcm_dir, folder)) for folder in os.listdir(gcm_dir)]    
    #Set up multiprocessing pool
    p = mp.Pool(mp.cpu_count())
    #Map function through input paths
    p.map(main, paths)

所以目标是锁定进程，以便子进程可以运行，直到收到输出。之后可以释放锁并且子进程可以继续，直到它完成，然后我想打印它已经完成。

我的问题是如何在释放进程上的锁（多个）之前等待子进程的单个（并且唯一）输出？

另外，我怎样才能等待进程终止然后打印它已完成？

Answer 1

您的代码使用了call 方法，该方法已经在等待子进程完成（这意味着所有输出都已生成）。我从您的问题推断出您希望能够区分何时首次写入输出和何时完成子流程。以下是我推荐的内联修改代码：

def main(args):

    l,inpath = args
    l.acquire()
    print "Running KNN.exe for files in %s" %   os.path.normpath(inpath).split('\\')[-1]

    #Run KNN executable as a subprocess
    #Use the Popen constructor
    proc = subprocess.Popen(os.path.join(os.getcwd(), "KNN.exe"), stdout=subprocess.PIPE)

    #This is where I want to wait for any output from the subprocess before releasing the lock
    # Wait until the subprocess has written at least 1 byte to STDOUT (modify if you need different logic)
    proc.stdout.read(1)

    l.release()

    #Here I would like to wait until subprocess is done then print that it is done
    #proc.wait()
    (proc_output, proc_error) = proc.communicate()

    l.acquire()
    print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
    l.release()

请注意，上述内容并不假设您想对子进程的输出做任何事情，除了检查它是否已生成。如果您想对该输出做任何不那么琐碎的事情（消耗 1 个字节，然后将其放在地板上），proc.stdout（这是一个file 对象）应该代表子进程在运行时生成的所有内容.