如何在 Python 的循环中将一个进程的执行与另一个进程联系起来

Question

如何保持循环继续进行，同时让一些进程在循环中等待其他进程？ （说明见下面代码，用代码解释更有意义）

for i in range(0, len(my_list), batch_size):
    current_batch = my_list[i:i+batch_size]
    download_processes = [Popen('do stuff')] # NOT ACTUAL CODE. This downloads files.
    _ = [p.wait() for p in download_processes] # wait until all files above download before executing the code below 

    more_processes = [Popen('do stuff')] # NOT ACTUAL CODE. This zips the downloaded files
    # execute yet more processes (that process the zips files) when "more_processes" end, but keep the loop going to start downloading more files

Answer 1

1. 创建一个池

2. 使用池内循环进行批处理

3. 将 result = pool.map_async() 与您的目标方法一起使用

4. 执行 result.get(timeout) 或 result.wait()

5. 如果超时或满足条件后，返回并跳出 while 循环和 pool.close，终止然后加入。

def process_url(url):
  # call url and process data
  
  pass

def pool_handler():
    with Pool() as p:
     for i in range(0, len(my_list), batch_size):
      current_batch_urls = my_list[i:i+batch_size]
    
      # this will create processes and process url
      r = p.map_async(process_url, current_batch_urls)
      r. wait()#wait for each batch

   #outside loop 
    p.close()
    p.join()#wait until all processes are done

if __name__ == '__main__':
    pool_handler()

Answer 2

你可以使用multiprocessing模块来实现这个

from multiprocessing import Pool
import time, requests

urls = ["file_url1","file_url2","file_url3"]


def download_file(url):
    return requests.get(url).content.strip()

def process_url(url):
    file_content = download_file(url)
    # Process File content 

def pool_handler():
    p = Pool(2)
    p.map(process_url, urls)

if __name__ == '__main__':
    pool_handler()