在 Python 中并行下载多个 S3 对象

Question

有没有办法在 Python3 中使用 boto3 同时下载 S3 文件？ 我知道aiobotocore 库，但我想知道是否有办法使用标准的boto3 库来做到这一点。

Answer 1

如果您想使用boto3 将大量较小的文件直接并行下载到磁盘，您可以使用multiprocessing 模块执行此操作。这里有一个小sn-p 可以做到这一点。你像这样运行它：./download.py bucket_name s3_key_0 s3_key_1 ... s3_key_n

#!/usr/bin/env python3
import multiprocessing
import boto3
import sys

# make a per process s3_client
s3_client = None
def initialize():
  global s3_client
  s3_client = boto3.client('s3')

# the work function of each process which will fetch something from s3
def download(job):
  bucket, key, filename = job
  s3_client.download_file(bucket, key, filename)

if __name__ == '__main__':
  # make the jobs, arguments to program are: bucket s3_key_0 s3_key_1 ... s3_key_n
  bucket = sys.argv[1]
  jobs = [(bucket, key, key.replace('/', '_')) for key in sys.argv[2:] ]

  # make a process pool to do the work
  pool = multiprocessing.Pool(multiprocessing.cpu_count(), initialize)
  pool.map(download, jobs)
  pool.close()
  pool.join()

其中一个重要的部分是我们为每个进程将重用的每个进程创建一个 s3 客户端实例。这很重要，原因有两个。首先，创建客户端很慢，所以我们希望尽可能少地这样做。其次，客户端不应跨进程共享，因为对 download_file 的调用可能会改变客户端的内部状态。

Answer 2

下面的 sn-p 将允许您使用多处理从 s3 下载多个对象

import boto3
import multiprocessing as mp
import os

s3 = boto3.resource('s3')
my_bucket = s3.Bucket('My_bucket')
        
def s3download(object_key_file):
    my_bucket.download_file(object_key_file[0], object_key_file[1])
    print('downloaded file with object name... {}'.format(object_key_file[0]))
    print('downloaded file with file name... {}'.format(object_key_file[1]))
        
def parallel_s3_download():
    object_key_file=[]
    for s3_object in my_bucket.objects.filter(Prefix="directory_name/"):
        # Need to split s3_object.key into path and file name, else it will give error file not found.
        path, filename = os.path.split(s3_object.key)
        object_key_file.append((s3_object.key,filename))
    object_key_file.pop(0)
    pool = mp.Pool(min(mp.cpu_count(), len(object_key_file)))  # number of workers
    pool.map(s3download, object_key_file, chunksize=1)
    pool.close()
if __name__ == "__main__":
    parallel_s3_download()
    print('downloading zip file')