增加连接池大小

Question

我们正在运行以下代码以并行上传到 GCP 存储桶。根据我们看到的警告，我们似乎正在迅速用完池中的所有连接。有没有办法配置库正在使用的连接池？

def upload_string_to_bucket(content: str):
        blob = bucket.blob(cloud_path)
        blob.upload_from_string(content)

with concurrent.futures.ThreadPoolExecutor() as executor:
            executor.map(upload_string_to_bucket, content_list)

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com
WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: www.googleapis.com

Answer 1

我在并行下载 blob 时遇到了类似的问题。

这篇文章可能会提供信息。 https://laike9m.com/blog/requests-secret-pool_connections-and-pool_maxsize,89/

就我个人而言，我不认为增加连接拉取是最好的解决方案， 我更喜欢按 pool_maxsize 分块“下载”。

def chunker(it: Iterable, chunk_size: int):
    chunk = []
    for index, item in enumerate(it):
        chunk.append(item)
        if not (index + 1) % chunk_size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

for chunk in chunker(content_list, 10):
    with concurrent.futures.ThreadPoolExecutor() as executor:
        executor.map(upload_string_to_bucket, chunk)

当然，我们可以在准备好后立即生成下载，一切随心所欲。