查找 s3 存储桶的 1 级前缀大小，同时包括使用 boto3 和 python 的版本

Question

我是 aws python 新手，并试图考虑通过 UI 上的指标选项卡显示的总存储桶大小与计算给定存储桶中的一个文件夹的大小。我尝试通过设置库存配置来获取它，但它没有显示我要查找的内容。

我有一个启用了版本控制的 s3 存储桶名称 my_bucket。
它有 100 个对象和 26 个子文件夹（每个子文件夹中有 100000+ 个对象，每个对象至少有两个版本）

我想做什么： 计算并显示总大小，包括 180 个子文件夹中每个子文件夹的版本。

A  Size 1GB  
B  Size 10TB    
.  
.  
.  
Z Size 13TB

我想怎么做 寻找结合
的解决方案 来自链接一的基于配置文件的身份验证并使用 bucket.object_versions
从链接 2 计算一级文件夹大小
同时也考虑到版本。 （Link2没有版本）

Link1 https://stackoverflow.com/a/58125684/4590025
链接2https://stackoverflow.com/a/49763268/4590025

import boto3

PROFILE = "my_profile"
BUCKET = "my_bucket"

session = boto3.Session(profile_name = PROFILE)
s3 = session.resource('s3')
bucket = s3.Bucket(BUCKET)

#bucket.object_versions.do_something_with_it


conn = boto3.client('s3')

top_level_folders = dict()

for key in conn.list_objects(Bucket='my_bucket')['Contents']:

    folder = key['Key'].split('/')[0]
    print("Key %s in folder %s. %d bytes" % (key['Key'], folder, key['Size']))

    if folder in top_level_folders:
        top_level_folders[folder] += key['Size']
    else:
        top_level_folders[folder] = key['Size']


for folder, size in top_level_folders.items():
    print("Folder: %s, size: %d" % (folder, size))

我还提到了https://stackoverflow.com/a/48867829，但我不确定如何使用这两者，目前当我运行它时，尽管设置了会话，但仍出现以下错误：

Traceback (most recent call last):
  File ".\folder_size.py", line 17, in <module>
    for key in conn.list_objects(Bucket='my_bucket')['Contents']:
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 622, in _make_api_call
    operation_model, request_dict, request_context)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\signers.py", line 160, in sign
    auth.add_auth(request)
  File "C:\Users\ginger\AppData\Local\Programs\Python\Python37\lib\site-packages\botocore\auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
PS C:\Users\ginger\test>

Answer 1

问题在于程序使用：

conn = boto3.client('s3')

这忽略了之前设置的配置文件：

session = boto3.Session(profile_name = PROFILE)

因此，如果您想使用配置文件创建 S3 客户端，那么它应该使用：

conn = session.client('s3')

为避免分页问题，​​您可以使用资源方法检索所有对象：

for object in bucket.objects.all():
    folder = object.key.split('/')[0]
    print("Key %s in folder %s. %d bytes" % (object.key, folder, object.size))
...