【发布时间】:2021-09-14 17:32:59
【问题描述】:
我正在尝试使用 AWS Glue 在跨账户 S3 存储桶之间移动文件。我正在使用 Glue 和 python shell。 我有源存储桶的列表和获取对象权限。我能够列出所有文件,但是当我尝试将文件加载到目标存储桶时,我收到错误“FileNotFoundError: [Errno 2] No such file or directory: 'test/f1=x/type=b/file1.镶木地板'
源 s3 上的文件有分区:
test/f1=x/type=a/file1.parquet
test/f1=x/type=a/file2.parquet
test/f1=x/type=b/file1.parquet
test/f1=x/type=b/file2.parquet
我只是想用 f1=x 和 type=b 加载文件
import pandas as pd
import boto3
client = boto3.client('s3')
bucket = 'mysourcebucketname'
folder_path = 'test/f1=x/type=b/'
def my_keys(bucket,folder_path):
keys = []
resp = client.list_objects(Bucket=bucket, Prefix=folder_path)
for obj in resp['Contents']:
keys.append(obj['Key'])
return keys
files = my_keys(bucket,folder_path)
#print(files)
for file in files:
bucketdest = 'mydestinationbucket'
new_file_name = file.split('/')[-1]
s3_file = 'destfolder1/destfolder2/'+"typeb"+new_file_name
client.upload_file(file,bucketdest,s3_file,ExtraArgs={'GrantFullControl':'id =""})
【问题讨论】:
标签: python amazon-web-services amazon-s3 boto3 aws-glue