【问题标题】:Amazon S3 Bulk File RenamingAmazon S3 批量文件重命名
【发布时间】:2021-10-07 00:12:11
【问题描述】:

我的 Amazon S3 存储桶中有一堆文件,看起来像这样(不止两个,还有数百个):

09292021-testpilot-state-test-callers-09-29-2021-970669454.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=BRIAN72GRAHWB6KIND7M%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210930T012101Z&X-Amz-Expires=172800&X-Amz-SignedHeaders=host&X-Amz-Signature=8d4de3f18g2c178e1f6c94d0a09bf6a07ffr731454da5ec8940868f499cfc2bc

09272021-testpilot-state-test-callers-09-29-2021-970669454.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=STEVE2313DJFSOODC%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210930T012101Z&X-Amz-Expires=172800&X-Amz-SignedHeaders=host&X-Amz-Signature=8d4de3f18g2c178e1f6c53-309f3f6a07ffr731454da5ec8940868f499cfc2bc

我正在尝试对它们进行批量重命名,以便您基本上删除 .csv 之后的所有内容,以便文件看起来像这样:

09292021-testpilot-state-test-callers-09-29-2021-970669454.csv

09272021-testpilot-state-test-callers-09-29-2021-970669454.csv

最有效的方法是什么?

【问题讨论】:

  • 有多少个文件?如果是 2,则手动重命名它们。如果是2万亿,写一个脚本。批处理操作也可能有所帮助:docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops.html
  • 不止两个,还有数百个。那会是什么样子?
  • 您可以使用 Amazon S3 SDK 来执行此任务。最简单的方法是调用 CopyObject 并传入一个新名称。
  • @smac2020 你能在答案中提供一个例子吗?
  • 请注意,您不能“重命名”S3 对象,至少在重命名某些东西的传统意义上是这样。您需要复制然后删除原件。一定要保留所有相关的元数据。

标签: amazon-web-services amazon-s3


【解决方案1】:

您可以使用 Amazon S3 开发工具包来执行此任务。最简单的方法是调用 CopyObject 并传入一个新名称。例如,如果您要在 Java 中实现此功能(您也可以使用 AWS 开发工具包支持的其他语言来执行此功能),您可以使用以下代码:

package com.example.s3;

// snippet-start:[s3.java2.copy_object.import]
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
import software.amazon.awssdk.services.s3.model.CopyObjectResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
// snippet-end:[s3.java2.copy_object.import]

/**
 * To run this AWS code example, ensure that you have setup your development environment, including your AWS credentials.
 *
 * For information, see this documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */


public class CopyObject {

    public static void main(String[] args) {
        final String USAGE = "\n" +
                "Usage:\n" +
                "    <objectKey> <fromBucket> <toBucket>\n\n" +
                "Where:\n" +
                "    objectKey - the name of the object (for example, book.pdf).\n\n" +
                "    fromBucket - the S3 bucket name that contains the object (for example, bucket1).\n" +
                "    toBucket - the S3 bucket to copy the object to (for example, bucket2).\n";

        if (args.length != 3) {
            System.out.println(USAGE);
            System.exit(1);
       }

        String objectKey = args[0];
        String fromBucket = args[1];
        String toBucket =  args[2];

        System.out.format("Copying object %s from bucket %s to %s\n",
                objectKey, fromBucket, toBucket);

        Region region = Region.US_EAST_1;
        S3Client s3 = S3Client.builder()
                .region(region)
                .build();

        copyBucketObject (s3, fromBucket, objectKey, toBucket);
        s3.close();
    }

    // snippet-start:[s3.java2.copy_object.main]
    public static String copyBucketObject (S3Client s3, String fromBucket, String objectKey, String toBucket) {

        String encodedUrl = null;
        try {
            encodedUrl = URLEncoder.encode(fromBucket + "/" + objectKey, StandardCharsets.UTF_8.toString());
        } catch (UnsupportedEncodingException e) {
            System.out.println("URL could not be encoded: " + e.getMessage());
        }
        CopyObjectRequest copyReq = CopyObjectRequest.builder()
                .copySource(encodedUrl)
                .destinationBucket(toBucket)
                .destinationKey(objectKey)
                .build();

        try {
            CopyObjectResponse copyRes = s3.copyObject(copyReq);
            return copyRes.copyObjectResult().toString();
        } catch (S3Exception e) {
            System.err.println(e.awsErrorDetails().errorMessage());
            System.exit(1);
        }
        return "";
    }
    // snippet-end:[s3.java2.copy_object.main]
}

【讨论】:

  • OP 可能还需要保留对象的元数据。不确定 CopyObjectRequest 是否默认发生这种情况。
  • @smac2020 谢谢你。尽管它实际上消除了.csv 之后的所有内容,但在脚本中的哪个位置?
  • 您可以将该任务添加到您的逻辑中。因此,请跟踪对象,然后在重命名它们后 - 您可以删除原始对象。
猜你喜欢
  • 2020-06-05
  • 2015-02-07
  • 2013-12-19
  • 2017-04-20
  • 2012-09-02
  • 1970-01-01
  • 2018-04-26
  • 1970-01-01
相关资源
最近更新 更多