Apache Beam：无法序列化和反序列化属性“awsCredentialsProvider”答案

【问题标题】：Apache Beam: Failed to serialize and deserialize property 'awsCredentialsProvider'Apache Beam：无法序列化和反序列化属性“awsCredentialsProvider”
【发布时间】：2020-09-04 07:52:23
【问题描述】：

我正在从https://github.com/apache/beam 扩展BigQueryTornadoes 示例。我正在进行更改，以便它将作为接收器写入 AWS S3。在我的第一次迭代中，我能够使用以下代码使其工作。

    public static void main(String[] args) {
        Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);

        options.setAwsCredentialsProvider(
                new AWSStaticCredentialsProvider(
                        new BasicAWSCredentials(options.getAwsAccessKey().get(), options.getAwsSecretKey().get())));

        runBigQueryTornadoes(options);
    }

在我的第二次迭代中，我想使用 STSAssumeRoleSessionCredentialsProvider 来支持跨账户 IAM 角色。我有以下代码。

    public static void main(String[] args) {
        Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);

        AWSCredentialsProvider provider = new AWSStaticCredentialsProvider(new BasicAWSCredentials(options.getAwsAccessKey().get(), options.getAwsSecretKey().get()));
        AWSSecurityTokenServiceClientBuilder stsBuilder = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(provider);
        AWSSecurityTokenService sts = stsBuilder.build();

        AWSCredentialsProvider credentialsProvider = new STSAssumeRoleSessionCredentialsProvider.Builder(options.getAwsRoleArn().get(), options.getAwsRoleSession().get())
                .withExternalId(options.getAwsExternalId().get())
                .withStsClient(sts)
                .build();
        options.setAwsCredentialsProvider(credentialsProvider);

        runBigQueryTornadoes(options);
    }

当我运行上面的代码时，我得到了以下异常。

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected IOException (of type java.io.IOException): Failed to serialize and deserialize property 'awsCredentialsProvider' with value 'com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider@4edb24da'
    at com.fasterxml.jackson.databind.JsonMappingException.fromUnexpectedIOE (JsonMappingException.java:338)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes (ObjectMapper.java:3432)
    at org.apache.beam.runners.direct.DirectRunner.run (DirectRunner.java:163)
    at org.apache.beam.runners.direct.DirectRunner.run (DirectRunner.java:67)
    at org.apache.beam.sdk.Pipeline.run (Pipeline.java:317)
    at org.apache.beam.sdk.Pipeline.run (Pipeline.java:303)
    at org.apache.beam.examples.cookbook.BigQueryTornadoesS3STS.runBigQueryTornadoes (BigQueryTornadoesS3STS.java:251)
    at org.apache.beam.examples.cookbook.BigQueryTornadoesS3STS.main (BigQueryTornadoesS3STS.java:267)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:748)

我使用以下mvn 命令运行。

mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.cookbook.BigQueryTornadoesS3STS "-Dexec.args=..." -P direct-runner

我在Beam: Failed to serialize and deserialize property 'awsCredentialsProvider 看到了类似的帖子。但是我没有将它包装成罐子就面临这个问题。

【问题讨论】：

标签： java amazon-s3 serialization deserialization apache-beam

【解决方案1】：

这篇文章I am trying to write to S3 using assumeRole via FileIO with ParquetIO 帮助我使我的代码工作。使用下面的代码，我能够承担跨账户 IAM 角色并写入另一个 AWS 账户拥有的 S3 存储桶。

        Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);

        AWSCredentialsProvider provider = new AWSStaticCredentialsProvider(new BasicAWSCredentials(options.getAwsAccessKey().get(), options.getAwsSecretKey().get()));
        AWSSecurityTokenServiceClientBuilder stsBuilder = AWSSecurityTokenServiceClientBuilder.standard().withCredentials(provider);
        AWSSecurityTokenService sts = stsBuilder.build();

        STSAssumeRoleSessionCredentialsProvider credentials = new STSAssumeRoleSessionCredentialsProvider.Builder(options.getAwsRoleArn().get(), options.getAwsRoleSession().get())
                .withExternalId(options.getAwsExternalId().get())
                .withStsClient(sts)
                .build();

        options.setAwsCredentialsProvider(
                new AWSStaticCredentialsProvider(
                        credentials.getCredentials()));

        runBigQueryTornadoes(options);
    }

注意：代码基于来自https://github.com/apache/beam 的BigQueryTornadoes 示例。

【讨论】：