Flink 读取 S3 文件导致 Jackson 依赖问题

Question

我在我的链接应用程序中读取配置 YAML 文件。我想将此配置文件保留在 S3 文件系统上，但是在我的 pom 中使用 aws-sdk 并尝试读取时出现此错误。我知道这是由于杰克逊的冲突依赖，但我无法解决它。请帮我解决它。

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper; 在 com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54) 在 com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) 在 com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:65) 在 com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:53) 在 com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:107) 在 com.amazonaws.services.s3.AmazonS3Client.createSigner(AmazonS3Client.java:4016) 在 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4913) 在 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872) 在 com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1472) 在 com.bounce.processor.EventProcessor.main(EventProcessor.java:71) 在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在 java.lang.reflect.Method.invoke(Method.java:498) 在 org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)

这是我用来读取文件的这段代码

        AmazonS3 amazonS3Client = new AmazonS3Client(credentials);

        S3Object object = amazonS3Client.getObject(new GetObjectRequest(S3_PROD_BUCKET, para.get("topology")));
        InputStream awsinputStream = object.getObjectContent();

这是我的 pom.xml

        <!-- Flink dependencies -->

        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>5.3.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-filesystem_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>commons-httpclient</groupId>
                    <artifactId>commons-httpclient</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpclient</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpcore</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.parquet</groupId>
            <artifactId>parquet-avro</artifactId>
            <version>${flink.format.parquet.version}</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.7</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-parquet_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>2.10.5</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.dataformat</groupId>
            <artifactId>jackson-dataformat-yaml</artifactId>
            <version>2.10.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.esotericsoftware.yamlbeans/yamlbeans -->
        <dependency>
            <groupId>com.esotericsoftware.yamlbeans</groupId>
            <artifactId>yamlbeans</artifactId>
            <version>1.13</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.uber/h3 -->
        <dependency>
            <groupId>com.uber</groupId>
            <artifactId>h3</artifactId>
            <version>3.6.3</version>
        </dependency>

        <dependency>
            <groupId>com.github.davidmoten</groupId>
            <artifactId>geo</artifactId>
            <version>0.7.7</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-elasticsearch6 -->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>tech.allegro.schema.json2avro</groupId>
            <artifactId>converter</artifactId>
            <version>0.2.9</version>
        </dependency>


        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-statebackend-rocksdb_2.12</artifactId>
            <version>1.10.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-avro-confluent-registry</artifactId>
            <version>${flink.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>


        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bundle</artifactId>
            <version>1.11.756</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.10.0</version>
        </dependency>

Answer 1

仅从 POM 中获得有关冲突的答案是极不可能的。相反，您应该通过调用以下命令来引用 maven 依赖插件：

mvn dependency:tree

这将打印所有依赖项和这些依赖项的依赖项，这样您将能够找到您正在导入的哪些库对 Jackson 具有传递依赖项，并且您将能够将其标记为已排除。

注意：您在此依赖关系树中真正要查找的是不同版本的 Jackson 依赖关系，因此您无需将它们全部排除。