无法从/向融合传输工件 io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT (${confluent.maven.repo})

Question

我是第一次尝试 Kafka 连接，我想将 SAP S/4 HANA 连接到 Hive。我已经使用这个创建了 SAP S/4 源 Kafka 连接器：

https://github.com/SAP/kafka-connect-sap

但是，我无法创建 HDFS 接收器连接器。该问题与 pom 文件有关。

我试过mvn clean package。 但是，我收到了这个错误：

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project io.confluent:kafka-connect-hdfs:[unknown-version] (/home/dev_dice_sa/test-dicestreaming/config/kafka-connect-hdfs/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for io.confluent:kafka-connect-hdfs:[unknown-version]: Could not transfer artifact io.confluent:kafka-connect-storage-common-parent:pom:6.0.0-SNAPSHOT from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 22, column 13: Cannot access ${confluent.maven.repo} using the registered transporter factories: WagonTransporterFactory: Unsupported transport protocol -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

这是我的 pom.xml：

    <?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright 2018 Confluent Inc.
  ~
  ~ Licensed under the Confluent Community License (the "License"); you may not use
  ~ this file except in compliance with the License.  You may obtain a copy of the
  ~ License at
  ~
  ~ http://www.confluent.io/confluent-community-license
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
  ~ WARRANTIES OF ANY KIND, either express or implied.  See the License for the
  ~ specific language governing permissions and limitations under the License.
  -->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-connect-storage-common-parent</artifactId>
        <version>6.0.0-SNAPSHOT</version>
    </parent>

    <artifactId>kafka-connect-hdfs</artifactId>
    <packaging>jar</packaging>
    <name>kafka-connect-hdfs</name>
    <organization>
        <name>Confluent, Inc.</name>
        <url>http://confluent.io</url>
    </organization>
    <url>http://confluent.io</url>
    <description>
        A Kafka Connect HDFS connector for copying data between Kafka and Hadoop HDFS.
    </description>

    <licenses>
        <license>
            <name>Confluent Community License</name>
            <url>http://www.confluent.io/confluent-community-license</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <scm>
        <connection>scm:git:git://github.com/confluentinc/kafka-connect-hdfs.git</connection>
        <developerConnection>scm:git:git@github.com:confluentinc/kafka-connect-hdfs.git</developerConnection>
        <url>https://github.com/confluentinc/kafka-connect-hdfs</url>
        <tag>HEAD</tag>
    </scm>

    <properties>
        <confluent.maven.repo>http://packages.confluent.io/maven/</confluent.maven.repo>
        <apacheds-jdbm1.version>2.0.0-M2</apacheds-jdbm1.version>
        <kafka.connect.maven.plugin.version>0.11.1</kafka.connect.maven.plugin.version>
    </properties>

    <repositories>
        <repository>
            <id>confluent</id>
            <name>Confluent</name>
            <url>${confluent.maven.repo}</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>connect-api</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>connect-json</artifactId>
            <version>${kafka.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-common</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-core</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-format</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-partitioner</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-wal</artifactId>
            <version>${confluent.version}</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-connect-storage-hive</artifactId>
            <version>${confluent.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty-all</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.github.spotbugs</groupId>
            <artifactId>spotbugs-annotations</artifactId>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-minicluster</artifactId>
            <version>${hadoop.version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.netty</groupId>
                    <artifactId>netty-all</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-minikdc</artifactId>
            <version>${hadoop.version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <!-- exclude this from hadoop minikdc as the minikdc depends on
                    the apacheds-jdbm1 bundle, which is not available in maven central-->
                    <groupId>org.apache.directory.jdbm</groupId>
                    <artifactId>apacheds-jdbm1</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <!-- add this to satisfy the dependency requirement of apacheds-jdbm1-->
            <groupId>org.apache.directory.jdbm</groupId>
            <artifactId>apacheds-jdbm1</artifactId>
            <version>${apacheds-jdbm1.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>io.confluent</groupId>
                <version>${kafka.connect.maven.plugin.version}</version>
                <artifactId>kafka-connect-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>kafka-connect</goal>
                        </goals>
                        <configuration>
                            <title>Kafka Connect HDFS</title>
                            <documentationUrl>https://docs.confluent.io/${project.version}/connect/connect-hdfs/docs/index.html</documentationUrl>
                            <description>
                                The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.

                                The connector periodically polls data from Kafka and writes them to HDFS. The data from each Kafka topic is partitioned by the provided partitioner and divided into chunks. Each chunk of data is represented as an HDFS file with topic, Kafka partition, start and end offsets of this data chunk in the filename. If no partitioner is specified in the configuration, the default partitioner which preserves the Kafka partitioning is used. The size of each data chunk is determined by the number of records written to HDFS, the time written to HDFS and schema compatibility.

                                The HDFS connector integrates with Hive and when it is enabled, the connector automatically creates an external Hive partitioned table for each Kafka topic and updates the table according to the available data in HDFS.
                            </description>

                            <supportProviderName>Confluent, Inc.</supportProviderName>
                            <supportSummary>Confluent supports the HDFS sink connector alongside community members as part of its Confluent Platform offering.</supportSummary>
                            <supportUrl>https://docs.confluent.io/current/</supportUrl>
                            <supportLogo>logos/confluent.png</supportLogo>

                            <ownerUsername>confluentinc</ownerUsername>
                            <ownerType>organization</ownerType>
                            <ownerName>Confluent, Inc.</ownerName>
                            <ownerUrl>https://confluent.io/</ownerUrl>
                            <ownerLogo>logos/confluent.png</ownerLogo>

                            <dockerNamespace>confluentinc</dockerNamespace>
                            <dockerName>cp-kafka-connect</dockerName>
                            <dockerTag>${project.version}</dockerTag>

                            <componentTypes>
                                <componentType>sink</componentType>
                            </componentTypes>

                            <tags>
                                <tag>hadoop</tag>
                                <tag>hdfs</tag>
                                <tag>hive</tag>
                            </tags>

                            <confluentControlCenterIntegration>true</confluentControlCenterIntegration>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <compilerArgs>
                        <arg>-Xlint:all,-deprecation,-processing</arg>
                        <arg>-Werror</arg>
                    </compilerArgs>
                    <showWarnings>true</showWarnings>
                    <showDeprecation>false</showDeprecation>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptors>
                        <descriptor>src/assembly/development.xml</descriptor>
                        <descriptor>src/assembly/package.xml</descriptor>
                    </descriptors>
                    <attach>false</attach>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <reuseForks>false</reuseForks>
                    <forkCount>1</forkCount>
                    <systemPropertyVariables>
                        <datanucleus.schema.autoCreateAll>true</datanucleus.schema.autoCreateAll>
                    </systemPropertyVariables>
                    <additionalClasspathElements>
                        <!--puts hive-site.xml on classpath - w/o HMS tables are not created-->
                        <additionalClasspathElement>src/test/resources/conf</additionalClasspathElement>
                    </additionalClasspathElements>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-checkstyle-plugin</artifactId>
                <executions>
                    <execution>
                        <id>validate</id>
                        <phase>validate</phase>
                        <configuration>
                            <suppressionsLocation>checkstyle/suppressions.xml</suppressionsLocation>
                        </configuration>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>.</directory>
                            <includes>
                                <include>derby.log</include>
                                <include>metastore_db/</include>
                            </includes>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
        </plugins>

        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
        </resources>
    </build>

    <profiles>
        <profile>
            <id>standalone</id>
            <build>
                <plugins>
                    <plugin>
                        <artifactId>maven-assembly-plugin</artifactId>
                        <configuration>
                            <descriptors>
                                <descriptor>src/assembly/standalone.xml</descriptor>
                            </descriptors>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>
</project>

我已经查看了这个链接：

https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ

但是，我无法遵循这一点。请帮帮我。

Answer 1

我建议你下载已经包含 HDFS Connect 的现有 Confluent 平台

否则，检查一个发布版本而不是只检查主分支来构建项目。