如何将连接器安装到 apache kafka connect 的 docker 映像

Question

我正在使用https://hub.docker.com/r/1ambda/kafka-connect/ 在 ubuntu 上安装 docker kafka connect。我可以运行它，但是我无法在其中安装更多连接器。

我尝试过的

1-我尝试将连接器文件从我的机器复制到 docker 容器连接器文件夹并重新启动，但是 api，http://localhost:8080/connectors 给出了空数组。

2-还在 connect-distributed.properties 中添加了插件路径。 如何做到这一点任何想法？？

Answer 1

不使用 Confluent Hub 的解决方案

第 1 步：构建您自己的 Kafka-Connect 映像

您的目录应如下所示：

my-directory /
  Dockerfile
  plugins /
    my-connector /
       <connector-jars>

Dockerfile：

FROM confluentinc/cp-kafka-connect-base:latest
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
ENV CONNECT_PLUGIN_PATH="/opt/kafka/plugins"
USER 1001

运行以下命令

cd /my-directory
sudo docker build . -t my-connector-image-name

第 2 步：为您的 kafka-connect 容器使用创建的映像

version: '3'
services:
        zookeeper:
                container_name: 'zookeeper'
                image: 'bitnami/zookeeper:latest'
                ports:
                        - '2181:2181'
                environment:
                        - ALLOW_ANONYMOUS_LOGIN=yes
        kafka:
                image: 'bitnami/kafka:latest'
                container_name: 'kafka'
                ports:
                        - '9092:9092'
                environment:
                        - KAFKA_BROKER_ID=1
                        - KAFKA_LISTENERS=PLAINTEXT://:9092
                        - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.178.70:9092
                        - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
                        - ALLOW_PLAINTEXT_LISTENER=yes
                depends_on:
                        - zookeeper
        kafka-connect:
                image: 'my-connector-image-name:latest'
                container_name: 'kafka-connect'
                ports:
                        - '8083:8083'
                environment:
                        - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
                        - CONNECT_REST_PORT=8083
                        - CONNECT_GROUP_ID=quickstart
                        - CONNECT_CONFIG_STORAGE_TOPIC=quickstart-config
                        - CONNECT_OFFSET_STORAGE_TOPIC=quickstart-offsets
                        - CONNECT_STATUS_STORAGE_TOPIC=quickstart-status
                        - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_REST_ADVERTISED_HOST_NAME=localhost
                depends_on:
                        - kafka

第 3 步获取可用的连接器插件

curl localhost:8083/connector-plugins | json_pp

关于我的 Kafka 配置的旁注：我在 Linux 中托管 Docker 我分配了 IP 192.168.178.70 的虚拟机。

Answer 2

该图像似乎没有更新为与 plugin.path 一起使用，因为它使用的是 Kafka 0.10

您可以使用 confluent-hub 安装连接器，就像我在 Connect 容器中记录的那样

https://github.com/OneCricketeer/apache-kafka-connect-docker

Answer 3

1-我尝试将连接器文件从我的机器复制到 docker 容器连接器文件夹并重新启动，但是 api，http://localhost:8080/connectors 给出了空数组。

localhost:8083/connectors ，将为您提供活动连接器的列表。 如果你需要检查可用插件列表，你应该点击 localhost:8083/connector-plugins

curl localhost:8083/connector-plugins
[{"class":"io.confluent.connect.activemq.ActiveMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.ibm.mq.IbmMQSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jdbc.JdbcSinkConnector","type":"sink","version":"5.5.1"},
{"class":"io.confluent.connect.jdbc.JdbcSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.jms.JmsSourceConnector","type":"source","version":"5.5.1"},{"class":"io.confluent.connect.s3.S3SinkConnector","type":"sink","version":"5.5.1"},{"class":"io.confluent.connect.storage.tools.SchemaSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"io.confluent.kafka.connect.datagen.DatagenConnector","type":"source","version":"null"},{"class":"org.apache.kafka.connect.file.FileStreamSinkConnector","type":"sink","version":"5.5.1-ccs"}, 
{"class":"org.apache.kafka.connect.file.FileStreamSourceConnector","type":"source","version":"5.5.1-ccs"},{"class":"org.apache.kafka.connect.mirror.MirrorCheckpointConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorHeartbeatConnector","type":"source","version":"1"},{"class":"org.apache.kafka.connect.mirror.MirrorSourceConnector","type":"source","version":"1"}]

此外，需要将新的连接器添加到所有工作节点的上述插件路径中，并且所有工作节点都需要重新启动

要进一步排除故障，您可以尝试将日志级别提高到 DEBUG 并检查启动日志是否正在遍历连接器路径 您将看到具有以下格式的日志：

DEBUG Loading plugin urls: ......

希望这有助于解决您的问题:)