从 Docker (ksqlDB) 连接到主机上的 Kafka

Question

我正在从此处找到的 docker-compor 运行 ksqldb-server https://ksqldb.io/quickstart.html#quickstart-content

我的 kafka 引导服务器在同一台虚拟机上以标准单独模式运行。 我可以通过控制台使用者查看一个主题中的消息：

sudo kafka-avro-console-consumer --from-beginning --bootstrap-server localhost:9092 --topic source-air-input  --property print.key=true --max-messages 2 

不幸的是，从 docker 运行 ksql 给了我这个错误。

ksqldb-server    | [2021-07-15 23:12:58,772] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
ksqldb-server    | java.lang.RuntimeException: Failed to get Kafka cluster information
ksqldb-server    |      at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:107)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:624)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:152)
ksqldb-server    |      at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
ksqldb-server    | Caused by: java.util.concurrent.TimeoutException
ksqldb-server    |      at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
ksqldb-server    |      at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
ksqldb-server    |      at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:105)

我的 docker-compose.yml 如下。

 ---
version: '3.9'

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.18.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: host.docker.internal:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.18.0
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

我尝试了许多可能的地址配置，但均未成功。 可能有什么问题？ 我尝试了这个问题From inside of a Docker container, how do I connect to the localhost of the machine? 的建议，但没有成功。

Answer 1

修改Kafka的server.properties

listeners=PLAINTEXT_DOCKER://:29092,PLAINTEXT_LOCAL://localhost:9092

advertised.listeners=PLAINTEXT_DOCKER://host.docker.internal:29092,PLAINTEXT_LOCAL://localhost:9092

listener.security.protocol.map=PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_LOCAL:PLAINTEXT
inter.broker.listener.name=PLAINTEXT_LOCAL

像 to point at the host 那样更新您的 Compose，而不是它本身

version: '3.9'

services:

  # TODO: add schema-registry

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.18.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
       KSQL_BOOTSTRAP_SERVERS: host.docker.internal:29092
       ...

（在 Mac 上测试），获取 KSQL 的 /info 端点

http :8088/info
HTTP/1.1 200 OK
content-length: 133
content-type: application/json

{
    "KsqlServerInfo": {
        "kafkaClusterId": "ZH2-h1W_SaivCW0qa8DQGA",
        "ksqlServiceId": "default_",
        "serverStatus": "RUNNING",
        "version": "0.18.0"
    }
}

如果Kafka是远程服务器，则将上面所有的host.docker.internal替换为机器的外部主机名/IP