minikube 中 Kubernetes 上的 Kafka 无法正常工作答案

【问题标题】：Kafka on Kubernetes in minikube not workingminikube 中 Kubernetes 上的 Kafka 无法正常工作
【发布时间】：2020-08-04 01:52:14
【问题描述】：

我正在尝试在 Minikube 上设置 Kafka，这是一个非常基本的设置。我无法验证 Kafka 和 Zookeeper 是否已正确设置，因为 kafkacat 失败。

这是我的配置：

动物园管理员


kind: Deployment
apiVersion: apps/v1
metadata:
  name: zookeeper-deployment-1
spec:
  selector:
    matchLabels:
      app: zookeeper-1
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
        - name: zoo1
          image: digitalwonderland/zookeeper
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_ID
              value: "1"
            - name: ZOOKEEPER_SERVER_1
              value: zoo1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
    - name: client
      port: 2181
      protocol: TCP
    - name: follower
      port: 2888
      protocol: TCP
    - name: leader
      port: 3888
      protocol: TCP
  selector:
    app: zookeeper-1

卡夫卡

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kafka-broker0
spec:
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
        - name: kafka
          image: wurstmeister/kafka
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_ADVERTISED_PORT
              value: "9092"
            - name: KAFKA_ADVERTISED_HOST_NAME
              value: kafka-service
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: zoo1:2181
            - name: KAFKA_BROKER_ID
              value: "0"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
    - port: 9092
      name: kafka-port
      protocol: TCP
  selector:
    app: kafka
    id: "0"

豆荚

kafka-broker0-6885746967-6vktz            1/1     Running   0          5m20s
zookeeper-deployment-1-7f5bb9785f-7pplk   1/1     Running   0          5m25s

svc

kafka-service          ClusterIP      10.99.226.129    <none>           9092/TCP                                    6m30s
zoo1                   ClusterIP      10.96.140.187    <none>           2181/TCP,2888/TCP,3888/TCP                  6m35s

kafkacat 日志

✗ kafkacat -L -b kafka-service:9092 -d broker
%7|1596239513.610|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1596239513.610|BROKER|rdkafka#producer-1| [thrd:app]: kafka-service:9092/bootstrap: Added new broker with NodeId -1
%7|1596239513.610|BRKMAIN|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: Enter main broker thread
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:app]: kafka-service:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1596239513.610|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.0 (0x10400ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: Received CONNECT op
%7|1596239513.610|STATE|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1596239513.610|STATE|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596239513.610|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596239513.614|BROKERFAIL|rdkafka#producer-1| [thrd:kafka-service:9092/bootstrap]: kafka-service:9092/bootstrap: failed: err: Local: Host resolution failure: (errno: Bad address)

NODEPORT 更新

✗ kafkacat -L -b kafka-service:30236 -d 经纪人

%7|1596476848.078|STATE|rdkafka#producer-1| [thrd:kafka-service:30236/bootstrap]: kafka-service:30236/bootstrap: Broker changed state CONNECT -> DOWN
%7|1596476848.078|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 46ms: application metadata request
%7|1596476848.078|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 46ms: application metadata request
%7|1596476849.065|CONNECT|rdkafka#producer-1| [thrd:app]: Cluster connection already in progress: application metadata request
%7|1596476849.065|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
% ERROR: Failed to acquire metadata: Local: Broker transport failure

minikube ip ✗ kafkacat -L -b 192.168.64.2:30236 -d 经纪人

|1596476908.164|BRKMAIN|rdkafka#producer-1| [thrd:192.168.64.2:30236/bootstrap]: 192.168.64.2:30236/bootstrap: Enter main broker thread
%7|1596476908.164|CONNECT|rdkafka#producer-1| [thrd:app]: 192.168.64.2:30236/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1596476908.164|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.0 (0x10400ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1596476908.164|CONNECT|rdkafka#producer-1| [thrd:192.168.64.2:30236/bootstrap]: 192.168.64.2:30236/bootstrap: Received CONNECT op
%7|1596476908.164|STATE|rdkafka#producer-1| [thrd:192.168.64.2:30236/bootstrap]: 192.168.64.2:30236/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1596476908.164|CONNECT|rdkafka#producer-1| [thrd:192.168.64.2:30236/bootstrap]: 192.168.64.2:30236/bootstrap: broker in state TRY_CONNECT connecting
%7|1596476908.164|STATE|rdkafka#producer-1| [thrd:192.168.64.2:30236/bootstrap]: 192.168.64.2:30236/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1596476908.164|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request

✗ kafkacat -L -b localhost:30236 -d 经纪人

%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Received CONNECT op
%7|1596477098.454|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.4.0 (0x10400ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1596477098.454|STATE|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: broker in state TRY_CONNECT connecting
%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596477098.454|STATE|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596477098.454|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1596477098.460|CONNECT|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Connecting to ipv4#127.0.0.1:30236 (plaintext) with socket 9
%7|1596477098.461|BROKERFAIL|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: failed: err: Local: Broker transport failure: (errno: Connection refused)
%7|1596477098.461|STATE|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Broker changed state CONNECT -> DOWN
%7|1596477098.461|STATE|rdkafka#producer-1| [thrd:localhost:30236/bootstrap]: localhost:30236/bootstrap: Broker changed state DOWN -> INIT
%7|1596477098.461|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 43ms: application metadata request
%7|1596477098.461|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 42ms: application metadata request

【问题讨论】：

你从哪里运行kafkacat？因为如果它是您自己的工作站，这就是它无法解析 kafka-service 的原因——该名称仅存在于集群中
添加到 @mdaniel ，暴露你的 kafka 服务和 Nodeport 服务，然后你应该可以从你的工作站访问。
我正在从本地（mac）运行kafkacat，将其公开为 NodePort，因为下面的答案给出了我已更新的有问题的错误。

标签： kubernetes apache-kafka apache-zookeeper kafkacat

【解决方案1】：

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  type: NodePort
  ports:
    - port: 9092
      nodePort: 30236
      name: kafka-port
      protocol: TCP
  selector:
    app: kafka
    id: "0"

我从来没有用过kafkacat，但是如果它是一个安装在你主机上的cli，而不是在另一个容器内，你现在可以这样使用它：

✗ kafkacat -L -b localhost:30236 -d broker

localhost，或者你的 kubernetes 节点的 ip。

使用服务默认是 ClusterIp 类型，并且只能在 kubernetes 集群内部访问

【讨论】：

感谢您的回答，我试过了，但由于更新了相关错误而失败。