【发布时间】：2023-02-26 04:00:53
【问题描述】：
来自Debezium 官方页面，这张图片显示多个 Debezium 连接器可以连接到同一个 Kafka。
所以我有 2 个数据库、2 个 Debeziums、1 个运行在 docker-compose 中的 Kafka，但似乎只有 1 个 debezium 将更新发送到 kafka（从 kafdrop 观看）。
这是我的 docker-compose 文件：
version: '3.6'
services:
hero_db:
image: postgres:14
restart: always
environment:
POSTGRES_PASSWORD: postgrespassword
ports:
- '5432:5432'
expose:
- '5432'
command: [ "postgres", "-c", "wal_level=logical" ]
volumes:
- hero_db_data:/var/lib/postgresql/data
villian_db:
image: postgres:14
restart: always
environment:
POSTGRES_PASSWORD: postgrespassword
ports:
- '2345:2345'
expose:
- '2345'
command: [ "postgres", "-c", "wal_level=logical" ]
volumes:
- villian_db_data:/var/lib/postgresql/data
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 22181:2181
kafka:
image: confluentinc/cp-kafka:5.3.1
ports:
- 29092:29092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
depends_on:
- zookeeper
kafdrop:
image: obsidiandynamics/kafdrop
container_name: kafdrop
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "kafka:9092"
JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
depends_on:
- kafka
hero_debezium:
image: debezium/connect:1.9
ports:
- 8083:8083
expose:
- '8083'
environment:
CONFIG_STORAGE_TOPIC: hero_configs
OFFSET_STORAGE_TOPIC: hero_offsets
STATUS_STORAGE_TOPIC: hero_statuses
BOOTSTRAP_SERVERS: kafka:9092
depends_on: [ zookeeper, kafka, hero_db ]
villian_debezium:
image: debezium/connect:1.9
ports:
- 8084:8083
expose:
- '8084'
environment:
CONFIG_STORAGE_TOPIC: villian_configs
OFFSET_STORAGE_TOPIC: villian_offsets
STATUS_STORAGE_TOPIC: villian_statuses
BOOTSTRAP_SERVERS: kafka:9092
depends_on: [ zookeeper, kafka, villian_db ]
volumes:
hero_db_data:
villian_db_data:
这是 hero_dbz 和 villian_dbz 的 json 中的 debezium 配置文件：
hero_dbz.json
{
"name": "hero-postgresql-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"database.hostname": "hero_db",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgrespassword",
"database.dbname": "postgres",
"database.server.name": "hero_server",
"table.include.list": "public.heroes",
"table.whitelist": "public.heroes",
"topic.prefix": "topic_heroes"
}
}
villian_dbz.json
{
"name": "villian-postgresql-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"plugin.name": "pgoutput",
"database.hostname": "villian_db",
"database.port": "2345",
"database.user": "postgres",
"database.password": "postgrespassword",
"database.dbname": "postgres",
"database.server.name": "villian_server",
"table.include.list": "public.villians",
"table.whitelist": "public.villians",
"topic.prefix": "topic_villian"
}
}
我用这些命令配置了 hero_dbz 和 villian_dbz：curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 127.0.0.1:8083/connectors/ --data "@hero_dbz.json"curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 127.0.0.1:8084/connectors/ --data "@villian_dbz.json"
这是来自 Kafdrop 的屏幕，只显示来自 hero_db (hero_server.public.heroes) 的数据，但没有显示来自 villian_db 的数据。
【问题讨论】：
标签： docker apache-kafka debezium debezium-engine