【发布时间】:2018-03-26 22:13:43
【问题描述】:
我正在尝试使用 SMT 功能添加密钥以将其用作 ES 文档的文档 ID,但它不起作用。我正在使用 confluent es 连接器。配置文件如下: connect-standalone.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
连接器配置:
#Connector name
name=logs-=false
#Connector class
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=test
topic.index.map=test:activity
connection.url=http://localhost:9200
type.name=Activity
#ignore key and schema
key.ignore=false
schema.ignore=true
transforms=InsertKey,ExtractId
transforms.InsertKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.InsertKey.fields=recordId
transforms.ExtractId.type=org.apache.kafka.connect.transforms.ExtractField$Key
transforms.ExtractId.field=recordId
我正在向 kafka 发送以下消息:
{"recordId":"999","activity":"test","description":"test Cont"}
在接收器连接器中出现此错误:
org.apache.kafka.connect.errors.ConnectException: Key is used as document id and can not be null.
at io.confluent.connect.elasticsearch.DataConverter.convertKey(DataConverter.java:56)
at io.confluent.connect.elasticsearch.DataConverter.convertRecord(DataConverter.java:86)
at io.confluent.connect.elasticsearch.ElasticsearchWriter.write(ElasticsearchWriter.java:210)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put(ElasticsearchSinkTask.java:119)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:384)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[2017-10-15 01:37:41,789] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask:405)
[2017-10-15 01:37:41,789] INFO WorkerSinkTask{id=logs-=false-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask:262)
[2017-10-15 01:37:41,804] ERROR Task logs-=false-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:142)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:406)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:240)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:172)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:143)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
请推荐!一段时间以来一直在尝试修复它。非常感谢任何帮助...在此先感谢
【问题讨论】:
-
您能否修改您的问题以包含您的源消息之一的示例?
-
它包含在消息中...我正在使用控制台生产者 {"recordId":"999","activity":"test","description":"test Cont"}.. 谢谢@RobinMoffatt
-
它适用于最新版本的 kafka ..kafka_2.11-0.11.0.1
-
我遇到了与 kafka 0.10.1 完全相同的问题。我无法更改我正在使用的 Kafka 版本.. 在您切换到 kafka_2.11-0.11.0.1 之前找出发生了什么运气好吗?
-
我会回答我自己的问题:transforms 功能是在 0.10.2 中引入的 ...
标签: elasticsearch apache-kafka apache-kafka-connect confluent-platform