带有 Kafka Connect 的 REST API 源连接器 - 是否支持 HTTPS 调用？答案

【问题标题】：REST API Source connector with Kafka Connect - Are HTTPs calls supported?带有 Kafka Connect 的 REST API 源连接器 - 是否支持 HTTPS 调用？
【发布时间】：2020-04-03 06:06:52
【问题描述】：

我正在尝试使用https://github.com/llofberg/kafka-connect-rest 的社区贡献连接器来轮询来自 HTTPS REST 端点的数据。我的连接器的配置如下。下面：

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{"name": "rest-source-connector", "config":{"key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.storage.StringConverter","connector.class": "com.tm.kafka.connect.rest.RestSourceConnector","tasks.max": "1", "rest.source.poll.interval.ms": "300000", "rest.source.method":"GET","rest.source.url": "https://<insert_endpoint>","rest.source.payload.converter.class": "com.tm.kafka.connect.rest.converter.StringPayloadConverter","rest.source.headers": "Content-Type:application/json,Accept:application/json","rest.source.topic.selector":"com.tm.kafka.connect.rest.selector.SimpleTopicSelector","rest.source.destination.topics": "test-topic"}}'

当连接器运行并尝试从端点提取数据时，我看到以下错误：

ERROR HTTP call execution failed null (com.tm.kafka.connect.rest.RestSourceTask)
org.apache.kafka.connect.errors.RetriableException
    at com.tm.kafka.connect.rest.http.executor.OkHttpRequestExecutor.execute(OkHttpRequestExecutor.java:67)
    at com.tm.kafka.connect.rest.RestSourceTask.poll(RestSourceTask.java:89)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:265)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
    at okio.RealBufferedSource.require(RealBufferedSource.java:65)
    at okio.GzipSource.consumeHeader(GzipSource.java:114)
    at okio.GzipSource.read(GzipSource.java:73)
    at okio.RealBufferedSource.select(RealBufferedSource.java:100)
    at okhttp3.internal.Util.bomAwareCharset(Util.java:467)
    at okhttp3.ResponseBody.string(ResponseBody.java:181)
    at com.tm.kafka.connect.rest.http.executor.OkHttpRequestExecutor.execute(OkHttpRequestExecutor.java:64)

连接器是否支持 HTTPS 调用，是否需要提供一些配置才能发出 HTTPS 请求？

【问题讨论】：

你试过不使用 GZIP 源吗？打开 Github 问题...？

标签： rest apache-kafka apache-kafka-connect confluent-platform

【解决方案1】：

我会将您的 json 从命令行移动到 .json 文件中，并使用该文件注册连接器

下面的例子

curl -X POST -H "Content-Type: application/json" --data @filename.json http://localhost:8083/connectors

【讨论】：