KafkaIO 连接器/Apache Beam 转换“go”SDK 可用吗？

Question

我正在使用 Apache Beam “go” SDK 构建数据摄取管道。

我的管道是使用来自 Kafka 队列的数据并将数据持久化到 Google Cloud Bigtable（和/或另一个 Kafka 主题）。

到目前为止，我还没有找到用“go”编写的 Kafka IO 连接器（也称为 Apache I/O 转换）（但是我找到了 java 版本）。

这里是支持的 Apache Beam 内置 I/O 转换的链接： https://beam.apache.org/documentation/io/built-in/

我正在寻找以下 Java 代码的“go”等价物：

    pipeline.apply("kafka_deserialization", KafkaIO.<String, String>read()
        .withBootstrapServers(KAFKA_BROKER)
        .withTopic(KAFKA_TOPIC)
        .withConsumerConfigUpdates(CONSUMER_CONFIG)
        .withKeyDeserializer(StringDeserializer.class)
        .withValueDeserializer(StringDeserializer.class))

您有关于 KafkaIO 连接器“go”SDK/库的可用性的任何信息吗？

Answer 1

@cricket_007 如果您也好奇，我收到了来自 Apache Beam 团队的 Robert Burke (rebo@google.com) 的以下更新：

There presently isn't a Kafka transform for Go. 

The Go SDK is still experimental, largely due to scalable IO support, which is why the Go SDK isn't represented in the built-in io page.

There's presently no way for an SDK user to write a Streaming source in the Go SDK, since there's no mechanism for a DoFn to "self terminate" bundles, such as to allow for scalability and windowing from streaming sources. 

However, SplittableDoFns are on their way, and will eventually be the solution for writing these.

At present, the Beam Go SDK IOs haven't been tested and vetted for production use. Until the initial SplittableDoFn support is added to the Go SDK, Batch transforms cannot split, and can't scale beyond a single worker thread. This batch version should land in the next few months, and the streaming version land a few months after that, after which a Kafka IO can be developed. 

I wish I had better news for you, but I can say progress is being made.

Robert Burke