从 AWS Glue 作业中的粘合目录读取数据

Question

我有连接到 Kafka 的胶水作业。数据保存在胶水数据目录中，在作业中我需要阅读它。我接下来尝试了：

df = glueContext.create_data_frame.from_catalog(database = "t-kafka-db", table_name = "t-kafka", transformation_ctx = "df", connectionType="kafka")
df = glueContext.create_dynamic_frame_from_catalog(database="t-kafka-db",table_name="t-kafka",transformation_ctx="df")
df = glueContext.create_dynamic_frame.from_catalog(database = "t-kafka-db",table_name = "t-kafka",transformation_ctx = "df",additional_options = {"startingOffsets": "earliest", "inferSchema": "false"})

它们都返回相同的错误：

An error occurred while calling o67.getCatalogSource. We don't support this connection type: KAFKA

帮忙解决

Answer 1

AWS Glue 现在支持从 Kafka 读取。请参考this，其中讨论了为流式源创建数据目录表。