向弹性搜索公开的 Spark 流数据帧答案

【问题标题】：Spark streaming data frame public to elasticsearch向弹性搜索公开的 Spark 流数据帧
【发布时间】：2021-05-24 11:23:34
【问题描述】：

SparkSession 配置：

val spark = SparkSession.builder()
    .config("es.index.auto.create", "true")
    .config("es.nodes", "1.2.3.4")
    .config("es.port", "9200")
    .config("es.nodes.wan.only", "true")
    .master("local[*]")
    .getOrCreate()

保存查询：

val query = tableDF
    .writeStream
    .format("es")
    .outputMode("append")
    .option("checkpointLocation", "/tmp/checkpoint")
    .option("resource", "spark/speeches")
    .start()

query.awaitTermination()

错误

Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(Lorg/apache/spark/sql/SparkSession;Lorg/apache/spark/sql/execution/QueryExecution;Lscala/Function0;)Ljava/lang/Object;

我的依赖：

compile group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.0.1'
compile group: 'org.apache.spark', name: 'spark-core_2.12', version: '3.0.1'
compile group: 'org.apache.spark', name: 'spark-streaming_2.12', version: '3.0.1'
implementation group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.11.1'

【问题讨论】：

标签： apache-spark elasticsearch spark-streaming

【解决方案1】：

替换依赖

implementation group: 'org.elasticsearch', name: 'elasticsearch-hadoop', version: '7.11.1'

到

implementation group: 'org.elasticsearch', name: 'elasticsearch-spark-20_2.11', version: '7.11.1'

您应该更改其他依赖项以使用 scala 2.11

【讨论】：