使用 Spark streaming + Kafka 时如何修复过期批次？

Question

我正在尝试使用 foreachBatch() 从 kafka 主题中读取数据，如下所示。

def write_stream_batches(spark: SparkSession, kafka_df: DataFrame, checkpoint_location: str, kafkaconfig: dict):
    query = kafka_df.writeStream \
        .format('kafka') \
        .foreachBatch(join_kafka_streams) \
        .option('checkpointLocation', checkpoint_location) \
        .start()
    query.awaitTermination()

def join_kafka_streams(kafka_df: DataFrame, batch_id: int):
    main_df = spark.sql('select * from table where some_filter_including_partitions')
    join_df = kafka_df.join(main_df, ['key_col1', 'key_col2', 'key_col3', 'key_col4'], 'inner')
    join_df.write.format('kafka') \
        .option('kafka.bootstrap.servers', kafkaconfig['kafka_broker']) \
        .option('kafka.batch.size', kafkaconfig['kafka_batch_size']) \
        .option('retries', kafkaconfig['retries']) \
        .option('kafka.max.request.size', kafkaconfig['kafka_max_request_size']) \
        .option('kafka.max.block.ms', kafkaconfig['kafka_max_block_ms']) \
        .option('kafka.metadata.max.age.ms', kafkaconfig['kafka_metadata_max_age_ms']) \
        .option('kafka.request.timeout.ms', kafkaconfig['kafka_request_timeout_ms']) \
        .option('kafka.linger.ms', kafkaconfig['kafka_linger_ms']) \
        .option('kafka.delivery.timeout.ms', kafkaconfig['kafka_delivery_timeout_ms']) \
        .option('acks', kafkaconfig['acks']) \
        .option('kafka.compression.type', kafkaconfig['kafka_compression_type']) \
        .option('kafka.security.protocol', kafkaconfig['kafka_security_protocol']) \
        .option('kafka.sasl.jaas.config', oauth_config) \
        .option('kafka.sasl.login.callback.handler.class', kafkaconfig['kafka_sasl_login_callback_handler_class']) \
        .option('kafka.sasl.mechanism', kafkaconfig['kafka_sasl_mechanism']) \
        .option('topic', topic_name) \
        .save()

kafka_df里面的数据是250万左右，main_df里面的数据是400万 当我开始作业时，连接结果包含 900k 条记录，加载 100k 条记录后，作业在运行 25 分钟后失败并出现以下异常。

py4j.protocol.Py4JJavaError: An error occurred while calling o500.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 4 times, most recent failure: Lost task 0.3 in stage 15.0 (TID 66, 100.67.55.233, executor 0): kafkashaded.org.apache.kafka.common.errors.TimeoutException: Expiring 13 record(s) for x1-dev-asw32-edr-02a1-ba87-332c7da70fc1-topic_name:130000 ms has passed since batch creation
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2519)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2466)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2460)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2460)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:999)
    at org.apache.spark.sql.kafka010.KafkaWriter$.write(KafkaWriter.scala:70)
    at org.apache.spark.sql.kafka010.KafkaSourceProvider.createRelation(KafkaSourceProvider.scala:180)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:91)
    at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:192)
    at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:158)
    at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:157)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:845)
    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:999)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:437)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:421)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:251)
    at java.lang.Thread.run(Thread.java:748)

我正在我的数据块集群上提交作业。 上面的异常是由于会话超时还是因为内存问题？ 谁能告诉我是什么导致了异常？ 任何帮助深表感谢。

Answer 1

尝试减少 kafka.batch.size 并增加 kafka.request.timeout.ms。

看问题：

Writing large DataFrame from PySpark to Kafka runs into timeout

Spark-kafka : org.apache.kafka.common.errors.TimeoutException while writing stream from Spark

Kafka producer TimeoutException: Expiring 1 record(s)