IllegalArgumentException：未知消息类型：9 在读取“delta”文件时答案

【问题标题】：IllegalArgumentException: Unknown message type: 9 while reading "delta" fileIllegalArgumentException：未知消息类型：9 在读取“delta”文件时
【发布时间】：2021-10-14 06:13:38
【问题描述】：

我在我的项目中使用 3.1.2 和“delta”湖 io.delta:delta-core_2.12:1.0.0。

在读取“delta”文件时，我遇到了 IllegalArgumentException：未知消息类型：9 错误

java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 4 ($anonfun$apply$2 at DatabricksLogging.scala:77) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: java.lang.IllegalArgumentException: Unknown message type: 9  at org.apache.spark.network.shuffle.protocol.BlockTransferMessage$Decoder.fromByteBuffer(BlockTransferMessage.java:71)  at org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:80)    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)    ... 1 more 
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)
    at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135)
    at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2410)
    at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:464)
    at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:401)
    at org.apache.spark.sql.delta.catalog.DeltaTableV2.deltaLog$lzycompute(DeltaTableV2.scala:73)
    at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:177)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:355)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:305)
    at scala.Option.map(Option.scala:230)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:265)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 4 ($anonfun$apply$2 at DatabricksLogging.scala:77) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: java.lang.IllegalArgumentException: Unknown message type: 9    at org.apache.spark.network.shuffle.protocol.BlockTransferMessage$Decoder.fromByteBuffer(BlockTransferMessage.java:71)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown message type: 9     at org.apache.spark.network.shuffle.protocol.BlockTransferMessage$Decoder.fromByteBuffer(BlockTransferMessage.java:71)  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)    ... 1 more 
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)

我正在提交如下 Spark 作业

export SPARK_HOME=/spark-3.1.2-bin-hadoop3.2
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--packages org.apache.hadoop:hadoop-aws:2.9.2,io.delta:delta-core_2.12:1.0.0,org.apache.hudi:hudi-spark-bundle_2.12:0.6.0

这里有什么问题？有什么线索吗？任何高度赞赏的帮助。

【问题讨论】：

你的火花中有spark.shuffle.service.enabled吗？ books.japila.pl/apache-spark-internals/external-shuffle-service/…, waitingforcode.com/apache-spark/…
检查这个 -> issues.apache.org/jira/browse/SPARK-27780
@Alex Ott ..是的，它已启用..--conf spark.cleaner.referenceTracking.cleanCheckpoints=true \ --conf spark.cleaner.referenceTracking.blocking=true \ --conf spark.cleaner .referenceTracking.blocking.shuffle=true \ --conf spark.sql.shuffle.partitions=600 \ --conf spark.default.parallelism=850 \ --conf spark.sql.autoBroadcastJoinThreshold=-1 \ --conf spark. shuffle.service.enabled=true \
如果已启用，则意味着 shuffle 服务运行在与您的工作不同的 Spark 版本上...我在您之前的问题中发布了与 Mohana 相同的 Jira
@Alex Ott 那么我的出路应该是什么？

标签： apache-spark apache-spark-sql databricks delta-lake

【解决方案1】：

我遇到了和https://issues.apache.org/jira/browse/SPARK-33093类似的东西

Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown message type: 9
at org.apache.spark.network.shuffle.protocol.BlockTransferMessage$Decoder.fromByteBuffer(BlockTransferMessage.java:71)
at org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:81)
at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)

以下内容也帮助了我

spark.shuffle.useOldFetchProtocol=true

【讨论】：