【发布时间】：2015-04-11 09:46:52
【问题描述】：
我正在一个集群上运行一个小作业，每台机器有 15G 内存和 8G 磁盘。
作业总是陷入死锁，最后一条错误消息是：
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at org.apache.spark.storage.DiskBlockObjectWriter$TimeTrackingOutputStream$$anonfun$write$3.apply$mcV$sp(BlockObjectWriter.scala:86)
at org.apache.spark.storage.DiskBlockObjectWriter.org$apache$spark$storage$DiskBlockObjectWriter$$callWithTiming(BlockObjectWriter.scala:221)
at org.apache.spark.storage.DiskBlockObjectWriter$TimeTrackingOutputStream.write(BlockObjectWriter.scala:86)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at org.xerial.snappy.SnappyOutputStream.dumpOutput(SnappyOutputStream.java:300)
at org.xerial.snappy.SnappyOutputStream.rawWrite(SnappyOutputStream.java:247)
at org.xerial.snappy.SnappyOutputStream.write(SnappyOutputStream.java:107)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876)
at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1914)
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1575)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:350)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:751)
at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:750)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:750)
at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:746)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:746)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
到它发生时，shuffle 写入大小为 0.0B，输入大小为 3.4MB。不知道有什么操作会很快吃光整个 5G 可用磁盘空间。
此外，整个作业的存储级别被限制在 MEMORY_ONLY_SERIALIZED 并且检查点被完全禁用。
【问题讨论】：
-
如果您的数据不适合内存，则很可能会发生这种情况。此外，Spark 必须为任何类型的 shuffle 操作（例如 reduce）写入磁盘。为了验证输入大小是否导致问题，我建议您使用 smaller 输入数据集并查看是否收到此错误。
-
我已经尝试过了，它确实通过了测试。但无论哪种情况，我的数据大小都远低于总内存（300G+，而我最大的数据集约为 4G - 正如您已经在“shuffle size”指标中看到的那样）
-
Spark 还坚持以减少在节点丢失时从头重新计算的需要。
-
只看到相反的情况发生：当在慢速网络上持续存在时，如果速度更快，它仍然会重新计算。恕我直言，坚持/检查点只能手动调用。
标签： scala apache-spark persist checkpoint