多节点集群中的 SparkJob：WARN TaskSetManager：在阶段 0.0 中丢失任务 0.0：java.io.FileNotFoundException

Question

我刚刚建立了一个 Spark 多节点集群。我的集群由一台 iMac 和几个 Raspberry 组成，所有这些都通过以太网链接，并且可以通过 ssh 无密码访问彼此。

我尝试执行的 Spark 命令是：

spark-submit --master spark://10.0.0.20:7077 rdd/WordCount.py

我的从节点是： 10.0.0.10 10.0.0.11

cod 退出，并在日志的以下 sn-p 中显示错误：

21/01/13 13:54:38 INFO Utils: Fetching ftp://myuser:mypassword@my-NAS-IP:21/Projects/Corso-Spark/word_count.text to /private/var/folders/0s/gkptv9tn6h100zv3m17ctsd400yjj9/T/spark-5c31c0e5-6385-4945-928a-3883332189ac/userFiles-abf87986-8096-4bf4-a9e5-44fc6a3d5676/fetchFileTemp8028573497969255747.tmp
...    
21/01/13 13:54:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.0.0.10, executor 1): java.io.FileNotFoundException: File file:/private/var/folders/0s/gkptv9tn6h100zv3m17ctsd400yjj9/T/spark-5c31c0e5-6385-4945-928a-3883332189ac/userFiles-abf87986-8096-4bf4-a9e5-44fc6a3d5676/word_count.text does not exist
            at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
            at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
            at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
            at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
            at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
            at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
            at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:109)
            at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
            at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:282)
            at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:281)
            at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:239)
            at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:96)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
            at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
            at org.apache.spark.scheduler.Task.run(Task.scala:127)
            at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
            at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:834)

python里面的word_count.text文件是通过FTP@这个URL检索到的：

"ftp://myuser:mypassword@my-NAS-IP:21/Projects/Corso-Spark/word_count.text"

显然，文件是在master里面的 /private/var/folders/0s/gkptv9tn6h100zv3m17ctsd400yjj9/T/spark-5c31c0e5-6385-4945-928a-3883332189ac/userFiles-abf87986-8096-4bf4-a9e5-44fc6a3d5676 然后Spark尝试检索目录从站上相同目录中的相同文件。当然，在 slaves 中 Spark 找不到路径。 为什么？

• 作为进一步的测试，我创建了路径 /private/var/folders/0s/gkptv9tn6h100zv3m17ctsd400yjj9/T/spark-5c31c0e5-6385-4945-928a-3883332189ac/userFiles-abf87986-8096-4bf3df766e在从机上并手动放置文件，但我无法摆脱错误。

有人可以帮忙吗？

提前谢谢你。

Answer 1

[已解决]：我在网上找到的所有教程都没有说的是，您必须安装完全相同的路径，输入文件将在每个工作人员的主服务器上获取。