df.show() 不工作 - Py4JJavaError: An error occurred while calling o95.showString

Question

我正在尝试解决 O'Reilly 的 Learning Spark 书中的问题。

以下部分代码运行良好

from pyspark.sql.types import *
from pyspark.sql import SparkSession
from pyspark.sql.functions import *

# define schema for our data
schema = StructType([
StructField("Id", IntegerType(), False),
StructField("First", StringType(), False),
StructField("Last", StringType(), False),
StructField("Url", StringType(), False),
StructField("Published", StringType(), False),
StructField("Hits", IntegerType(), False),
StructField("Campaigns", ArrayType(StringType()), False)])

#create our data
data = [[1, "Jules", "Damji", "https://tinyurl.1", "1/4/2016", 4535, ["twitter", "LinkedIn"]],
   [2, "Brooke","Wenig","https://tinyurl.2", "5/5/2018", 8908, ["twitter", "LinkedIn"]],
   [3, "Denny", "Lee", "https://tinyurl.3","6/7/2019",7659, ["web", "twitter", "FB", 
"LinkedIn"]],
   [4, "Tathagata", "Das","https://tinyurl.4", "5/12/2018", 10568, ["twitter", "FB"]],
   [5, "Matei","Zaharia", "https://tinyurl.5", "5/14/2014", 40578, ["web", "twitter", "FB", 
"LinkedIn"]],
   [6, "Reynold", "Xin", "https://tinyurl.6", "3/2/2015", 25568, ["twitter", "LinkedIn"]]
  ]
 # main program
 if __name__ == "__main__":
 # create a SparkSession
spark = (SparkSession
   .builder
   .appName("Example-3_6")
   .getOrCreate())
# create a DataFrame using the schema defined above
blogs_df = spark.createDataFrame(data, schema)

但是当我尝试执行 .show() 时，我收到了 java 错误。有人可以帮我解决这个错误吗？

blogs_df.show()

错误： Py4JJavaError：调用 o95.showString 时出错。 ：org.apache.spark.SparkException：作业因阶段失败而中止：阶段 3.0 中的任务 0 失败 1 次，最近一次失败：阶段 3.0 中丢失任务 0.0（TID 3）（..com 执行程序驱动程序） ：java.io.IOException：无法运行程序“python3”：CreateProcess error=2，系统找不到指定的文件 在 java.lang.ProcessBuilder.start（未知来源） 在 org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:165) 在 org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:107) 在 org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119) 在 org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:145)

---

在执行以下代码的同时，我得到了 df.show() 的结果

from pyspark.sql.types import StructType, IntegerType, StringType
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession

schema = StructType() \
.add("city", StringType(), True) \
.add("state", StringType(), True) \
.add("pop", IntegerType(), True)

df_with_schema1 = spark.read.format("csv") \
  .option("delimiter", ",") \
  .option("header", True) \
  .schema(schema) \
  .load("<directory>\\pyspark-test.csv")

df_with_schema1.show()

Answer 1

您很可能没有正确安装 Python。也试了一下

spark = (SparkSession
   .builder
   .master('local[*]') # add this
   .appName("Example-3_6")
   .getOrCreate())