数据框对象在 pyspark 中不可调用

Question

temp = Window.partitionBy("id").orderBy("time").rowsBetween(-5, 5)
spark_df.withColumn("movingAvg",fn.avgspark_df("average")).over(temp)).show()

我在最后一行收到此错误。

数据框对象不可调用

Answer 1

您缺少一个括号，但似乎有些语法是错误的。我认为这是您在括号丢失之前的代码：

fn.avgspark_df("average")

这就是您收到错误的原因；您正在尝试将 DataFrame 作为函数调用。我相信您可以通过以下方式实现您想要的：

import pyspark.sql.functions as fn
from pyspark.sql import Window

df = pd.DataFrame({'id': [0,0,0,0,0,1,1,1,1,1],
                   'time': [1,2,3,4,5,1,2,3,4,5],
                   'average':[0,1,2,3,4,5,6,7,8,9] })
df = sqlContext.createDataFrame(df)

temp = Window.partitionBy("id").orderBy("time").rowsBetween(-1, 1)
df.withColumn("movingAvg",fn.avg("average").over(temp)).show()

Answer 2

 from pyspark.sql import SparkSession
 from pyspark.sql import Window
 from pyspark.sql.functions import max,min,avg
 spark = SparkSession.builder.appName("Data Frame Example") \
.config("spark.some.config.option", "some-value") \
.getOrCreate()

 l=[("Alice", "2016-05-01", 50.00),
                                ("Alice", "2016-05-03", 45.00),
                                ("Alice", "2016-05-04", 55.00),
                                ("Bob", "2016-05-01", 25.00),
                                ("Bob", "2016-05-04", 29.00),
                                ("Bob", "2016-05-06", 27.00)]

customers = spark.sparkContext.parallelize(l).toDF(["name", "date", "amountSpent"])

temp = Window.partitionBy("name").orderBy("date")

customers.withColumn( "movingAvg",avg("amountSpent").over(temp)).show()