在 Spark Dataframe 中查找每周的日期

Question

我想在 Dataframe 中添加一个包含每个对应周日期的列（在每个日期中附加星期五） 我的数据框看起来像这样

+----+------+---------+

|Week| City|sum(Sale)|
+----+------+---------+
| 29|City 2| 72|
| 28|City 3| 48|
| 28|City 2| 19|
| 27|City 2| 16|
| 28|City 1| 84|
| 28|City 4| 72|
| 29|City 4| 39|
| 27|City 3| 42|
| 26|City 3| 68|
| 27|City 1| 89|
| 27|City 4| 104|
| 26|City 2| 19|
| 29|City 3| 27|
+----+------+---------+

我需要把它转换成下面的数据框

----+------+---------+---------------  |
|Week| City|sum(Sale)|perticular day(dd/mm/yyyy)  |
+----+------+---------+---------------|
| 29|City 2| 72|Friday(07/21/2017)|
| 28|City 3| 48|Friday(07/14/2017)|
| 28|City 2| 19|Friday(07/14/2017)|
| 27|City 2| 16|Friday(07/07/2017)|
| 28|City 1| 84|Friday(07/14/2017)|
| 28|City 4| 72|Friday(07/14/2017)|
| 29|City 4| 39|Friday(07/21/2017)|
| 27|City 3| 42|Friday(07/07/2017)|
| 26|City 3| 68|Friday(06/30/2017)|
| 27|City 1| 89|Friday(07/07/2017)|
| 27|City 4| 104|Friday(07/07/2017)|
| 26|City 2| 19|Friday(06/30/2017)|
| 29|City 3| 27|Friday(07/21/2017)|
+----+------+---------+

请帮帮我

Answer 1

您可以编写一个简单的 UDF 并通过在其中添加星期来获取日期。

这是一个简单的例子

import spark.implicits._
val data = spark.sparkContext.parallelize(Seq(
  (29,"City 2", 72),
  (28,"City 3", 48),
  (28,"City 2", 19),
  (27,"City 2", 16),
  (28,"City 1", 84),
  (28,"City 4", 72),
  (29,"City 4", 39),
  (27,"City 3", 42),
  (26,"City 3", 68),
  (27,"City 1", 89),
  (27,"City 4", 104),
  (26,"City 2", 19),
  (29,"City 3", 27)
)).toDF("week", "city", "sale")


val getDateFromWeek = udf((week : Int) => {
  //create a default date for week 1
  val week1 = LocalDate.of(2016, 12, 30)
  val day = "Friday"
  //add week from the week column 
  val result = week1.plusWeeks(week).format(DateTimeFormatter.ofPattern("MM/dd/yyyy"))
  //return result as Friday (date)
  s"${day} (${result})"
})

//use the udf and create a new column named day
data.withColumn("day", getDateFromWeek($"week")).show

任何人都可以将其转换为 Pyspark 吗？