Pyspark 排名基于前/当前行

Question

以下是我的源和预期输出数据帧

我需要应用以下逻辑并计算最终排名值 如果上一行(hdr) == 当前行(hdr) & 上一行(dtl) == 当前行(dtl),
然后分配上一行排名，否则上一行排名+ 1

我无法在密集排名后继续前进。你能分享你的意见吗？考虑到潜在的性能开销，我试图避免没有 partitionBy 列的 Window

sample = [(100,1000),(100, 1000), (100, 2000), (200, 1000), (200,1000), (300,1000), (300,2000)]
test = spark.createDataFrame(sample,['hdr','dtl'])
spec = Window.partitionBy('hdr').orderBy('hdr','dtl')
test.withColumn('dense', func.dense_rank().over(spec)).show()

Answer 1

我不认为没有窗口的排名是可能的，在你的情况下，因为排名需要在整个数据集上发生，所以不可能避免没有 partitionBy 的窗口函数，但是我们可以使用下面的代码减少进入一个分区的大量数据。

sample = [(100,1000),(100, 1000), (100, 2000), (200, 1000), (200,1000), (300,1000), (300,2000)]
test = spark.createDataFrame(sample,['hdr','dtl'])

# Since we select only distinct of hdr and dtl huge amount of data is eliminated.
dist_hdr_dtl=test.select("hdr","dtl").distinct()

# Since data size is reduced we can use this window spec.
spec = Window.orderBy('hdr','dtl')
dist_hdr_dtl=dist_hdr_dtl.withColumn('final_rank', dense_rank().over(spec))

# join it with original data to get the ranks.
Note: if distinct dataset is not very huge you can use broadcast join which will improve performance
test.join(dist_hdr_dtl,["hdr","dtl"],"inner").orderBy('hdr','dtl').show()

+---+----+----------+
|hdr| dtl|final_rank|
+---+----+----------+
|100|1000|         1|
|100|1000|         1|
|100|2000|         2|
|200|1000|         3|
|200|1000|         3|
|300|1000|         4|
|300|2000|         5|
+---+----+----------+