Spark SQL - 使用 Spark SQL 窗口函数获取每个窗口的行数答案

【问题标题】：Spark SQL - getting row count for each window using spark SQL window functionsSpark SQL - 使用 Spark SQL 窗口函数获取每个窗口的行数
【发布时间】：2020-05-19 10:26:23
【问题描述】：

我想使用 spark SQL 窗口函数来做一些聚合和窗口化。

假设我使用的是此处提供的示例表：https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html

我想运行查询，为我提供每个类别的最大 2 收入以及每个类别的产品数量。

在我运行这个查询之后

SELECT
  product,
  category,
  revenue
FROM (
  SELECT
    product,
    category,
    revenue,
    dense_rank() OVER (PARTITION BY category ORDER BY revenue DESC) as rank
    count(*) OVER (PARTITION BY category ORDER BY revenue DESC) as count
  FROM productRevenue) tmp
WHERE
  rank <= 2

我得到了这样的表：

product category    revenue count
pro2    tablet  6500    1
mini    tablet  5500    2

而不是

product category    revenue count
pro2    tablet  6500    5
mini    tablet  5500    5

这是我的预期。

我应该如何编写代码以获得每个类别的正确计数（而不是使用另一个单独的 Group By 语句）？

【问题讨论】：

标签： apache-spark apache-spark-sql

【解决方案1】：

在Spark 中，如果具有order by 窗口的window 子句默认为ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW。

对于您的情况，在 count(*) 窗口子句中添加 ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING。

Try with:

 SELECT
  product,
  category,
  revenue,count
FROM (
  SELECT
    product,
    category,
    revenue,
    dense_rank() OVER (PARTITION BY category ORDER BY revenue DESC) as rank,
    count(*) OVER (PARTITION BY category ORDER BY revenue DESC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) as count
  FROM productRevenue) tmp
WHERE
  rank <= 2

【讨论】：

【解决方案2】：

将count(*) OVER (PARTITION BY category ORDER BY revenue DESC) as count 更改为count(*) OVER (PARTITION BY category ORDER BY category DESC) as count。你会得到预期的结果。

试试下面的代码。

scala> spark.sql("""SELECT
     |   product,
     |   category,
     |   revenue,
     |   rank,
     |   count
     | FROM (
     |   SELECT
     |     product,
     |     category,
     |     revenue,
     |     dense_rank() OVER (PARTITION BY category ORDER BY revenue DESC) as rank,
     |     count(*) OVER (PARTITION BY category ORDER BY category DESC) as count
     |   FROM productRevenue) tmp
     | WHERE
     |   tmp.rank <= 2 """).show(false)

+----------+----------+-------+----+-----+
|product   |category  |revenue|rank|count|
+----------+----------+-------+----+-----+
|Pro2      |tablet    |6500   |1   |5    |
|Mini      |tablet    |5500   |2   |5    |
|Thin      |cell phone|6000   |1   |5    |
|Very thin |cell phone|6000   |1   |5    |
|Ultra thin|cell phone|5000   |2   |5    |
+----------+----------+-------+----+-----+

【讨论】：