bigquery 分区表的分区修剪

Question

我有一个使用分析函数查询天分区表的查询。我希望它只读取在 where 子句中过滤的分区中的数据，但它会读取表中的所有分区。

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` ) q_nested
  ) q
WHERE
  rank < 1000
)
SELECT
  num_mean_temp_samples ,
  count(1) as samples
FROM query
 WHERE
   day in ( '20100101', '20100103')
GROUP BY  1 ORDER BY 1

我验证了没有解析函数的分区修剪工作：

WITH query AS (
SELECT
  FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
  *
FROM
  `mydataset.gsod_partitioned`
)

或在添加 UNION ALL 嵌套选择之后：

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` WHERE _PARTITIONDATE < "1970-01-01" ) q_nested1
  UNION ALL SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` WHERE _PARTITIONDATE >= "1970-01-01" ) q_nested2
  ) q
WHERE
  rank < 1000
)

表 mydataset.gsod_partitioned 是基于 gsod 的公共数据集，其中 day=20100101 分区创建如下：

bq query --destination_table 'private.gsod_partitioned$20100101' --time_partitioning_type=DAY --use_legacy_sql=false
'SELECT station_number, mean_temp, num_mean_temp_samples FROM `bigquery-public-data.samples.gsod` where year=2010 and month=01 and day=01'

您能否找到一种方法来为分析函数启用分区修剪而不向查询添加额外的联合？

Answer 1

关于 _PARTITIONDATE - 它没有记录的功能，建议改用 _PARTITIONETIME，您可以寻找其他一些问题来查看其中一位 Google 员工的说法：Use of the _PARTITIONDATE vs. the _PARTITIONTIME pseudo-columns in BigQuery

关于partition pruning with analitycal functions，Google 在去年增加了对过滤器下推的支持，但仅适用于 _PARTITIONTIME，它应该包含在 PARTITON BY 子句涵盖的字段中

应该是这样的：

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY _pt order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_TIMESTAMP("%Y%m%d", _PARTITIONTIME) AS day,
      _PARTITIONTIME as _pt,
      *
    FROM
      `mydataset.gsod_partitioned` ) q_nested
  ) q
WHERE
  rank < 1000
)
SELECT
  num_mean_temp_samples ,
  count(1) as samples
FROM query
 WHERE
   day in ( '20100101', '20100103')
GROUP BY  1 ORDER BY 1