【发布时间】:2019-01-29 20:55:51
【问题描述】:
我有一个表 Sub_Table1_Clus,其架构如下:
name Type Mode
acc_id INTEGER NULLABLE
imp_id STRING NULLABLE
con_id STRING NULLABLE
in_id STRING NULLABLE
date TIMESTAMP NULLABLE
tr_nm STRING NULLABLE
tr_ve STRING NULLABLE
d_cd TIMESTAMP NULLABLE
该表有 140501802 行,我正在尝试获取此查询的结果:
SELECT imp.*,
COALESCE(LAG(imp.date) OVER(PARTITION BY con_id, in_id, tr_nm
ORDER BY d_cd DESC), imp.date) AS imp_date
FROM `Nikhil_Practice.Sub_Table1_Clus` as imp
我收到这个错误
Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 118% of limit. Top memory consumer(s): sort operations used for analytic OVER() clauses: 98% other/unattributed: 2%
谁能帮我解决一些其他的方法来减轻这个错误。谢谢
--添加更多细节以更好地理解--
我查询的最终目标是让整个表按 con_id、in_id、tr_nm 分区,并按 d_cd 对各个分区进行排序,这样我就可以获得先前发生的行的日期列。
例如,考虑以下数据
acc_id imp_id con_id in_id date tr_nm tr_ve d_cd
1 a a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:28:35
1 a a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:29:04
1 b a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:29:04
1 b a1 b2 2018-04-26 2:28:21 c1 d1 2018-04-26 2:28:35
2 c a1 b2 2018-04-26 2:30:56 c1 d1 2018-04-26 2:30:58
2 c a1 b2 2018-04-26 2:30:56 c1 d1 2018-04-26 2:30:58
2 d a1 b3 2018-04-26 10:08:38 c1 d1 2018-04-26 10:08:39
2 d a1 b3 2018-04-26 10:08:38 c1 d1 2018-04-26 10:08:39
2 d a2 b3 2018-04-26 10:52:07 c1 d1 2018-04-26 10:52:08
3 d a2 b3 2018-04-26 10:52:07 c2 d2 2018-04-26 10:52:08
3 e a2 b4 2018-04-26 21:33:29 c2 d2 2018-04-26 21:33:29
3 e a2 b4 2018-04-26 21:33:29 c2 d2 2018-04-26 21:33:29
3 e a2 b4 2018-04-26 10:14:01 c2 d2 2018-04-26 10:14:06
3 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29
3 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29
4 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53
4 f a2 b5 2018-04-26 5:48:32 c2 d2 2018-04-26 5:49:28
所需的 O/p:
acc_id imp_id con_id in_id date tr_nm tr_ve d_cd imp_date
1 a a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:29:04 2018-04-26 2:28:21
1 b a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:29:04 2018-04-26 2:28:21
1 a a1 b1 2018-04-26 2:28:21 c1 d1 2018-04-26 2:28:35 2018-04-26 2:28:21
2 c a1 b2 2018-04-26 2:30:56 c1 d1 2018-04-26 2:30:58 2018-04-26 2:30:56
2 c a1 b2 2018-04-26 2:30:56 c1 d1 2018-04-26 2:30:58 2018-04-26 2:30:56
1 b a1 b2 2018-04-26 2:28:21 c1 d1 2018-04-26 2:28:35 2018-04-26 2:30:56
2 d a1 b3 2018-04-26 10:08:38 c1 d1 2018-04-26 10:08:39 2018-04-26 10:08:38
2 d a1 b3 2018-04-26 10:08:38 c1 d1 2018-04-26 10:08:39 2018-04-26 10:08:38
2 d a2 b3 2018-04-26 10:52:07 c1 d1 2018-04-26 10:52:08 2018-04-26 10:52:07
3 d a2 b3 2018-04-26 10:52:07 c2 d2 2018-04-26 10:52:08 2018-04-26 10:52:07
3 e a2 b4 2018-04-26 21:33:29 c2 d2 2018-04-26 21:33:29 2018-04-26 21:33:29
3 e a2 b4 2018-04-26 21:33:29 c2 d2 2018-04-26 21:33:29 2018-04-26 21:33:29
3 e a2 b4 2018-04-26 10:14:01 c2 d2 2018-04-26 10:14:06 2018-04-26 21:33:29
4 f a2 b5 2018-04-26 5:48:32 c2 d2 2018-04-26 5:49:28 2018-04-26 5:48:32
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53 2018-04-26 5:48:32
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53 2018-04-26 4:46:52
4 f a2 b5 2018-04-26 4:46:52 c2 d2 2018-04-26 4:46:53 2018-04-26 4:46:52
3 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29 2018-04-26 4:46:52
3 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29 2018-04-26 3:06:29
4 e a2 b5 2018-04-26 3:06:29 c2 d2 2018-04-26 3:06:29 2018-04-26 3:06:29
我希望你现在能更好地理解
【问题讨论】:
-
可能被骗或与stackoverflow.com/questions/46005418/…有关。在这个问题中,他们建议删除
order by,因为这显然是一项昂贵的操作。 -
您好,我在窗口函数中有一个排序依据和分区依据。您提供的问题无法解决此问题,因为我的分区太大。
-
@NikhilGurram - 通常诸如“如何修复我的查询”之类的问题在 SO 上并不顺利。您应该展示您的用例/逻辑,说明您做什么和为什么做 - 这样您将有更好的机会获得正确的答案
-
我们愿意为您提供帮助。你能告诉我们这个查询的目标是什么吗?如果你能提供一个样本数据集,我们可以在知道最终目标是什么后重复错误,那就太棒了。
-
@MikhailBerlyant 我认为您现在可以清楚地了解了.. 因为我添加了用例和示例数据。如果您有任何潜在的解决方案,请告诉我。
标签: sql google-bigquery bigquery-standard-sql