Postgresql - 左外连接性能问题答案

【问题标题】：Postgresql - LEFT OUTER JOIN performance issuePostgresql - 左外连接性能问题
【发布时间】：2016-06-07 10:09:13
【问题描述】：

这更多是关于 postgres 如何实际工作的理论问题。我有四个表，比如 A、B、C 和 D，分别有 20k、870k、770k 和 150 万条记录。 Id 是主键，并且在外键上进行连接（默认情况下被索引，并且不允许空值）。这是我的查询：

SELECT "A"."id" AS "a", 
  COUNT("B"."id") AS "b_count", 
   COUNT("C"."id") AS "c_count",
   COUNT("D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id") 
GROUP BY "A"."id"

表格和查询由 django 框架生成。由于未知原因，它运行了这么长时间，该服务器崩溃了。但是，如果我将这些 LEFT OUTER JOIN 中的任何一个更改为 INNER JOIN，则查询将运行 4.2 秒。我知道 LEFT 和 INNER JOIN 在实践中是如何工作的，但是为什么在这种情况下性能差异如此之大呢？

附加信息： EXPLAIN 用于带有 LEFT OUTER JOIN 的查询（EXPLAIN (ANALYZE, BUFFERS) 不起作用，因为查询没有结束）：

"GroupAggregate  (cost=606144.91..628195.81 rows=20784 width=16)"
"  ->  Sort  (cost=606144.91..610513.52 rows=1747445 width=16)"
"        Sort Key: A.id"
"        ->  Hash Right Join  (cost=282896.74..365231.69 rows=1747445 width=16)"
"              Hash Cond: (D.C_id = C.id)"
"              ->  Seq Scan on D  (cost=0.00..32113.94 rows=1494094 width=8)"
"              ->  Hash  (cost=267717.03..267717.03 rows=873257 width=12)"
"                    ->  Hash Right Join  (cost=208924.92..267717.03 rows=873257 width=12)"
"                          Hash Cond: (B.A_id = A.id)"
"                          ->  Hash Right Join  (cost=207935.28..250353.82 rows=873257 width=12)"
"                                Hash Cond: (C.B_id = B.id)"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8)"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8)"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8)"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4)"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4)"

EXPLAIN(ANALYZE, BUFFERS) 用于使用一个 INNER JOIN 而不是 LEFT OUTER JOIN 的查询：

"GroupAggregate  (cost=559935.67..578819.69 rows=20784 width=16) (actual time=4565.338..5090.632 rows=19567 loops=1)"
"  Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
"  ->  Sort  (cost=559935.67..563670.91 rows=1494094 width=16) (actual time=4565.244..4832.596 rows=1494094 loops=1)"
"        Sort Key: A.id"
"        Sort Method: external merge  Disk: 37992kB"
"        Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
"        ->  Hash Join  (cost=278322.24..355638.06 rows=1494094 width=16) (actual time=2274.363..3341.921 rows=1494094 loops=1)"
"              Hash Cond: (D.C_id = C.id)"
"              Buffers: shared hit=10622 read=205521, temp read=13681 written=13429"
"              ->  Seq Scan on D  (cost=0.00..32113.94 rows=1494094 width=8) (actual time=0.007..270.841 rows=1494094 loops=1)"
"                    Buffers: shared hit=3828 read=13345"
"              ->  Hash  (cost=265343.13..265343.13 rows=746649 width=12) (actual time=2271.959..2271.959 rows=746649 loops=1)"
"                    Buckets: 4096  Batches: 64  Memory Usage: 512kB"
"                    Buffers: shared hit=6794 read=192176, temp read=5640 written=8351"
"                    ->  Hash Join  (cost=208924.92..265343.13 rows=746649 width=12) (actual time=1107.516..2138.249 rows=746649 loops=1)"
"                          Hash Cond: (B.A_id = A.id)"
"                          Buffers: shared hit=6794 read=192176, temp read=5640 written=5514"
"                          ->  Hash Join  (cost=207935.28..250353.82 rows=746649 width=12) (actual time=1099.799..1784.403 rows=746649 loops=1)"
"                                Hash Cond: (C.B_id = B.id)"
"                                Buffers: shared hit=6272 read=192176, temp read=5640 written=5514"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.005..203.923 rows=746649 loops=1)"
"                                      Buffers: shared hit=4600 read=8973"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8) (actual time=1095.407..1095.407 rows=873453 loops=1)"
"                                      Buckets: 4096  Batches: 64  Memory Usage: 547kB"
"                                      Buffers: shared hit=1672 read=183203, temp written=2907"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..939.839 rows=873453 loops=1)"
"                                            Buffers: shared hit=1672 read=183203"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4) (actual time=7.701..7.701 rows=20784 loops=1)"
"                                Buckets: 4096  Batches: 1  Memory Usage: 731kB"
"                                Buffers: shared hit=522"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4) (actual time=0.004..4.661 rows=20784 loops=1)"
"                                      Buffers: shared hit=522"
"Total runtime: 5099.046 ms"

还有奖励：如果我使用稍微修改的查询：

SELECT "A"."id" AS "a", 
  COUNT("B"."id") AS "b_count", 
   COUNT("C"."id") AS "c_count",
   COUNT("D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."D_id") 
WHERE "D"."id" < 1500000 -- this is always true
GROUP BY "A"."id"

然后它也可以工作，但比使用至少一个 INNER JOIN 长一点。 EXPLAIN (ANALYZE, BUFFERS) 这个案例：

"GroupAggregate  (cost=563670.91..582554.92 rows=20784 width=16) (actual time=6779.121..7286.640 rows=19567 loops=1)"
"  Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
"  ->  Sort  (cost=563670.91..567406.14 rows=1494094 width=16) (actual time=6777.606..7033.885 rows=1494094 loops=1)"
"        Sort Key: A.id"
"        Sort Method: external merge  Disk: 37992kB"
"        Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
"        ->  Hash Join  (cost=278322.24..359373.29 rows=1494094 width=16) (actual time=4601.432..5674.321 rows=1494094 loops=1)"
"              Hash Cond: (D.C_id = C.id)"
"              Buffers: shared hit=10814 read=205329, temp read=13681 written=13429"
"              ->  Seq Scan on D  (cost=0.00..35849.18 rows=1494094 width=8) (actual time=0.005..374.660 rows=1494094 loops=1)"
"                    Filter: (id < 1500000)"
"                    Buffers: shared hit=3892 read=13281"
"              ->  Hash  (cost=265343.13..265343.13 rows=746649 width=12) (actual time=4600.782..4600.782 rows=746649 loops=1)"
"                    Buckets: 4096  Batches: 64  Memory Usage: 512kB"
"                    Buffers: shared hit=6922 read=192048, temp read=5640 written=8351"
"                    ->  Hash Join  (cost=208924.92..265343.13 rows=746649 width=12) (actual time=3363.352..4469.474 rows=746649 loops=1)"
"                          Hash Cond: (B.A_id = A.id)"
"                          Buffers: shared hit=6922 read=192048, temp read=5640 written=5514"
"                          ->  Hash Join  (cost=207935.28..250353.82 rows=746649 width=12) (actual time=3257.869..4066.067 rows=746649 loops=1)"
"                                Hash Cond: (C.B_id = B.id)"
"                                Buffers: shared hit=6400 read=192048, temp read=5640 written=5514"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.006..372.317 rows=746649 loops=1)"
"                                      Buffers: shared hit=4664 read=8909"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8) (actual time=3257.327..3257.327 rows=873453 loops=1)"
"                                      Buckets: 4096  Batches: 64  Memory Usage: 547kB"
"                                      Buffers: shared hit=1736 read=183139, temp written=2907"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..3097.367 rows=873453 loops=1)"
"                                            Buffers: shared hit=1736 read=183139"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4) (actual time=105.467..105.467 rows=20784 loops=1)"
"                                Buckets: 4096  Batches: 1  Memory Usage: 731kB"
"                                Buffers: shared hit=522"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4) (actual time=0.002..101.506 rows=20784 loops=1)"
"                                      Buffers: shared hit=522"
"Total runtime: 7294.388 ms"

为什么原来的查询会破坏 postgres，而其他两个则像魅力一样工作？

编辑：即使没有 count() 子句，所有查询的行为都相同。问题仅存在于 JOIN 中

【问题讨论】：

wiki.postgresql.org/wiki/SlowQueryQuestions
谢谢，我在我的问题中添加了更多信息

标签： performance postgresql left-join

【解决方案1】：

因为 LEFT JOIN 可以将您的结果集乘以很多，具体取决于表格内容。

首先，您的查询似乎是错误的，您还会计算重复项，因此您应该使用COUNT(DISTINCT columnName)

SELECT "A"."id" AS "a", 
  COUNT(distinct "B"."id") AS "b_count", 
   COUNT(distinct "C"."id") AS "c_count",
   COUNT(distinct "D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id") 
GROUP BY "A"."id

其次，考虑添加适当的索引。

A - (id)
B - (id,A_id)
C - (id,B_id)
D - (id,C_id)

【讨论】：

这里不需要 DISTINCT，因为这些 id 是主键。索引已经存在，因为 JOIN 是在外键上进行的（默认情况下是索引的）
如果IDs 是主键无关紧要，例如，如果表c 有2 条记录相同b_id 和表d 有2 条记录相同@ 987654329@ ，C.id 的计数将是 4 而不是 2。@PawelRoman
你说得对，对不起，我的错。尽管如此，它并没有改变我们的观点。 COUNT 根本不是问题。如果我们删除 COUNTs，则会出现同样的问题：LEFT JOINS 需要无限时间才能完成，而即使将一个 LEFT JOIN 替换为 INNER JOIN 也会将查询执行时间减少到 5 秒。为什么？正在外键上进行 JOINS（不允许空值，使用默认索引）