在 PostgreSQL 上首次加载时查询运行速度非常慢答案

【问题标题】：Queries running very slow on first load on PostgreSQL在 PostgreSQL 上首次加载时查询运行速度非常慢
【发布时间】：2016-02-10 08:31:45
【问题描述】：

我们在 Amazon EC2 上使用 PostgreSQL 版本 9.4 数据库。我们所有的查询在第一次尝试时都运行得非常慢，直到它被缓存之后它们非常快，但这并不是一种调解，因为它会减慢页面加载速度。

在我们使用的查询之一中：

SELECT HE.fs_perm_sec_id,
   HE.TICKER_EXCHANGE,
   HE.proper_name,
   OP.shares_outstanding,

(SELECT factset_industry_desc
 FROM factset_industry_map AS fim
 WHERE fim.factset_industry_code = HES.industry_code) AS industry,

(SELECT SUM(POSITION) AS ST_HOLDINGS
 FROM OWN_STAKES_HOLDINGS S
  WHERE S.POSITION > 0
   AND S.fs_perm_sec_id = HE.fs_perm_sec_id
 GROUP BY FS_PERM_SEC_ID) AS stake_holdings,

(SELECT SUM(CURRENT_HOLDINGS)
 FROM
   (SELECT CURRENT_HOLDINGS
    FROM OWN_INST_HOLDINGS IHT
    WHERE FS_PERM_SEC_ID=HE.FS_PERM_SEC_ID
    ORDER BY CURRENT_HOLDINGS DESC LIMIT 10)A) AS top_10_inst_hodings,

 (SELECT SUM(OIH.current_holdings)
  FROM own_inst_holdings OIH
  WHERE OIH.fs_perm_sec_id = HE.fs_perm_sec_id) AS inst_holdings

FROM own_prices OP
JOIN h_security_ticker_exchange HE ON OP.fs_perm_sec_id = HE.fs_perm_sec_id
JOIN h_entity_sector HES ON HES.factset_entity_id = HE.factset_entity_id
WHERE HE.ticker_exchange = 'PG-NYS'
ORDER BY OP.price_date DESC LIMIT 1

运行 EXPLAIN ANALYZE 并收到以下结果：

  QUERY PLAN
  Limit  (cost=223.39..223.39 rows=1 width=100) (actual time=2420.644..2420.645 rows=1 loops=1)
  ->  Sort  (cost=223.39..223.39 rows=1 width=100) (actual time=2420.643..2420.643 rows=1 loops=1)
    Sort Key: op.price_date
    Sort Method: top-N heapsort  Memory: 25kB
    ->  Nested Loop  (cost=0.26..223.39 rows=1 width=100) (actual time=2316.169..2420.566 rows=36 loops=1)
          ->  Nested Loop  (cost=0.17..8.87 rows=1 width=104) (actual time=3.958..5.084 rows=36 loops=1)
                ->  Index Scan using h_sec_exch_factset_entity_id_idx on h_security_ticker_exchange he  (cost=0.09..4.09 rows=1 width=92) (actual time=1.452..1.454 rows=1 loops=1)
                      Index Cond: ((ticker_exchange)::text = 'PG-NYS'::text)
                ->  Index Scan using alex_prices on own_prices op  (cost=0.09..4.68 rows=33 width=23) (actual time=2.496..3.592 rows=36 loops=1)
                      Index Cond: ((fs_perm_sec_id)::text = (he.fs_perm_sec_id)::text)
          ->  Index Scan using alex_factset_entity_idx on h_entity_sector hes  (cost=0.09..4.09 rows=1 width=14) (actual time=0.076..0.077 rows=1 loops=36)
                Index Cond: (factset_entity_id = he.factset_entity_id)
          SubPlan 1
            ->  Index Only Scan using alex_factset_industry_code_idx on factset_industry_map fim  (cost=0.03..2.03 rows=1 width=20) (actual time=0.006..0.007 rows=1 loops=36)
                  Index Cond: (factset_industry_code = hes.industry_code)
                  Heap Fetches: 0
          SubPlan 2
            ->  GroupAggregate  (cost=0.08..2.18 rows=2 width=17) (actual time=0.735..0.735 rows=1 loops=36)
                  Group Key: s.fs_perm_sec_id
                  ->  Index Only Scan using own_stakes_holdings_perm_position_idx on own_stakes_holdings s  (cost=0.08..2.15 rows=14 width=17) (actual time=0.080..0.713 rows=39 loops=36)
                        Index Cond: ((fs_perm_sec_id = (he.fs_perm_sec_id)::text) AND (\position\ > 0::numeric))
                        Heap Fetches: 1155
          SubPlan 3
            ->  Aggregate  (cost=11.25..11.26 rows=1 width=6) (actual time=0.166..0.166 rows=1 loops=36)
                  ->  Limit  (cost=0.09..11.22 rows=10 width=6) (actual time=0.081..0.150 rows=10 loops=36)
                        ->  Index Only Scan Backward using alex_current_holdings_idx on own_inst_holdings iht  (cost=0.09..194.87 rows=175 width=6) (actual time=0.080..0.147 rows=10 loops=36)
                              Index Cond: (fs_perm_sec_id = (he.fs_perm_sec_id)::text)
                              Heap Fetches: 288
          SubPlan 4
            ->  Aggregate  (cost=194.96..194.96 rows=1 width=6) (actual time=66.102..66.102 rows=1 loops=36)
                  ->  Index Only Scan using alex_current_holdings_idx on own_inst_holdings oih  (cost=0.09..194.87 rows=175 width=6) (actual time=0.060..65.209 rows=2505 loops=36)
                        Index Cond: (fs_perm_sec_id = (he.fs_perm_sec_id)::text)
                 Heap Fetches: 33453
 Planning time: 1.581 ms
 Execution time: 2420.830 ms

一旦我们为 3 个聚合禁用 SELECT SUM()，它会大大加快速度，但它会破坏拥有关系数据库的意义。

我们正在使用 NodeJS 运行查询，使用 PG 插件 (https://www.npmjs.com/package/pg) 连接并在数据库上运行查询

我们如何加快查询速度？我们可以采取哪些额外步骤？我们已经为数据库建立了索引，所有字段似乎都被正确索引了，但速度仍然不够快。

感谢任何帮助、cmets 和/或建议。

【问题讨论】：

您使用的是 EC2 而不是 RDS？您使用的是什么实例大小？你在使用 EBS 吗？您使用的是哪种 EBS 卷类型？ EBS 卷的大小是多少？您的服务器监控、磁盘 IO、CPU、网络传输的瓶颈似乎是什么？
我正在使用似乎在 AWS 上托管数据库的 heroku 插件
如果您甚至不知道它是否托管在 EC2 实例上，也许您应该使用“Heroku”而不是任何 AWS 标签来标记问题。 Heroku 的 PostgreSQL 产品特定于 Heroku，虽然它可能在幕后使用 AWS 资源，但像 amazon-ec2 这样的标签与问题无关。
感谢您的反馈，但这并不能回答问题。
请注意，我对您的问题发表了评论。我没有发布答案。我的评论旨在帮助您更新您的问题以提供必要的详细信息，以便有人能够提供答案。

标签： performance postgresql heroku pg

【解决方案1】：

带有聚合的嵌套循环通常是一件坏事。下面应该避免这种情况。（未经测试；SQLFiddle 会很有帮助。）试一试，让我知道。我很好奇引擎如何使用窗口函数过滤器。

WITH    security
AS  (
    SELECT  HE.fs_perm_sec_id
,       HE.TICKER_EXCHANGE
,       HE.proper_name
,       OP.shares_outstanding
,       OP.price_date
FROM    own_prices                  AS OP
 JOIN   h_security_ticker_exchange  AS HE
  ON    OP.fs_perm_sec_id       = HE.fs_perm_sec_id
 JOIN   h_entity_sector             AS HES
  ON    HES.factset_entity_id   = HE.factset_entity_id
WHERE HE.ticker_exchange = 'PG-NYS'
)
SELECT  SE.fs_perm_sec_id
,       SE.TICKER_EXCHANGE
,       SE.proper_name
,       SE.shares_outstanding
,       S.stake_holdings
,       IHT.top_10_inst_holdings
,       OIH.inst_holdings
FROM    security    SE
 JOIN   (
        SELECT  S.fs_perm_sec_id
        ,       SUM(S.POSITION)     AS stake_holdings
        FROM    OWN_STAKES_HOLDINGS AS S
        WHERE   S.fs_perm_sec_id    IN (
                    SELECT  fs_perm_sec_id
                    FROM    security
                )
         AND    S.POSITION          > 0
        GROUP BY    S.fs_perm_sec_id
        )   AS S
  ON    SE.fs_perm_sec_id   = S.fs_perm_sec_id
 JOIN   (
        SELECT  IHT.FS_PERM_SEC_ID
        ,       SUM(IHT.CURRENT_HOLDINGS)   AS top_10_inst_holdings
        FROM    OWN_INST_HOLDINGS   AS IHT
        WHERE   IHT.FS_PERM_SEC_ID  IN (
                    SELECT  fs_perm_sec_id
                    FROM    security
                )
         AND    ROW_NUMBER() OVER (
                    PARTITION BY IHT.FS_PERM_SEC_ID
                    ORDER BY IHT.CURRENT_HOLDINGS DESC
                )                   <= 10
        GROUP BY    IHT.FS_PERM_SEC_ID
        )   AS IHT
  ON    SE.fs_perm_sec_id   = IHT.fs_perm_sec_id
 JOIN   (
        SELECT  S.fs_perm_sec_id
        ,       SUM(OIH.current_holdings)   AS inst_holdings
        FROM    own_inst_holdings   AS OIH
        WHERE   OIH.fs_perm_sec_id  IN (
                    SELECT  fs_perm_sec_id
                    FROM    security
                )
        GROUP BY    OIH.fs_perm_sec_id
        )   AS OIH
  ON    SE.fs_perm_sec_id   = OIH.fs_perm_sec_id
ORDER BY    SE.price_date
LIMIT   1

【讨论】：