查询大量报价数据的 SQLite 性能

Question

我有一个数据库，其中包含 S&P/TSX 综合指数中所有 229 只股票的大量分时数据。作为参考，一天的数据价值约为 1300 万行。

这是一个sn-p数据：

ID  TICKER  TIME                     TYPE      VALUE  SIZE  EXCHANGE  CONDN_CODES  BUY_BRK  SELL_BRK
--  ------  -----------------------  --------  -----  ----  --------  -----------  -------  --------
15  ABX CN  2021-05-07T13:30:00.000  BEST_BID  29.21  918   T
16  ABX CN  2021-05-07T13:30:00.000  BEST_BID  29.21  917   T
17  ABX CN  2021-05-07T13:30:00.000  BEST_BID  29.21  927   T
18  ABX CN  2021-05-07T13:30:00.000  BEST_BID  29.21  928   T
19  ABX CN  2021-05-07T13:30:00.000  TRADE     29.21  100   T         OPA,XT       85       85
20  ABX CN  2021-05-07T13:30:00.000  TRADE     29.21  100   T         OPA,XT       79       79
21  ABX CN  2021-05-07T13:30:00.000  TRADE     29.21  200   T         OPA,XT       79       79
22  ABX CN  2021-05-07T13:30:00.000  TRADE     29.21  100   T         OPA,XT       79       79

其中一件有趣的事情是分析交易是发生在出价方还是出价方。所以我写了一个查询，它将最新的最佳出价和最新的最佳卖价附加到每笔交易中。运行查询时，输出如下所示：

ID      TICKER  TIME                     TYPE   VALUE  SIZE  EXCHANGE  CONDN_CODES  BUY_BRK  SELL_BRK  LATEST_BBID  LATEST_BASK
------  ------  -----------------------  -----  -----  ----  --------  -----------  -------  --------  -----------  -----------
267795  AC CN   2021-05-07T13:45:03.000  TRADE  24.5   100   T                      2        1         24.5         24.51
267797  AC CN   2021-05-07T13:45:03.000  TRADE  24.5   100   C                      1        79        24.5         24.51
267803  AC CN   2021-05-07T13:45:03.000  TRADE  24.5   1     B         MN           80       79        24.49        24.5
267817  AC CN   2021-05-07T13:45:05.000  TRADE  24.5   200   T                      79       80        24.49        24.5
267834  AC CN   2021-05-07T13:45:07.000  TRADE  24.52  50    B         MN           80       212       24.5         24.52
267837  AC CN   2021-05-07T13:45:07.000  TRADE  24.5   100   T                      2        79        24.5         24.51
267858  AC CN   2021-05-07T13:45:08.000  TRADE  24.48  100   X                      79       9         24.48        24.5
267859  AC CN   2021-05-07T13:45:08.000  TRADE  24.48  100   O                      1        9         24.48        24.5

那么问题来了：

查询很慢。我以每只股票为基础运行它，每个查询都需要几秒钟。迭代 229 只不同的股票，进行一些计算，然后输出一些汇总统计数据大约需要 5 分钟，这对我来说太慢了。

查询编写如下（股票代码和日期是硬编码以便于阅读）：

SELECT 
    ID, TICKER, TIME, TYPE, VALUE, SIZE, EXCHANGE, CONDN_CODES, BUY_BRK, SELL_BRK, 
    (SELECT VALUE from TICKDATA AS td1 WHERE td1.ID = (SELECT max(ID) FROM TICKDATA AS td2 WHERE td2.TICKER = tickdata.TICKER AND td2.ID < tickdata.ID AND TYPE = "BEST_BID") ) AS LATEST_BBID,
    (SELECT VALUE from TICKDATA AS td1 WHERE td1.ID = (SELECT max(ID) FROM TICKDATA AS td2 WHERE td2.TICKER = tickdata.TICKER AND td2.ID < tickdata.ID AND TYPE = "BEST_ASK") ) AS LATEST_BASK
FROM tickdata
WHERE 
    TICKER = "AC CN" AND 
    TYPE = "TRADE" AND
    TIME > "2021-05-07T13:45:00.00" AND
    TIME <= "2021-05-07T14:00:00.00";

所以我接下来尝试的是使用窗口函数，写法如下：

SELECT
    ID, TICKER, TIME, TYPE, VALUE, SIZE, EXCHANGE, CONDN_CODES, BUY_BRK, SELL_BRK,
    LAG(VALUE) 
        OVER
            (
                PARTITION BY TICKER
                ORDER BY (CASE WHEN TYPE = "BEST_BID" THEN 1 ELSE 2 END), ID
            ) AS LATEST_BBID
FROM tickdata
WHERE 
    TICKER = "AC CN" AND 
    TYPE = "TRADE" AND
    TIME > "2021-05-07T13:45:00.00" AND
    TIME <= "2021-05-07T14:00:00.00";

但是窗口函数更慢！

所以我的问题归结为以下几点：

-我可以重写查询或窗口函数以显着提高性能吗？

-如果无法以高效的方式使用查询或窗口函数执行此操作，我可以/应该使用触发器之类的东西在 INSERT 操作期间自动更新最新的最佳出价/最新的最佳出价吗？之后，我可以直接 SELECT * 并获取所有数据，而无需 SQLite 即时执行任何计算。

-最后，如果这也不起作用，我应该使用其他类型的数据库吗？

感谢您的帮助！

Answer 1

我希望您提供一些样本数据进行测试。你会尝试这样的查询吗：

with best(ticker, type, id) as 
(
SELECT ticker, Type, max(ID) 
    FROM TICKDATA 
    WHERE  TYPE in ('BEST_BID', 'BEST_ASK')
    group by ticker, type
),
bids as 
(
   select td.ticker, td.value
   from tickdata td
   inner join best b on td.id = b.id
   where b.type = 'BEST_BID'
),
asks as
(
   select td.ticker, td.value
   from tickdata td
   inner join best b on td.id = b.id
   where b.type = 'BEST_ASK'
)
SELECT 
    t.ID, t.TICKER, t.TIME, t.TYPE, t.VALUE, t.SIZE, t.EXCHANGE, t.CONDN_CODES, t.BUY_BRK, t.SELL_BRK, 
    bids.value AS LATEST_BBID,
    asks.value AS LATEST_BASK
FROM tickdata  t 
inner join bids on t.ticker = bid.ticker
inner join asks on t.ticker = ask.tciker
WHERE 
    t.TICKER = "AC CN" AND 
    t.TYPE = "TRADE" AND
    t.TIME > "2021-05-07T13:45:00.00" AND
    t.TIME <= "2021-05-07T14:00:00.00";