SQLite 查询的运行速度比 MSAccess 查询慢 10 倍答案

【问题标题】：SQLite query runs 10 times slower than MSAccess querySQLite 查询的运行速度比 MSAccess 查询慢 10 倍
【发布时间】：2011-01-02 07:01:29
【问题描述】：

我有一个 800MB 的 MS Access 数据库，已迁移到 SQLite。数据库结构如下（迁移后的SQLite数据库约为330MB）：

表 Occurrence 有 1,600,000 条记录。表格如下：

CREATE TABLE Occurrence 
(
SimulationID  INTEGER,    SimRunID   INTEGER,    OccurrenceID   INTEGER,
OccurrenceTypeID    INTEGER,    Period    INTEGER,    HasSucceeded    BOOL, 
PRIMARY KEY (SimulationID,  SimRunID,   OccurrenceID)
)

它有以下索引：

CREATE INDEX "Occurrence_HasSucceeded_idx" ON "Occurrence" ("HasSucceeded" ASC)

CREATE INDEX "Occurrence_OccurrenceID_idx" ON "Occurrence" ("OccurrenceID" ASC)

CREATE INDEX "Occurrence_SimRunID_idx" ON "Occurrence" ("SimRunID" ASC)

CREATE INDEX "Occurrence_SimulationID_idx" ON "Occurrence" ("SimulationID" ASC)

表 OccurrenceParticipant 有 3,400,000 条记录。表格如下：

CREATE TABLE OccurrenceParticipant 
(
SimulationID    INTEGER,     SimRunID    INTEGER,    OccurrenceID     INTEGER,
RoleTypeID     INTEGER,     ParticipantID    INTEGER
)

它有以下索引：

CREATE INDEX "OccurrenceParticipant_OccurrenceID_idx" ON "OccurrenceParticipant" ("OccurrenceID" ASC)

CREATE INDEX "OccurrenceParticipant_ParticipantID_idx" ON "OccurrenceParticipant" ("ParticipantID" ASC)

CREATE INDEX "OccurrenceParticipant_RoleType_idx" ON "OccurrenceParticipant" ("RoleTypeID" ASC)

CREATE INDEX "OccurrenceParticipant_SimRunID_idx" ON "OccurrenceParticipant" ("SimRunID" ASC)

CREATE INDEX "OccurrenceParticipant_SimulationID_idx" ON "OccurrenceParticipant" ("SimulationID" ASC)

表InitialParticipant 有130 条记录。表的结构是

CREATE TABLE InitialParticipant 
(
ParticipantID    INTEGER  PRIMARY KEY,     ParticipantTypeID    INTEGER,
ParticipantGroupID     INTEGER
)

该表具有以下索引：

CREATE INDEX "initialpart_participantTypeID_idx" ON "InitialParticipant" ("ParticipantGroupID" ASC)

CREATE INDEX "initialpart_ParticipantID_idx" ON "InitialParticipant" ("ParticipantID" ASC)

表ParticipantGroup 有22 条记录。好像

CREATE TABLE ParticipantGroup   (
ParticipantGroupID    INTEGER,    ParticipantGroupTypeID     INTEGER,
Description    varchar (50),      PRIMARY KEY(  ParticipantGroupID  )
)

该表具有以下索引：在“ParticipantGroup”（“ParticipantGroupID”ASC）上创建索引“ParticipantGroup_ParticipantGroupID_idx”

表tmpSimArgs 有18 条记录。它的结构如下：

CREATE TABLE tmpSimArgs (SimulationID varchar, SimRunID int(10))

还有以下索引：

CREATE INDEX tmpSimArgs_SimRunID_idx ON tmpSimArgs(SimRunID ASC)

CREATE INDEX tmpSimArgs_SimulationID_idx ON tmpSimArgs(SimulationID ASC)

“tmpPartArgs”表有 80 条记录。它具有以下结构：

CREATE TABLE tmpPartArgs(participantID INT)

还有下面的索引：

CREATE INDEX tmpPartArgs_participantID_idx ON tmpPartArgs(participantID ASC)

我有一个涉及多个 INNER JOIN 的查询，我面临的问题是查询的 Access 版本大约需要一秒钟，而同一查询的 SQLite 版本需要 10 秒（大约慢 10 倍！）这是不可能的对我来说，迁移回 Access 和 SQLite 是我唯一的选择。

我是编写数据库查询的新手，因此这些查询可能看起来很愚蠢，所以请就您看到的任何错误或儿童菜提出建议。

Access中的查询是（整个查询执行需要1秒）：

SELECT ParticipantGroup.Description, Occurrence.SimulationID, Occurrence.SimRunID, Occurrence.Period, Count(OccurrenceParticipant.ParticipantID) AS CountOfParticipantID FROM 
( 
   ParticipantGroup INNER JOIN InitialParticipant ON ParticipantGroup.ParticipantGroupID =  InitialParticipant.ParticipantGroupID
) INNER JOIN 
(
tmpPartArgs INNER JOIN 
  (
     (
        tmpSimArgs INNER JOIN Occurrence ON (tmpSimArgs.SimRunID = Occurrence.SimRunID)   AND (tmpSimArgs.SimulationID = Occurrence.SimulationID)
     ) INNER JOIN OccurrenceParticipant ON (Occurrence.OccurrenceID =    OccurrenceParticipant.OccurrenceID) AND (Occurrence.SimRunID = OccurrenceParticipant.SimRunID) AND (Occurrence.SimulationID = OccurrenceParticipant.SimulationID)
  ) ON tmpPartArgs.participantID = OccurrenceParticipant.ParticipantID
) ON InitialParticipant.ParticipantID = OccurrenceParticipant.ParticipantID WHERE (((OccurrenceParticipant.RoleTypeID)=52 Or (OccurrenceParticipant.RoleTypeID)=49)) AND Occurrence.HasSucceeded = True GROUP BY ParticipantGroup.Description, Occurrence.SimulationID, Occurrence.SimRunID, Occurrence.Period;

SQLite 查询如下（这个查询大约需要 10 秒）：

SELECT ij1.Description, ij2.occSimulationID, ij2.occSimRunID, ij2.Period, Count(ij2.occpParticipantID) AS CountOfParticipantID FROM 
(
   SELECT ip.ParticipantGroupID AS ipParticipantGroupID, ip.ParticipantID AS ipParticipantID, ip.ParticipantTypeID, pg.ParticipantGroupID AS pgParticipantGroupID, pg.ParticipantGroupTypeID, pg.Description FROM ParticipantGroup as pg INNER JOIN InitialParticipant AS ip ON pg.ParticipantGroupID = ip.ParticipantGroupID
) AS ij1 INNER JOIN 
(
   SELECT tpa.participantID AS tpaParticipantID, ij3.* FROM tmpPartArgs AS tpa INNER JOIN 
     (
       SELECT ij4.*, occp.SimulationID as occpSimulationID, occp.SimRunID AS occpSimRunID, occp.OccurrenceID AS occpOccurrenceID, occp.ParticipantID AS occpParticipantID, occp.RoleTypeID FROM 
          (
              SELECT tsa.SimulationID AS tsaSimulationID, tsa.SimRunID AS tsaSimRunID, occ.SimulationID AS occSimulationID, occ.SimRunID AS occSimRunID, occ.OccurrenceID AS occOccurrenceID, occ.OccurrenceTypeID, occ.Period, occ.HasSucceeded FROM tmpSimArgs AS tsa INNER JOIN Occurrence AS occ ON (tsa.SimRunID = occ.SimRunID) AND (tsa.SimulationID = occ.SimulationID)
          ) AS ij4 INNER JOIN OccurrenceParticipant AS occp ON (occOccurrenceID =      occpOccurrenceID) AND (occSimRunID = occpSimRunID) AND (occSimulationID = occpSimulationID)
    ) AS ij3 ON tpa.participantID = ij3.occpParticipantID
) AS ij2 ON ij1.ipParticipantID = ij2.occpParticipantID WHERE (((ij2.RoleTypeID)=52 Or (ij2.RoleTypeID)=49)) AND ij2.HasSucceeded = 1 GROUP BY ij1.Description, ij2.occSimulationID, ij2.occSimRunID, ij2.Period;

我不知道我在这里做错了什么。我有所有的索引，但我想我错过了声明一些对我有用的关键索引。有趣的是，在迁移之前，我对 SQLite 的“研究”表明，SQLite 在各个方面都比 Access 更快、更小、更好。但在查询方面，我似乎无法让 SQLite 比 Access 更快地工作。我重申，我是 SQLite 的新手，显然没有太多的想法和经验，所以如果有任何博学的灵魂可以帮助我解决这个问题，我将不胜感激。

【问题讨论】：

那个查询让我头疼。我不明白您为什么要执行所有派生查询（子选择）。你能用英语（而不是 SQL）解释你试图从查询中返回什么吗？会更容易回答您的问题。
我将用英语解释每个 sub select 语句的作用。由于此评论框只能容纳 600 个字符，因此我将这些解释发布为我的问题的答案。

标签： sqlite query-optimization

【解决方案1】：

我已重新格式化您的代码（使用我的自制 sql formatter），希望能让其他人更容易阅读......

重新格式化的查询：

SELECT
    ij1.Description,
    ij2.occSimulationID,
    ij2.occSimRunID,
    ij2.Period,
    Count(ij2.occpParticipantID) AS CountOfParticipantID

FROM (

    SELECT
        ip.ParticipantGroupID AS ipParticipantGroupID,
        ip.ParticipantID AS ipParticipantID,
        ip.ParticipantTypeID,
        pg.ParticipantGroupID AS pgParticipantGroupID,
        pg.ParticipantGroupTypeID,
        pg.Description

    FROM ParticipantGroup AS pg

    INNER JOIN InitialParticipant AS ip
            ON pg.ParticipantGroupID = ip.ParticipantGroupID

) AS ij1

INNER JOIN (

    SELECT
        tpa.participantID AS tpaParticipantID,
        ij3.*

    FROM tmpPartArgs AS tpa

    INNER JOIN (

        SELECT
            ij4.*,
            occp.SimulationID AS occpSimulationID,
            occp.SimRunID AS occpSimRunID,
            occp.OccurrenceID AS occpOccurrenceID,
            occp.ParticipantID AS occpParticipantID,
            occp.RoleTypeID

        FROM (

            SELECT
                tsa.SimulationID AS tsaSimulationID,
                tsa.SimRunID AS tsaSimRunID,
                occ.SimulationID AS occSimulationID,
                occ.SimRunID AS occSimRunID,
                occ.OccurrenceID AS occOccurrenceID,
                occ.OccurrenceTypeID,
                occ.Period,
                occ.HasSucceeded

            FROM tmpSimArgs AS tsa

            INNER JOIN Occurrence AS occ
                    ON (tsa.SimRunID = occ.SimRunID)
                   AND (tsa.SimulationID = occ.SimulationID)

        ) AS ij4

        INNER JOIN OccurrenceParticipant AS occp
                ON (occOccurrenceID = occpOccurrenceID)
               AND (occSimRunID = occpSimRunID)
               AND (occSimulationID = occpSimulationID)

    ) AS ij3
      ON tpa.participantID = ij3.occpParticipantID

) AS ij2
  ON ij1.ipParticipantID = ij2.occpParticipantID

WHERE (

    (

        (ij2.RoleTypeID) = 52
        OR
        (ij2.RoleTypeID) = 49

    )

)
  AND ij2.HasSucceeded = 1

GROUP BY
    ij1.Description,
    ij2.occSimulationID,
    ij2.occSimRunID,
    ij2.Period;

根据 JohnFx（上图），我对派生的观点感到困惑。我认为实际上没有必要，特别是因为它们都是内部连接。所以，下面我试图降低复杂性。请检查并测试性能。我不得不与 tmpSimArgs 进行交叉连接，因为它只连接到 Occurence - 我认为这是所需的行为。

SELECT
    pg.Description,
    occ.SimulationID,
    occ.SimRunID,
    occ.Period,
    COUNT(occp.ParticipantID) AS CountOfParticipantID

FROM ParticipantGroup AS pg

INNER JOIN InitialParticipant AS ip
        ON pg.ParticipantGroupID = ip.ParticipantGroupID

CROSS JOIN tmpSimArgs AS tsa

INNER JOIN Occurrence AS occ
        ON tsa.SimRunID = occ.SimRunID
       AND tsa.SimulationID = occ.SimulationID

INNER JOIN OccurrenceParticipant AS occp
        ON occ.OccurrenceID = occp.OccurrenceID
       AND occ.SimRunID = occp.SimRunID
       AND occ.SimulationID = occp.SimulationID

INNER JOIN tmpPartArgs AS tpa
        ON tpa.participantID = occp.ParticipantID

WHERE occ.HasSucceeded = 1
  AND (occp.RoleTypeID = 52 OR occp.RoleTypeID = 49 )

GROUP BY
    pg.Description,
    occ.SimulationID,
    occ.SimRunID,
    occ.Period;

【讨论】：

【解决方案2】：

我提出了我的查询的一个较小的缩小版本。希望这比我之前的更清晰易读。

SELECT5 * FROM 
(
SELECT4 FROM ParticipantGroup as pg INNER JOIN InitialParticipant AS ip ON pg.ParticipantGroupID = ip.ParticipantGroupID
) AS ij1 INNER JOIN 
(
   SELECT3 * FROM tmpPartArgs AS tpa INNER JOIN 
      (
          SELECT2 * FROM 
              (
                  SELECT1 * FROM tmpSimArgs AS tsa INNER JOIN Occurrence AS occ ON (tsa.SimRunID = occ.SimRunID) AND (tsa.SimulationID = occ.SimulationID)
              ) AS ij4 INNER JOIN OccurrenceParticipant AS occp ON (occOccurrenceID =      occpOccurrenceID) AND (occSimRunID = occpSimRunID) AND (occSimulationID = occpSimulationID)
      ) AS ij3 ON tpa.participantID = ij3.occpParticipantID
) AS ij2 ON ij1.ipParticipantID = ij2.occpParticipantID WHERE (((ij2.RoleTypeID)=52 Or (ij2.RoleTypeID)=49)) AND ij2.HasSucceeded = 1

我正在开发的应用程序是一个模拟应用程序，为了理解上述查询的上下文，我认为有必要对应用程序进行简要说明。让我们假设有一个星球有一些初始资源和生命体。地球被允许存在 1000 年，并且代理执行的操作被监控并存储在数据库中。 1000 年后，地球被摧毁，并再次使用与第一次相同的初始资源和生命代理重新创建。这（创建和销毁）重复了 18 次，并且代理在这 1000 年中执行的所有操作都存储在数据库中。因此，我们的整个实验由 18 个重新创建组成，称为“模拟”。地球被重建的 18 次中的每一次都被称为一次运行，而运行的 1000 年中的每一次被称为一个周期。因此，“模拟”由 18 次运行组成，每次运行由 1000 个周期组成。在每次运行开始时，我们为“模拟”分配一组初始的知识项和动态代理，这些代理与彼此以及这些项进行交互。知识项由代理存储在知识库中。知识库也被认为是我们模拟中的参与实体。但是这个概念（关于知识库）并不重要。我试图详细了解每个 SELECT 语句和所涉及的表。

SELECT1：我认为这个查询可以只替换为“Occurrence”表，因为它什么都不做。 Occurrence 表存储代理在特定“模拟”的每次模拟运行的每个时期采取的不同行动。通常每个“模拟”由 18 次运行组成。每次运行由 1000 个周期组成。允许代理在“模拟”中每次运行的每个时间段内采取行动。但是 Occurrence 表不存储有关执行操作的代理的任何详细信息。 Occurrence 表可能存储与多个“模拟”相关的数据。

SELECT2：此查询仅返回“模拟”每次运行的每个时期执行的操作的详细信息以及“模拟”的所有参与者的详细信息，例如他们各自的参与者 ID。 OccurrenceParticipant 表存储模拟的每个参与实体的记录，其中包括代理、知识库、知识项等。

SELECT3：此查询仅返回伪表 ij3 中属于代理和知识项的记录。 ij3中关于知识项的所有记录都会被过滤掉。

SELECT4：此查询将“Description”字段附加到“InitialParticipant”的每条记录。请注意，“描述”列是整个查询的输出列。 InitialParticipant 表包含每个代理和最初分配给“模拟”的每个知识项的记录

SELECT5：此最终查询返回伪表 ij2 中参与实体（可能是代理或知识项）的 RoleType 为 49 或 52 的所有记录。

【讨论】：

【解决方案3】：

我建议将 ij2.RoleTypeID 过滤从最外层查询移至 ij3，使用 IN 而不是 OR，并将 HasSucceeded 查询移至 ij4。

【讨论】：