为什么这个查询太慢了？获取 A 的列表，其中每个项目都有一个关联的外键 B

Question

我有Website(Id) 表，每条记录可能有多个关联的CheckLog(FK WebsiteId) 条目。 CheckLog 也有一个复合索引 [WebsiteId, CreatedTime]。 Website 只有大约 20 条记录，但加班时间 CheckLog 会增长，当时有 300 万条记录我有这个问题。 （请参阅问题末尾使用 EF Core 的架构）。

我的一个常见查询是查询所有Websites 的列表，以及零/一个 最新的CheckLog 记录：

return await this.ctx.Websites.AsNoTracking()
    .Select(q => new  WebsiteListItem()
    {
        Website = q,
        LatestCheckLog = q.CheckLogs
            .OrderByDescending(q => q.CreatedTime)
            .FirstOrDefault(),
    })
    .ToListAsync();

我相信[WebsiteId, CreatedTime] 索引应该会有所帮助。但是，查询需要大约 11 秒才能执行。这是翻译后的查询以及EXPLAIN QUERY PLAN：

      SELECT "w"."Id", "t0"."Id", "t0"."CreatedTime", "t0"."WebsiteId"
      FROM "Websites" AS "w"
      LEFT JOIN (
          SELECT "t"."Id", "t"."CreatedTime", "t"."WebsiteId"
          FROM (
              SELECT "c"."Id", "c"."CreatedTime", "c"."WebsiteId", ROW_NUMBER() OVER(PARTITION BY "c"."WebsiteId" ORDER BY "c"."CreatedTime" DESC) AS "row"
              FROM "CheckLogs" AS "c"
          ) AS "t"
          WHERE "t"."row" <= 1
      ) AS "t0" ON "w"."Id" = "t0"."WebsiteId"

MATERIALIZE 1
CO-ROUTINE 4
SCAN TABLE CheckLogs AS c USING INDEX IX_CheckLogs_WebsiteId_CreatedTime
USE TEMP B-TREE FOR RIGHT PART OF ORDER BY
SCAN SUBQUERY 4
SCAN TABLE Websites AS w
SEARCH SUBQUERY 1 AS t USING AUTOMATIC COVERING INDEX (WebsiteId=?)

这可以用 Index 解决吗？如果没有，是否有一种有效的方法来查询它而不创建 N+1 查询？我试图想办法用 2 个查询来做到这一点，但想不出任何更好的方法来像 EF Core 那样翻译它）。

另外，我认为这是一个非常常见的问题，但我不知道应该使用什么关键字来找到此类问题的解决方案。我对此类问题的通用解决方案没有意见（即获取Categories 列表中的最新Product）。谢谢。

---

我将 EF Core 用于 DB Schema：

    public class Website
    {

        public int Id { get; set; }

        // Other properties

        public ICollection<CheckLog> CheckLogs { get; set; }

    }

    [Index(nameof(CreatedTime))]
    [Index(nameof(WebsiteId), nameof(CreatedTime))]
    public class CheckLog
    {
        public int Id { get; set; }
        public DateTime CreatedTime { get; set; }

        public int WebsiteId { get; set; }
        public Website Website { get; set; }

        // Other properties

    }

Answer 1

如果您想要为每个 WebsiteId 获取具有最新 CreatedTime 的行，则无需任何联接。

只需聚合并设置条件：

HAVING MAX(CreatedTime)

这不是标准 SQL，而是利用 SQLite 的bare columns:

SELECT *
FROM CheckLogs 
GROUP BY WebsiteId
HAVING MAX(CreatedTime);

如果你想加入Websites:

SELECT  w.Id, t.Id, t.CreatedTime, t.WebsiteId 
FROM Websites AS w 
LEFT JOIN (
  SELECT *
  FROM CheckLogs 
  GROUP BY WebsiteId
  HAVING MAX(CreatedTime)
) AS t ON w.Id = t.WebsiteId;

Answer 2

感谢this answer，我知道了如何重写查询：

SELECT L.*
FROM CheckLogs L
INNER JOIN
(SELECT WebsiteId, Max(CreatedTime) AS CreatedTime
FROM CheckLogs
GROUP BY WebsiteId) L2
ON L.WebsiteId = L2.WebsiteId AND L.CreatedTime = L2.CreatedTime

由于 EF Core 无法翻译 JOIN 查询，我以这种方式重写了代码，2 次往返数据库服务器。 注意，由于 WebsiteIds 是数字，所以这样写是安全的，如果你输入字符串参数，你需要清理它们。

            var websites = await query.ToListAsync();
            var websiteIds = websites.Select(q => q.Id).ToList();

            var websiteIdsString = string.Join(",", websiteIds);
            var logQuery = this.ctx.CheckLogs.FromSqlRaw(@$"
SELECT L.*
FROM CheckLogs L
INNER JOIN
(SELECT WebsiteId, Max(CreatedTime) AS CreatedTime
FROM CheckLogs
GROUP BY WebsiteId) L2
ON L.WebsiteId = L2.WebsiteId AND L.CreatedTime = L2.CreatedTime
WHERE L.WebsiteId IN ({websiteIdsString})");

            var logs = await logQuery.AsNoTracking().ToListAsync();
            var logDict = logs.ToLookup(q => q.WebsiteId);

            return websites.Select(q => new WebsiteListItem()
            {
                Website = q,
                LatestCheckLog = logDict[q.Id].FirstOrDefault(),
            }).ToList();

Answer 3

首先尝试这个查询：

SELECT MAX("c"."CreatedTime"), "c"."WebsiteId" 从“检查日志”作为“c” GROUP BY "c"."WebsiteId"

很难将您的查询优化为比上述更快。 如果速度快，那就有希望了。

你可以试试：

with logs as (
    select max("c"."Id") "Id", max("c"."CreatedTime") "CreatedTime", "c"."WebsiteId"
    from "CheckLogs" AS "c"
    group by "c"."WebsiteId"
    having count(*) = 1)

select "w"."Id", "l"."Id", "l"."CreatedTime", "l"."WebsiteId"
from "Websites" AS "w"
join logs as "l" on "w"."Id" = "l"."WebsiteId"

union all

select "w"."Id", null, null, null
from "Websites" AS "w"
where not exists (
    select 'x'
    from "CheckLogs" AS "l"
    where "l"."WebsiteId" = "w"."Id")

你也可以试试上面union all 的两半，看看它们有多快。 如果仍然很慢，您可以添加一个包含您正在寻找的结果的新表，并在“CheckLogs”上创建触发器来填充它，这样数据就准备好了。