如何提高 mysql 中 ORDER BY 的性能？

Question

您好，我有一张大约有 10 亿行的表。在索引字段上执行 ORDER BY 大约需要 3 秒来获取 LIMIT 的 30 条记录，而没有 ORDER BY 需要 195 毫秒。我想加快速度。

谁能帮我解决这个问题？

这里是查询的简化版本（删除了一些字段和一些连接）。

SELECT DISTINCT `auctions_opportunity`.`id`,
                `auctions_opportunity`.`employer_id`,
                `auctions_opportunity`.`salary`,
                `auctions_opportunity`.`is_active`,
                `auctions_opportunity`.`is_interested`,
                `auctions_opportunity`.`interview_status`,
                `auctions_opportunity`.`previous_interview_status`,
                `auctions_opportunity`.`creation_source`,
                `auctions_opportunity`.`created_at`,
                `auctions_opportunity`.`last_modified`,
                `auctions_opportunity`.`last_instant_alert_email_at`,
                `auctions_opportunity`.`last_daily_alert_email_at`,
                `auctions_opportunity`.`last_periodic_alert_email_at`,
                `auctions_opportunity`.`reviewed_at`,
                `auctions_opportunity`.`last_modified_by_id`,
                `auctions_opportunity`.`candidate_id`,
                `auctions_opportunity`.`job_id`,
                `auctions_opportunity`.`interview_request_notes`,
                `auctions_opportunity`.`application_email_at`,
                `auctions_opportunity`.`batch_application_email_at`,
                `auctions_opportunity`.`is_location_match`,
                `auctions_opportunity`.`is_strong_match`,
                `auctions_opportunity`.`score`,
                `auctions_opportunity`.`es_score`,
                `auctions_opportunity`.`message`,
                `auctions_opportunity`.`es_maybe`,
                `candidates_candidate`.`id`,
                `candidates_candidate`.`user_id`,
                `candidates_candidate`.`phone`,
                `candidates_candidate`.`is_new`,
                `candidates_candidate`.`last_seen`,
                `candidates_candidate`.`last_agreed_to_terms_at`,
                `candidates_candidate`.`email_backend_status`,
                `candidates_candidate`.`email_suppressed_at`,
                `candidates_candidate`.`email_verified_at`,
                `candidates_candidate`.`number_verified_at`,
                `candidates_candidate`.`last_emailed_at`,
                `candidates_candidate`.`last_updated`,
                `candidates_candidate`.`internal_note`,
                `candidates_candidate`.`main_skills`,
                `candidates_candidate`.`main_skills_nopunc`,
                `candidates_candidate`.`total_experience`,
                `candidates_candidate`.`current_company`,
                `candidates_candidate`.`current_company_nopunc`,
                `candidates_candidate`.`current_designation`,
                `candidates_candidate`.`onboarding_completed_at`,
                `candidates_candidate`.`previously_onboarded`,
                `candidates_candidate`.`talent_advocate_id`,
                `candidates_candidate`.`job_function_skills`,
                `candidates_candidate`.`availability`,
                `candidates_candidate`.`deactivated_at`,
                `candidates_candidate`.`deactivated_by_id`,
                `candidates_candidate`.`deactivation_source`,
                `candidates_candidate`.`is_private`,
                `candidates_candidate`.`previous_companies`,
                `candidates_candidate`.`companies_interned_at`,
                `candidates_candidate`.`gender`,
                `candidates_candidate`.`skype_id`,
                `candidates_candidate`.`alternate_phone`,
                `candidates_candidate`.`shadow_linkedin`,
                `candidates_candidate`.`last_job_post_email`,
                `candidates_candidate`.`last_job_alert_email`,
                `candidates_candidate`.`last_joining_email_sent`,
                `candidates_candidate`.`last_hired_email_sent`,
                `candidates_candidate`.`last_reonboarding_email`,
                `candidates_candidate`.`last_indexed_at`,
                `candidates_candidate`.`resume_viewed_notification_type`,
                `candidates_candidate`.`last_resume_viewed_email_at`,
                `candidates_candidate`.`seen_go_premium_modal_at`,
                `candidates_candidate`.`seen_active_check_modal_at`,
                `candidates_candidate`.`alerts_limit_reached_at`,
                `candidates_candidate`.`calculation_done_at`,
                `candidates_candidate`.`calculation_attempted_at`,
                `candidates_candidate`.`bio`,
                `candidates_candidate`.`last_seen_activity_at`,
                `candidates_candidate`.`job_unsubscribed_at`,
                `candidates_candidate`.`monthly_alerts_unsubscribed_at`,
                `candidates_candidate`.`resume_views_unsubscribed_at`,
                `candidates_candidate`.`update_preferences_unsubscribed_at`,
                `candidates_candidate`.`onboarding_reminder_unsubscribed_at`,
                `candidates_candidate`.`is_hireable`,
                `candidates_candidate`.`recruiter_message_push_unsubscribed_at`,
                `candidates_candidate`.`resume_views_email_unsubscribed_at`,
                `candidates_candidate`.`resume_views_push_unsubscribed_at`,
                `candidates_candidate`.`profile_reminder_email_unsubscribed_at`,
                `candidates_candidate`.`profile_reminder_push_unsubscribed_at`,
                `candidates_candidate`.`newsletter_email_unsubscribed_at`,
                `candidates_candidate`.`newsletter_push_unsubscribed_at`,
                `candidates_candidate`.`product_updates_email_unsubscribed_at`,
                `candidates_candidate`.`product_updates_push_unsubscribed_at`,
                `candidates_candidate`.`push_notifications_shown_at`,
                `candidates_candidate`.`push_notifications_enabled`,
                `candidates_candidate`.`push_notifications_verified_at`,
                `candidates_candidate`.`whatsapp_number`,
                `candidates_candidate`.`whatsapp_enabled`,
                `candidates_candidate`.`whatsapp_verified_at`,
                `candidates_candidate`.`last_whatsapp_sent_at`,
                `auth_user`.`id`,
                `auth_user`.`password`,
                `auth_user`.`last_login`,
                `auth_user`.`is_superuser`,
                `auth_user`.`username`,
                `auth_user`.`first_name`,
                `auth_user`.`last_name`,
                `auth_user`.`email`,
                `auth_user`.`is_staff`,
                `auth_user`.`is_active`,
                `auth_user`.`date_joined`
FROM `auctions_opportunity`
INNER JOIN `employers_employer` ON (`auctions_opportunity`.`employer_id` = `employers_employer`.`id`)
INNER JOIN `jobs_job` ON (`auctions_opportunity`.`job_id` = `jobs_job`.`id`)
INNER JOIN `profiles_profilestatus` ON (`employers_employer`.`status_id` = `profiles_profilestatus`.`id`)
INNER JOIN `candidates_candidate` ON (`auctions_opportunity`.`candidate_id` = `candidates_candidate`.`id`)
INNER JOIN `auth_user` ON (`candidates_candidate`.`user_id` = `auth_user`.`id`)
INNER JOIN `candidates_resume` ON (`candidates_candidate`.`id` = `candidates_resume`.`candidate_id`)
LEFT OUTER JOIN `candidates_jobsearchpreferences` ON (`candidates_candidate`.`id` = `candidates_jobsearchpreferences`.`candidate_id`)
LEFT OUTER JOIN `candidates_customaction` ON (`auctions_opportunity`.`id` = `candidates_customaction`.`opportunity_id`)
LEFT OUTER JOIN `candidates_emailaction` ON (`auctions_opportunity`.`id` = `candidates_emailaction`.`opportunity_id`)
LEFT OUTER JOIN `candidates_saveaction` ON (`auctions_opportunity`.`id` = `candidates_saveaction`.`opportunity_id`)
LEFT OUTER JOIN `candidates_hideaction` ON (`auctions_opportunity`.`id` = `candidates_hideaction`.`opportunity_id`)
LEFT OUTER JOIN `candidates_hireaction` ON (`auctions_opportunity`.`id` = `candidates_hireaction`.`opportunity_id`)
WHERE (`auctions_opportunity`.`employer_id` = 4
       AND NOT (`auctions_opportunity`.`job_id` IS NULL)
       AND `profiles_profilestatus`.`name` = Approved
       AND NOT (`auth_user`.`email` = deactivated@blobinfotech.com)
       AND NOT (`candidates_candidate`.`availability` = 3)
       AND NOT (`auctions_opportunity`.`job_id` IS NULL)
       AND NOT (`candidates_resume`.`id` IS NULL)
       AND (`jobs_job`.`is_active` = TRUE
            AND `auctions_opportunity`.`is_active` = TRUE)
       AND ((((`candidates_candidate`.`is_hireable` = TRUE
               AND `candidates_candidate`.`last_seen` >= 2020-12-18 09:19:42.873898
               AND `candidates_jobsearchpreferences`.`status` = 0)
              OR (`candidates_candidate`.`is_hireable` = TRUE
                  AND (NOT (`candidates_jobsearchpreferences`.`status` = 0)
                       OR (`candidates_candidate`.`last_seen` < 2020-12-18 09:19:42.873989
                           AND `candidates_jobsearchpreferences`.`status` = 0))))
             AND `candidates_candidate`.`is_private` = FALSE)
            OR `auctions_opportunity`.`interview_status` = 1
            OR NOT (`auctions_opportunity`.`id` IN
                      (SELECT U0.`id` AS Col1
                       FROM `auctions_opportunity` U0
                       LEFT OUTER JOIN `candidates_emailaction` U1 ON (U0.`id` = U1.`opportunity_id`)
                       WHERE (U1.`reply_email_at` IS NULL
                              AND U0.`id` = (`auctions_opportunity`.`id`)))))
       AND NOT (`auctions_opportunity`.`candidate_id` IN
                  (SELECT U2.`candidate_id` AS Col1
                   FROM `candidates_blockedemployerintermediate` U2
                   WHERE U2.`employer_id` = 4))
       AND `auctions_opportunity`.`job_id` IN (40729)
       AND NOT (((`auctions_opportunity`.`creation_source` = 8
                  AND `auctions_opportunity`.`creation_source` IS NOT NULL)
                 OR (`auctions_opportunity`.`interview_status` = 1
                     AND `auctions_opportunity`.`is_active` = FALSE)))
       AND ((((`candidates_candidate`.`is_hireable` = TRUE
               AND (NOT (`candidates_jobsearchpreferences`.`status` = 0
                         AND `candidates_jobsearchpreferences`.`status` IS NOT NULL)
                    OR (`candidates_candidate`.`last_seen` < 2020-12-18 09:19:42.863410
                        AND `candidates_jobsearchpreferences`.`status` = 0)))
              OR (`candidates_candidate`.`is_hireable` = TRUE
                  AND `candidates_jobsearchpreferences`.`status` = 2))
             AND `candidates_candidate`.`is_private` = FALSE
             AND `auctions_opportunity`.`interview_status` = 0
             AND (`auctions_opportunity`.`is_location_match` = TRUE
                  OR ((`candidates_jobsearchpreferences`.`current_location` LIKE %Delhi%
                       OR `candidates_jobsearchpreferences`.`current_location` LIKE %Noida%
                       OR `candidates_jobsearchpreferences`.`current_location` LIKE %Gurgaon%
                       OR `candidates_jobsearchpreferences`.`current_location` LIKE %Faridabad%
                       OR `candidates_jobsearchpreferences`.`current_location` LIKE %Greater Noida%)
                      AND (`jobs_job`.`locations` LIKE %Delhi%
                           OR `jobs_job`.`locations` LIKE %Noida%
                           OR `jobs_job`.`locations` LIKE %Gurgaon%
                           OR `jobs_job`.`locations` LIKE %Faridabad%
                           OR `jobs_job`.`locations` LIKE %Greater Noida%)
                      AND `jobs_job`.`accept_outstation` = FALSE)))
            OR (((`candidates_candidate`.`is_hireable` = TRUE
                  AND (NOT (`candidates_jobsearchpreferences`.`status` = 0
                            AND `candidates_jobsearchpreferences`.`status` IS NOT NULL)
                       OR (`candidates_candidate`.`last_seen` < 2020-12-18 09:19:42.863410
                           AND `candidates_jobsearchpreferences`.`status` = 0)))
                 OR (`candidates_candidate`.`is_hireable` = TRUE
                     AND `candidates_jobsearchpreferences`.`status` = 2))
                AND `auctions_opportunity`.`reviewed_at` < 2020-12-18 09:19:42.863329
                AND `auctions_opportunity`.`interview_status` = 1))
       AND `auctions_opportunity`.`employer_id` = 4
       AND `candidates_customaction`.`id` IS NULL
       AND `candidates_emailaction`.`id` IS NULL
       AND `candidates_saveaction`.`id` IS NULL
       AND (`candidates_hideaction`.`id` IS NULL
            OR `candidates_hideaction`.`is_deleted` = TRUE)
       AND `candidates_hireaction`.`id` IS NULL
       AND `auctions_opportunity`.`employer_id` = 4)
ORDER BY `auctions_opportunity`.`reviewed_at` DESC
LIMIT 30

Answer 1

一般的答案相当简单。如果您拿一本电话簿并被要求说出三十个电话号码（LIMIT 30 没有ORDER BY），其中该人的街道名称包含 A，您需要多长时间？大概几分钟。现在，您被要求说出该人的街道名称中包含 A 的 30 个最低电话号码（LIMIT 30 和 ORDER BY）。您必须阅读整本书，挑选出街道名称匹配的所有数字，然后按数字对所有条目进行排序，然后取前 30 个。相比之下，这需要多长时间？

因此，无论是否使用ORDER BY，您都会看到显着差异，这并不奇怪。意料之中。

在您的情况下，DBMS 可能只选择带有 is_location_match = TRUE 的行，从而避免使用 LIKE 进行昂贵的位置搜索（如果这足以最终找到 30 行）。它不能这样做，因为它必须考虑所有匹配项才能找到ORDER BY 指定的 top 30 行。

回到电话簿：如果电话簿只有 50 个条目，工作不会太难。如果它有一百万个条目，那么搜索它需要很长时间。所以我们的目标一定是处理一个小数据集。您正在加入表 candidates_resume 而不使用它。 （条件AND NOT candidates_resume.id IS NULL 是多余的，没有任何意义。）您还将加入所有candidates_hideaction 行，其中is_deleted = TRUE。假设每个候选人有三份简历和五个已删除的隐藏操作。因此，您将数据集放大了 15 (3 x 5) 倍，只是为了稍后使用DISTINCT 消除重复项。 DISTINCT 在大型数据集上是一项非常昂贵的操作。它也是一个写得不好的查询的典型指标，就像你的情况一样：不必要的连接会破坏你的中间结果，而不是修复连接问题，你最终会应用DISTINCT。

您还使用了许多反连接（例如LEFT OUTER JOIN candidates_customaction ... WHERE candidates_customaction.id IS NULL）。这让我一开始很困惑。我想知道为什么在我找到IS NULL 条件之前，您要交叉加入所有这些候选操作。我认为这些反连接不可读。现在假设每个候选人的四种反联合动作类型中的每一种都有十个动作。如果 DBMS 进行连接，它将为必须全部解雇的单个候选人创建 10000 (10 x 10 x 10 x 10) 行。我会选择NOT EXISTS 子句来获得它的可读性，并可能避免不必要的大中间结果。我也会使用NOT EXISTS 来检查未删除的隐藏操作。

话虽如此，我的建议是您更改查询，以便您可以关闭DISTINCT。这样可以为DBMS节省大量的工作量，并且可以体现在最终的执行速度上。

您可以做一些小事来使查询更具可读性。您不必三次声明您想要employer_id = 4 :-) 然后，status = 0 AND status IS NOT NULL 当然只是status = 0。因此NOT (status = 0 AND status IS NOT NULL) OR (last_seen < xxx AND status = 0) 就是status <> 0 OR last_seen < xxx。这不会加快查询速度，但会使其更具可读性和可维护性，甚至可能您想考虑状态 NULL 并在清理条件的过程中犯了一个更明显的错误。

Answer 2

虽然看起来ORDER BY 是罪魁祸首，但很可能是其他原因导致性能不佳。

• 在可行的情况下应避免使用WHERE 和ON 中的OR。

• NOT x IN ( SELECT ... ) 应转换为 NOT EXISTS ( SELECT 1 ... ) 或 LEFT JOIN ( SELECT y ... ) ... WHERE y IS NULL

• 一些可能提供帮助的索引：

    auctions_opportunity:  INDEX(employer_id, interview_status, job_id)
    candidates_saveaction:  INDEX(opportunity_id, id)
    profiles_profilestatus:  INDEX(name)
    candidates_jobsearchpreferences:  INDEX(status)
    U1:  INDEX(reply_email_at)
  

基于大版本查询

• 构成过滤部分的 6 个LEFT JOINs 可能是性能问题的一部分。被测试的列可以移到主表吗？

• 大约有两打ORs。每个都抑制了对正在测试的任何内容进行索引的可能性。对于他们中的大多数人，我没有建设性的建议。

• status = 0 暗示 status IS NOT NULL。我怀疑一些复杂的布尔表达式可以简化。

• 有一个可能是NULL 或0 的列，但解释这两个“值”会导致不必要的复杂查询。选择NULL 或0，然后在插入数据时将两者映射到选择的值。

• 有AND NOT (auctions_opportunity.job_id IS NULL)等冗余测试。当然，优化器可能过滤掉额外的工作，但我们不要相信它。

• 标志 is_hireable、is_private、availability 能否以某种方式组合起来，既使查询更简单，又能提供详细信息？

• AND NOT (auth_user.email = deactivated@blobinfotech.com) 是语法错误——还有其他错误吗？ （缺少引号）

• 而不是多个LIKE '%...%'，单个RLIKE '...|...' 可能会更快。

• 你需要candidates_resume吗？好像没用过。

• U1 的使用似乎很奇怪。 LEFT JOIN 是 ON opportunity_id，但 IS NULL 测试位于不同的列 (reply_email_at)。这可能无法如您所愿。

• 以下一些索引可能会有所帮助。

    auctions_opportunity:  INDEX(employer_id, job_id, reviewed_at)
    jobs_job:  INDEX(is_active, locations, accept_outstation, id)
    profiles_profilestatus:  INDEX(name, id)
    candidates_jobsearchpreferences:  INDEX(candidate_id, status, current_location, candidate_id)
    blobinfotech:  INDEX(com)
    U1:  INDEX(opportunity_id, reply_email_at)
    U2:  INDEX(employer_id,  candidate_id)
  

• 改变

              AND  NOT (`auctions_opportunity`.`candidate_id` IN (
                        SELECT  U2.`candidate_id` AS Col1
                            FROM  `candidates_blockedemployerintermediate` U2
                            WHERE  U2.`employer_id` = 4)
                       )
  

到

     AND NOT EXISTS( SELECT 1 
              FROM `candidates_blockedemployerintermediate` U2
              WHERE `auctions_opportunity`.`candidate_id` =  U2.`candidate_id`
                AND  U2.`employer_id` = 4 )