【问题标题】:Reduce MySQL Query Runtime减少 MySQL 查询运行时间
【发布时间】:2020-07-27 06:30:26
【问题描述】:

我有一个这样的查询:

SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
    JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer
                               and cr.isDeleted = 0
    JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion
                                and rbase.isDeleted = 0
where (
    (rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate   >= '2020-07-06 00:00:00') or
    (rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate   >= '2020-07-28 00:00:00') or
    (rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59') or
    (rbase.endDate   >= '2020-07-06 00:00:00' and rbase.endDate   <= '2020-07-28 23:59:59')
)
  • 数据库:MySQL

  • 客户:132,000 行

  • CustomersRegion:1,754,000 行

  • CalendarItems:3,838,000 行(条件减少到 555,000 行)

  • t.idCustomer & cr.idCustomer & cr.isDeleted & rbase.idAgentsRegion & cr.idRegion & rbase.isDeleted 是索引

此查询运行时间约为 100 秒,我想减少此查询的运行时间

我不能限制行数或在表格中有其他条件

你能帮帮我吗?

谢谢

解释查询:

客户 DDL:

create table customers
(
    idCustomer           int auto_increment
        primary key,
    CustomerName         varchar(255)                                  not null comment 'نام فروشگاه',
    FirstName            varchar(60)                                   null comment 'نام رابط',
    LastName             varchar(60)                                   null comment 'نام مشتري',
    idUser               int                                           null comment '!#dont show',
    idPayment            int                                           null,
    idCompany            int             default 0                     not null,
    LatitudePoint        decimal(18, 12) default 0.000000000000        null comment 'gpslat',
    LongitudePoint       decimal(18, 12) default 0.000000000000        null comment 'gpslongs',
    LastOrderDate        datetime        default '0000-00-00 00:00:00' null comment 'lastorderdate',
    VisitPeriod          int             default 0                     null comment 'visitperiod',
    LastVisit            datetime        default '0000-00-00 00:00:00' null comment 'LastVisitDate',
    LastNoOrderDate      datetime        default '0000-00-00 00:00:00' null,
    Credit               decimal(20, 4)  default 0.0000                null comment 'credit',
    RemainCredit         decimal(20, 4)  default 0.0000                null comment 'remaincredit',
    Balance              decimal(20, 4)  default 0.0000                null comment '!#dont show',
    RFID                 varchar(60)                                   null comment 'rfid',
    ReturnCheck          tinyint(1)      default 0                     null comment '!#dont show',
    AccountStatus        tinyint(1)      default 0                     null comment 'accountstatus',
    FaxNumber            varchar(20)                                   null,
    LiquidationDate      date            default '0000-00-00'          null comment '!#dont show',
    EldestDue            date            default '0000-00-00'          null comment '!#dont show',
    MaturityDate         date            default '0000-00-00'          null comment '!#dont show',
    PriceKind            int                                           null,
    isDefault            tinyint(1)      default 0                     not null comment '!#dont show',
    TimeStamp            timestamp       default current_timestamp()   not null on update current_timestamp(),
    isDeleted            tinyint(1)      default 0                     not null,
    Address              varchar(255)                                  null,
    PhoneNumber          varchar(60)                                   null,
    MobileNumber         varchar(60)                                   null,
    CustomerErpCode      varchar(60)                                   null comment '!#dont show',
    StoreType            int                                           null,
    country              varchar(255)                                  null,
    state                varchar(255)                                  null,
    City                 varchar(30)                                   null,
    Region               varchar(30)                                   null,
    idUserCreator        int                                           null,
    idBranche            int                                           null,
    idTagsinfo           int                                           null,
    shop_id              int                                           null,
    shop_id_address      int                                           null,
    lastActivityDate     datetime                                      null,
    lastActivityType     tinyint(1)                                    null,
    duplicateOf          int                                           null,
    isConfirmed          tinyint(1)      default 2                     not null comment '0:rejected - 1:confirmed - 2:notChecked',
    Status               tinyint(1)      default 1                     not null,
    createDate           datetime                                      null,
    idProcess            int                                           null comment 'نیازی نیست به اینکه حتما پروسه داشته باشد',
    idUserConfirmer      int                                           null comment 'this is refered to agents table',
    nextDate             datetime                                      null,
    prevDate             datetime                                      null,
    idImage              int                                           null,
    idColor              int                                           null,
    idRate               int                                           null,
    LastImageDate        datetime                                      null,
    LastOrderAgentName   varchar(255)                                  null,
    LastVisitAgentName   varchar(255)                                  null,
    LastNoOrderAgentName varchar(255)                                  null,
    LastImageAgentName   varchar(255)                                  null,
    LastOrderIdAgent     int                                           null,
    LastVisitIdAgent     int                                           null,
    LastNoOrderIdAgent   int                                           null,
    LastImageIdAgent     int                                           null,
    isSaleActive         tinyint(1)      default 1                     null,
    isReturnActive       tinyint(1)      default 1                     null,
    alley                varchar(256)                                  null,
    street               varchar(256)                                  null,
    plaque               varchar(256)                                  null,
    secondAddress        varchar(255)                                  null,
    description          varchar(255)                                  null,
    appType              varchar(50)     default 'iorder'              not null,
    idPipeline           varchar(255)    default '0'                   null,
    constraint shop_id
        unique (shop_id),
    constraint shop_id_address
        unique (shop_id_address),
    constraint ux_customererp
        unique (CustomerErpCode),
    constraint customers_ibfk_1
        foreign key (idBranche) references branches (idBranche)
            on update set null on delete set null,
    constraint customers_ibfk_2
        foreign key (idTagsinfo) references tagsinfo (idTag)
            on update set null on delete set null,
    constraint customers_ibfk_3
        foreign key (idRate) references rates (idRate)
            on update set null on delete set null,
    constraint customers_ibfk_4
        foreign key (idColor) references colors (idColor)
            on update set null on delete set null,
    constraint customers_ibfk_5
        foreign key (idRate) references rates (idRate)
            on update set null on delete set null,
    constraint customers_ibfk_6
        foreign key (idColor) references colors (idColor)
            on update set null on delete set null,
    constraint fk_customer_agents
        foreign key (idUser) references agents (idAgents)
            on update set null on delete set null,
    constraint fk_customer_paymant
        foreign key (idPayment) references payment (idPayment),
    constraint fk_customer_pricelist
        foreign key (PriceKind) references pricelist (idPriceList),
    constraint fk_customer_storeinfo
        foreign key (StoreType) references storesinfo (idStore)
)
    charset = utf8;

create index fk_customer_agents_idx
    on customers (idUser);

create index fk_customer_paymant_idx
    on customers (idPayment);

create index fk_customer_pricelist_idx
    on customers (PriceKind);

create index fk_customer_storeinfo_idx
    on customers (StoreType);

create index idBranche
    on customers (idBranche);

create index idColor
    on customers (idColor);

create index idProcess
    on customers (idProcess);

create index idRate
    on customers (idRate);

create index idTagsinfo
    on customers (idTagsinfo);

create index idx_isdeleted_customername
    on customers (isDeleted, CustomerName);

create index isdeleted_lat_lng
    on customers (isDeleted, LatitudePoint, LongitudePoint);

create index isdeleted_status_isconfirmed
    on customers (isDeleted, Status, isConfirmed);

create index lat_lng
    on customers (LatitudePoint, LongitudePoint);

日历项 DDL:

create table calendaritems
(
    id             int auto_increment
        primary key,
    TimeStamp      timestamp  default current_timestamp() not null on update current_timestamp(),
    isDone         tinyint(1)                             null,
    isDeleted      tinyint(1) default 0                   not null,
    subject        varchar(255)                           null,
    startDate      datetime                               not null,
    endDate        datetime                               not null,
    isAllDayEvent  tinyint(1) default 1                   null,
    message        varchar(255)                           null,
    color          varchar(200)                           null,
    rMessage       varchar(255)                           null,
    rTime          datetime                               null,
    rLocationLat   decimal(18, 12)                        null,
    rLocationLong  decimal(18, 12)                        null,
    idAgent        int                                    not null,
    idCustomer     int                                    null,
    idVisitPath    int                                    null,
    isFinal        tinyint(1)                             null,
    idUserCreator  int                                    not null,
    idAgentsRegion int                                    null,
    type           int(5)     default 1                   not null,
    systemFill     tinyint(1) default 0                   not null,
    createDate     datetime                               null,
    reqUp          tinyint(1) default 0                   not null,
    dependOn       int                                    null,
    idPlan         int                                    null comment 'to keep track of customer types of a region inside a plan',
    idPlanTour     int                                    null,
    startTime      time                                   null,
    endTime        time                                   null,
    constraint calendaritems_ibfk_agents
        foreign key (idAgent) references agents (idAgents),
    constraint calendaritems_ibfk_agents2
        foreign key (idUserCreator) references agents (idAgents),
    constraint calendaritems_ibfk_customers
        foreign key (idCustomer) references customers (idCustomer)
            on delete set null
)
    charset = utf8;

create index `Index 10`
    on calendaritems (isDeleted, idAgent, startDate, idCustomer);

create index `Index 14`
    on calendaritems (isDeleted, idAgent, idAgentsRegion, idPlan, startDate, endDate);

create index `Index 7`
    on calendaritems (startDate);

create index `Index 8`
    on calendaritems (isDeleted, idAgent, startDate, idVisitPath);

create index `Index 9`
    on calendaritems (isDeleted, idAgent, startDate, idAgentsRegion);

create index createDate
    on calendaritems (createDate);

create index idAgent
    on calendaritems (idAgent);

create index idAgentsRegion
    on calendaritems (idAgentsRegion);

create index idCustomer
    on calendaritems (idCustomer);

create index idUserCreator
    on calendaritems (idUserCreator);

create index idVisitPath
    on calendaritems (idVisitPath);

create index reqUp
    on calendaritems (reqUp);

create index `systemFill-startDate-idAgent-idPlan`
    on calendaritems (systemFill, startDate, idAgent, idPlan);


CustomersRegion DDL:

create table customersregion
(
    idCustomer int                                      not null,
    idRegion   int                                      not null,
    idCompany  int          default 0                   null,
    isDeleted  tinyint(1)   default 0                   null,
    TimeStamp  timestamp    default current_timestamp() null on update current_timestamp(),
    ERPCode    varchar(255) default ''                  null,
    createDate datetime                                 null,
    primary key (idCustomer, idRegion),
    constraint customersregion_ibfk_1
        foreign key (idCustomer) references customers (idCustomer)
            on update cascade on delete cascade,
    constraint customersregion_ibfk_2
        foreign key (idRegion) references region (idRegion)
            on update cascade on delete cascade
)
    charset = utf8;

create index idRegion
    on customersregion (idRegion);

create index isdeleted_idregion_idcustomer
    on customersregion (isDeleted, idRegion, idCustomer);

【问题讨论】:

  • 检查你的查询执行计划
  • 请包含准确的索引定义。
  • 添加全表的 DDL。
  • 您的某些日期条件看起来很奇怪;你能用英语解释一下他们应该做什么吗?
  • 不幸的是,在不知道所有涉及的表的 SHOW CREATE TABLE 输出的情况下,EXPLAIN 计划有点没用......

标签: mysql sql database mariadb


【解决方案1】:

EXPLAIN 计划表明,采取的第一步是扫描 calendaritems 表(“rbase”),估计总共扫描了 160 万行。

有一个索引正在使用,但它不是很合适,因为它有太多没有真正使用的额外列。一个更好的索引应该是由 (isDeleted, startDate, endDate, idAgentsRegion) 组成的索引,按照这个顺序,前三列对于 WHERE 条件的前三个 OR 部分来说是完美的,但不幸的是对于最后一个不合适。

WHERE 或 JOIN 条件根本不需要 idAgentsRegion 列,通过添加它可以使索引成为“覆盖”索引,这样所有需要的数据都可以单独从索引中检索,而无需额外的查找步骤对于实际的表格行。

在这种情况下,我要做的是有两个索引,一个在 (isDeleted, startDate, endDate, idAgentsRegion) 上,一个在 (isDeleted, startDate, endDate, idAgentsRegion) 上,然后将查询拆分为两个单独的由 UNION 组合的:

SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
    JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
    JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
where (
    (rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate   >= '2020-07-06 00:00:00') or
    (rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate   >= '2020-07-28 00:00:00') or
    (rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59') 
)
UNION
SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
FROM `customers` `t`
    JOIN `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
    JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
where (rbase.endDate   >= '2020-07-06 00:00:00' and rbase.endDate   <= '2020-07-28 23:59:59')

对于第一部分,第一个索引是完美的,对于第二部分,第二个索引是完美的,导致索引范围扫描小得多,最后只需要合并结果并删除重复项。

【讨论】:

    【解决方案2】:

    首先,从customersreqion 到customers 之间存在外键关系,因此您的查询中不需要customers 表。您不会从中选择任何内容,并且外键关系已经确保您不会选择任何不在客户表中的客户 ID。这不会显着减少您的 100 秒,但每一点都有帮助。

    要充分利用索引,您需要两个额外的索引:

    CREATE INDEX firstindextoadd ON calendaritems(idAgentsRegion, isDeleted, startDate, endDate);
    CREATE INDEX secondindextoadd ON calendaritems(idAgentsRegion, isDeleted, endDate);
    

    第一个索引将用于您的前 3 个条件:

    (rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate   >= '2020-07-06 00:00:00') or
    (rbase.startDate <= '2020-07-28 00:00:00' and rbase.endDate   >= '2020-07-28 00:00:00') or
    (rbase.startDate >= '2020-07-06 00:00:00' and rbase.startDate <= '2020-07-28 23:59:59')
    

    第二个将是你的第四个条件:

     (rbase.endDate   >= '2020-07-06 00:00:00' and rbase.endDate   <= '2020-07-28 23:59:59')
    

    是否应包含 isDeleted 取决于已删除记录的数量,但我添加它们是“以防万一”。

    它没有在庞大的数据集上进行测试,所以你需要告诉我这是否适合你。

    此外,您可以将条件简化为:

    SELECT DISTINCT `cr`.`idCustomer`, `rbase`.`id`
    FROM `customersregion` `cr` ON t.idCustomer = cr.idCustomer and cr.isDeleted = 0
        JOIN `calendaritems` `rbase` ON rbase.idAgentsRegion = cr.idRegion and rbase.isDeleted = 0
    where 
        rbase.startDate <= '2020-07-06 00:00:00' and rbase.endDate >= '2020-07-28 00:00:00' OR
        rbase.startDate BETWEEN '2020-07-06 00:00:00' and '2020-07-28 00:00:00' OR
        rbase.endDate BETWEEN '2020-07-06 00:00:00' and '2020-07-28 00:00:00'
    

    【讨论】:

    • MySQL 很少在单个SELECT 中使用两个索引。
    • @RickJames 我刚刚检查了GitHub 上的规划器实现,并且有可能只使用第一个索引,在这种情况下需要拆分查询。但是您需要使用查询和索引来查看实际发生的情况。
    • 当表中满足它的行数少于另一个表时,它可能会更早地选择带有startDate 的索引。再次尝试查询,但将两个日期都向后移几个月或几年。
    • 至于那个 GitHub,除了这六个词“为单个选择创建计划”之外,还有什么可以解释如何运行它或它适用于哪些版本或为什么要使用它而不是 @987654328 @?
    【解决方案3】:
    rbase:  INDEX(isDeleted, startDate, endDate, idAgentsRegion, id)
    rbase:  INDEX(isDeleted, endDate, startDate, idAgentsRegion, id)
    

    这些人具有以下品质:

    • 前两列在 ON 和 WHERE 中很有用。
    • 优化器将根据startDate 或endDate 是否更具选择性在它们之间进行选择。
    • 覆盖

    假设优化器将以rbase 开头。相反,如果它以 cr 开头,则让优化器在这两者之间进行选择:

    rbase:  INDEX(idAgentsRegion, isDeleted, startDate, endDate, id)
    rbase:  INDEX(idAgentsRegion, isDeleted, endDate, startDate, id)
    

    cr 是优化器可能开始使用的唯一其他表。 (有一个WHERE 子句可供过滤。)

    cr:  INDEX(isDeleted, idRegion,  -- first, (in either order)
               idCustomer)           -- last
    

    假设 start

    WHERE rbase.startDate  < '2020-07-28'
      AND rbase.endDate   >= '2020-07-06'
    

    (我不认识“2020-07-28”与“2020-07-28 23:59:59”的有趣业务。)

    我建议始终使用“=午夜”。一个简单的日期相当于那天早上的午夜。指定'2020-07-28' 的另一种方法是'2020-07-06' + INTERVAL 22 DAY。当您知道跨度(22 天)并且不想对闰日等大惊小怪时,后者很方便。

    ON 指定表如何“相关”,WHERE 用于过滤是“适当的”。也就是说,isDeleted 测试属于WHERE 子句。 (JOIN 的执行不受影响,但对LEFT JOIN 很重要。)

    Last...Id... 和 Last...Name 列似乎是多余的?在其他地方有从id 到name 的映射?

    Rates 和 Colors -- 这些看起来是不值得标准化的东西?如果您需要搜索其中任何一个,撤消此规范化将有助于提高性能,可能很多。

    这个组合似乎“错误”;有什么原因吗:

    startDate  DATETIME
    startTime  TIME
    

    当你同时拥有这两个时,

    INDEX(a)     -- drop
    INDEX(a,b)   -- keep (it takes care of the other case)
    

    LatitudePoint decimal(18, 12) 占用 9 个字节;严重的矫枉过正。建议的替代方案:http://mysql.rjweb.org/doc.php/latlng#representation_choices

    【讨论】:

      【解决方案4】:

      除非您可以假设 startDate 不超过例如在 endDate 前 30 天,没有实用的方法为此查询建立索引以避免检查所有行。您可以尝试 (startDate,endDate) 上的复合索引,这可能会有所帮助。

      您可以尝试使用开始日期索引和使用结束日期索引的一些 where 条件进行联合,但如果您真的希望 380 万行中的 50 万行被选中,它可能无济于事完全没有。

      【讨论】:

        猜你喜欢
        • 1970-01-01
        • 1970-01-01
        • 2014-07-04
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        • 1970-01-01
        相关资源
        最近更新 更多