我的查询执行时间过长？答案

【问题标题】：My query is taking too much time to execute?我的查询执行时间过长？
【发布时间】：2018-03-12 15:01:12
【问题描述】：

我正在从拥有数百万条记录的数据库中获取用户线索，并且获取时间为 5-10 秒。我怎样才能让它更快？我应该从 ORM 切换到直接查询还是可以用当前的 ORM 做点什么？

public static function getBtqUsers($search_params) {
    $criteria = new Criteria();

    $criteria->clearSelectColumns();
    $criteria->addSelectColumn("btq_user.id as id");
    $criteria->addSelectColumn("btq_user.drid as drid");
    $criteria->addSelectColumn("btq_user.name as name");
    $criteria->addSelectColumn("btq_user.email as email");
    $criteria->addSelectColumn("btq_user.active as active");
    $criteria->addSelectColumn("btq_user.lead_from as lead_from");
    $criteria->addSelectColumn("btq_user.source as source");
    $criteria->addSelectColumn("btq_user.http_referer as http_referer");
    $criteria->addSelectColumn("btq_user.sms_status as sms_status");
    $criteria->addSelectColumn("btq_user.telefon as telefon");
    $criteria->addSelectColumn("btq_user.datain as datain");
    $criteria->addSelectColumn("btq_user.kall as kall");
    $criteria->addSelectColumn("btq_user.ip_address as ip_address");

    $criteria->addSelectColumn("count(btq_user_track_blog_video.counter) as total_video_viewed");

    $criteria->addSelectColumn("btq_doctors.id as dr_id");
    $criteria->addSelectColumn("btq_doctors.drname as dr_name");
    $criteria->addSelectColumn("btq_doctors.drphone as dr_phone");
    $criteria->addSelectColumn("btq_doctors.initials as dr_initials");
    $criteria->addSelectColumn("btq_doctors.drmail as dr_mail");

    $criteria->addSelectColumn("event_data.btq_user_id as event_user_id");
    $criteria->addSelectColumn("count(if(event_data.event_parent_id=2, event_data.event_parent_id, null)) as kms_total_video_viewed");
    $criteria->addSelectColumn("count(if(event_data.event_parent_id=1, event_data.event_parent_id, null)) as kms_total_blog_viewed");

    $criteria->addSelectColumn("state.pres as pres");
    $criteria->addSelectColumn("state.state as state");

    $criteria->addSelectColumn("btq_user_sales_choice.type as choice_type");
    $criteria->addSelectColumn("btq_user_sales_choice.opt_value as choice_value");
    $criteria->addSelectColumn("btq_user_sales_choice.opt_text as choice_text");
    $criteria->addSelectColumn("btq_user.pfu_customer_id as pfu_customer_id");
    $criteria->addSelectColumn("lead_schedule.id as schedule_id");
    $criteria->addSelectColumn("lead_schedule.created_at as schedule_date");

    $criteria->addJoin(self::STATE_ID, StatePeer::ID, Criteria::LEFT_JOIN);
    $criteria->addJoin(self::BTQ_USER_SALES_CHOICE_ID, BtqUserSalesChoicePeer::ID, Criteria::LEFT_JOIN);
    $criteria->addJoin(self::ID, LeadSchedulePeer::LEAD_ID, Criteria::LEFT_JOIN);
    $criteria->addJoin(self::DRID, BtqDoctorsPeer::ID, Criteria::LEFT_JOIN);
    $criteria->addJoin(self::ID, BtqUserTrackBlogVideoPeer::USER_ID, Criteria::LEFT_JOIN);
//    $criteria->addJoin(self::DRID, BtqDoctorPfuAdminPeer::BTQ_DOCTOR_ID, Criteria::LEFT_JOIN);
//    $criteria->addExtraJoin(BtqDoctorPfuAdminPeer::PFU_ADMIN_ID, PfuCustomerPeer::PFU_ADMIN_ID, Criteria::LEFT_JOIN, " AND " . PfuCustomerPeer::EMAIL . " = " . self::EMAIL);

    $criteria->addJoin(self::ID, EventDataPeer::BTQ_USER_ID, Criteria::LEFT_JOIN);

    //$criteria->addGroupByColumn(self::ID);
    $criteria->addGroupByColumn(self::EMAIL);
    //$criteria->addGroupByColumn(EventDataPeer::BTQ_USER_ID);
    $criteria->add(BtqUserPeer::IS_DUMMY_DETAIL, "1", Criteria::NOT_EQUAL);

    $criteria->addDescendingOrderByColumn(self::DATAIN);
    if (!empty($search_params)) {
      foreach ($search_params as $key => $param) {

        if (trim($param)) {
          $param = addslashes($param);

          switch ($key) {

            case 'name':
              $criteria->add(BtqUserPeer::NAME, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'email':
              $criteria->add(BtqUserPeer::EMAIL, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'dr_name':
              $criteria->add(BtqDoctorsPeer::DRNAME, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'phone':
              $criteria->add(BtqUserPeer::TELEFON, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'lead_from':
              $criteria->add(BtqUserPeer::LEAD_FROM, $param, Criteria::EQUAL);
              break;

            case 'start_date':
              $criteria->add(BtqUserPeer::DATAIN, $param . " 00:00:00", Criteria::GREATER_EQUAL);
              break;

            case 'end_date':
              $criteria->addAnd(BtqUserPeer::DATAIN, $param . " 23:59:59", Criteria::LESS_EQUAL);
              break;

            case 'location':
              if ($param == "Local")
                $criteria->add(StatePeer::PRES, array("MD", "VA", "DC"), Criteria::IN);
              else if ($param == "Non Local")
                $criteria->add(StatePeer::PRES, array("MD", "VA", "DC"), Criteria::NOT_IN);
              break;
          }
        }
      }
    }//FB::log($criteria);
    return $criteria;
//    return self::doSelectStmt($criteria)->fetchAll();
  }

我正在尝试获取前 2 天的记录，但仍然花费了太多时间。

我作为参数传递给 searchParams 的代码：

> $searchParams = array('start_date' => date('Y-m-d', strtotime('-2
> days')) , 'end_date' => date("Y-m-d"));
>       $this->pager->setCriteria(btqAdminBtqUserPeer::getBtqUsers($searchParams));
>       $this->form->setDefault('start_date', $searchParams['start_date']);
>       $this->form->setDefault('end_date', $searchParams['end_date']);

原始 SQL：

SELECT  btq_user.id as id, btq_user.drid as drid, btq_user.name as name,
        btq_user.email as email, btq_user.active as active, btq_user.lead_from as lead_from,
        btq_user.source as source, btq_user.http_referer as http_referer,
        btq_user.sms_status as sms_status, btq_user.telefon as telefon,
        btq_user.datain as datain, btq_user.kall as kall, btq_user.ip_address as ip_address,
        count(btq_user_track_blog_video.counter) as total_video_viewed,
        btq_doctors.id as dr_id, btq_doctors.drname as dr_name,
        btq_doctors.drphone as dr_phone, btq_doctors.initials as dr_initials,
        btq_doctors.drmail as dr_mail, event_data.btq_user_id as event_user_id,
        count(if(event_data.event_parent_id=2, event_data.event_parent_id,
                        null)) as kms_total_video_viewed,
        count(if(event_data.event_parent_id=1,
                        event_data.event_parent_id, null)
             ) as kms_total_blog_viewed,
        state.pres as pres, state.state as state, btq_user_sales_choice.type as choice_type,
        btq_user_sales_choice.opt_value as choice_value, btq_user_sales_choice.opt_text as choice_text,
        btq_user.pfu_customer_id as pfu_customer_id, lead_schedule.id as schedule_id,
        lead_schedule.created_at as schedule_date
    FROM  btq_user
    LEFT JOIN  state  ON (btq_user.STATE_ID=state.ID)
    LEFT JOIN  btq_user_sales_choice
         ON (btq_user.BTQ_USER_SALES_CHOICE_ID=btq_user_sales_choice.ID)
    LEFT JOIN  lead_schedule  ON (btq_user.ID=lead_schedule.LEAD_ID)
    LEFT JOIN  btq_doctors  ON (btq_user.DRID=btq_doctors.ID)
    LEFT JOIN  btq_user_track_blog_video
         ON (btq_user.ID=btq_user_track_blog_video.USER_ID)
    LEFT JOIN  event_data  ON (btq_user.ID=event_data.BTQ_USER_ID)
    WHERE  btq_user.IS_DUMMY_DETAIL<>1
      AND  (btq_user.DATAIN>='2008-03-11 00:00:00'
      AND   btq_user.DATAIN<='2018-03-13 23:59:59'
           )
    GROUP BY  btq_user.EMAIL
    ORDER BY  btq_user.DATAIN DESC
    LIMIT  20

查看：

<?php
      foreach ($pager->getResults() as $row):
        $customer = btqAdminBtqUserPeer::getQuizUserGroupByEmaiIdAndTelephone($row['email'], $params);
        if ($customer['quizes']) {
          $quizzes = explode(",", $customer['quizes']);
          $quizzes_count = count($quizzes);
        } else {
          $quizzes = '';
          $quizzes_count = 0;
        }

        $scheduleClass = ($row['schedule_date']) ? 'schedule_available':'schedule_empty';
        ?>

从模板/视图中，我正在调用以下函数，这会大大降低查询速度：

public static function getQuizUserGroupByEmaiIdAndTelephone($userEmailId, $searchParameters) {
    $criteria = new Criteria();
    $criteria->addSelectColumn(" IF( COUNT(btq_user.id) > 1, GROUP_CONCAT(CONCAT(btq_user.id),'~',CONCAT(btq_user.datain)) , null) as quizes");
    $criteria->add(BtqUserPeer::EMAIL, $userEmailId, Criteria::EQUAL);
    $criteria->add(BtqUserPeer::IS_DUMMY_DETAIL, "1", Criteria::NOT_EQUAL);
    if (!empty($searchParameters)) {
      foreach ($searchParameters as $key => $param) {

        if (trim($param)) {
          $param = addslashes($param);

          switch ($key) {

            case 'name':
              $criteria->add(BtqUserPeer::NAME, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'email':
              $criteria->add(BtqUserPeer::EMAIL, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'dr_name':
              $criteria->add(BtqDoctorsPeer::DRNAME, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'phone':
              $criteria->add(BtqUserPeer::TELEFON, "%" . $param . "%", Criteria::LIKE);
              break;

            case 'lead_from':
              $criteria->add(BtqUserPeer::LEAD_FROM, $param, Criteria::EQUAL);
              break;

            case 'start_date':
              $criteria->add(BtqUserPeer::DATAIN, $param . " 00:00:00", Criteria::GREATER_EQUAL);
              break;

            case 'end_date':
              $criteria->addAnd(BtqUserPeer::DATAIN, $param . " 23:59:59", Criteria::LESS_EQUAL);
              break;

            case 'location':
              $criteria->addJoin(self::STATE_ID, StatePeer::ID, Criteria::LEFT_JOIN);
              if ($param == "Local")
                $criteria->add(StatePeer::PRES, array("MD", "VA", "DC"), Criteria::IN);
              else if ($param == "Non Local")
                $criteria->add(StatePeer::PRES, array("MD", "VA", "DC"), Criteria::NOT_IN);
              break;
          }
        }
      }
    }
$criteria->addDescendingOrderByColumn(self::DATAIN);
return self::doSelectStmt($criteria)->fetch(PDO::FETCH_ASSOC);

如果需要其他任何内容，请告诉我，我将编辑问题和描述。

谢谢

【问题讨论】：

为所有表连接创建一个视图并对该视图进行过滤
检查是否是对数据库的查询需要时间（例如没有索引的 where 条件）。为此，您必须分析查询的 SQL 连接部分。如果您的查询返回数千行，则可能是对象水合需要时间，您可以考虑使用本机查询或对您的应用程序进行分页
请提供那个冗长冗长的 ORM 生成的 SQL。也许我们可以建议一个更好的 SQL 或INDEX，然后您可以转换回 ORM-speak。
@RickJames 添加了请检查更新

标签： mysql sql performance symfony symfony1

【解决方案1】：

如果您只是获得COUNT(*)，则不需要所有列。 LEFT JOINs 不会影响计数（在这种情况下）。我认为这会给你同样的计数：

        SELECT COUNT(DISTINCT btq_user.EMAIL)
            FROM  `btq_user`
            WHERE  btq_user.IS_DUMMY_DETAIL<>1
              AND  btq_user.DATAIN<='2018-02-28 23:59:59'

这个索引可能有帮助：

INDEX(DATAIN)

这没有意义：

SELECT  IF( COUNT(btq_user.id) > 1,
                   GROUP_CONCAT(CONCAT(btq_user.id),
                                '~',CONCAT(btq_user.datain)) ,
                   null ) as quizes
    FROM  `btq_user`
    WHERE  btq_user.EMAIL='melinbakar@hotmail.com'
      AND  btq_user.IS_DUMMY_DETAIL<>1
      AND  (btq_user.DATAIN>='2008-03-11 00:00:00'
      AND   btq_user.DATAIN<='2018-03-13 23:59:59'
           )
    ORDER BY  btq_user.DATAIN DESC

只有一行（由于聚合函数COUNT），所以ORDER BY 没有可使用的。也许你想要ORDER BY _inside_ theGROUP_CONCAT`？

那需要INDEX(email, datain)

【讨论】：

谢谢，但我已经更新了查询，请检查。旧的加错了。谢谢。
@DojoDev - 检查来自COUNTs 的值。 JOIN + GROUP BY 可能夸大了这些值。在我尝试加快查询速度之前，请确保它们是正确的。
请检查这里是我从调试工具栏获得的页面的详细信息。 pastebin.com/H8TzCuR1
我对其中一个问题发表了评论。
哥们，请检查新的更新，我发现了另一个让所有事情变得一团糟的查询。因为我是从视图中调用它，并且对于每条记录它都在调用它。