非常大的 SQL 查询的问题

Question

问题：

我正在创建一个 Trac 报告，该报告显示我们图书馆每个章节的开发周期的每个阶段有多少票证。 工单代表一项工作，通常是一个单独的例程。

例如，即将发布的版本（里程碑）有多少票处于第 X 章的同行评审阶段。

共有 10 个发展阶段和 47 个章节。

给定的 MySQL 查询针对所有 10 个开发阶段，但仅针对一章，有 25 行长，因此所有章节的整个查询超过 1200 行。

Trac 给出的错误是KeyError: 'numrows'，其中查询变得很大。

当直接将查询输入 MySQL 时，给出的错误是 Out of resources when opening file (Errcode: 24) (23)

问题：

• 重构 - 这是否可以“更好”地完成 sql 专家，是否有一些巧妙的技巧/先进技术？

• 方法 - 我是否需要完全不同的方法？

• 配置 - 可以将 MySQL 和/或 Trac 配置为接受非常大的查询

注意事项：

表中的数据很小，在明显的大小限制下查询不会花费很长时间。

查询从 Trac 系统传递到 MySQL，这对可以执行的操作设置了一些限制，例如，只能从 trac 发送单个查询来生成报告。

可以查看 Trac 报告的示例here.

查询中的%c%* 只是我在通过脚本生成查询时用于替换实际章节的唯一字符串。

SELECT '%c%' as Chapter,
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status IN ('new','assigned') ) AS 'New',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='document_interface' ) AS 'Document\
 Interface',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='interface_development' ) AS 'Inter\
face Development',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='interface_check' ) AS 'Interface C\
heck',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='document_routine' ) AS 'Document R\
outine',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='full_development' ) AS 'Full Devel\
opment',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='peer_review_1' ) AS 'Peer Review O\
ne',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%'AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='peer_review_2' ) AS 'Peer Review Tw\
o',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='qa' ) AS 'QA',
(SELECT count(ticket.id) AS Matches FROM engine.ticket INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%'AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='closed' ) AS 'Closed',
count(id) AS Total,
ticket.id AS _id
FROM engine.ticket
INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine'

Answer 1

不要为每个计数创建一个子查询，而是使用 case 从已经为查询获取的数据中进行计数：

select '%c%' as Chapter,
  sum(case when ticket.status IN ('new','assigned') then 1 else 0 end) as 'New',
  sum(case when ticket.status='document_interface' then 1 else 0 end) as 'DocumentInterface',
  sum(case when ticket.status='interface_development' then 1 else 0 end) as 'Interface Development',
  sum(case when ticket.status='interface_check' then 1 else 0 end) as 'Interface Check',
  sum(case when ticket.status='document_routine' then 1 else 0 end) as 'Document Routine',
  sum(case when ticket.status='full_development' then 1 else 0 end) as 'Full Development',
  sum(case when ticket.status='peer_review_1' then 1 else 0 end) as 'Peer Review One',
  sum(case when ticket.status='peer_review_2' then 1 else 0 end) as 'Peer Review Two',
  sum(case when ticket.status='qa' then 1 else 0 end) as 'QA',
  sum(case when ticket.status='closed' then 1 else 0 end) as 'Closed',
  count(id) as Total,
  ticket.id as _id
from
  engine.ticket
  inner join engine.ticket_custom on ticket.id = ticket_custom.ticket
where
  ticket_custom.name='chapter' and
  ticket_custom.value LIKE '%c%' and
  type='New material' and
  milestone='1.1.12' and
  component NOT LIKE 'internal_engine'

Answer 2

不会完全重写它...但这是我的建议：

SELECT '%c%' as Chapter,
    SUM(CASE WHEN ticket.status IN ('new','assigned') THEN 1 ELSE 0 END) as `New`,
    ...
    SUM(CASE WHEN ticket.status='closed' THEN 1 ELSE 0 END) as 'Closed',
    count(id) AS Total,
    ticket.id AS _id
FROM engine.ticket
INNER JOIN engine.ticket_custom 
    ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' 
    AND ticket_custom.value LIKE '%c%' 
    AND type='New material' 
    AND milestone='1.1.12' 
    AND component NOT LIKE 'internal_engine'
GROUP BY ticket.id
;

Answer 3

在通过 Trac 生成此类复杂报告时，最好不要使用报告。报告对于相对简单的查询来说是可以的，但是当您组织许多不同的阶段和章节时，它们会变得笨拙。

请尝试使用 wiki 页面创建您的“报告”。这将使您能够更好地控制布局和表示，而且您可以完全避免编写 SQL。以下是虚假报告的一些示例 wiki 代码，显示所有未关闭的工单，按里程碑分组，然后按状态分组：

= Custom Report =
My custom report, as a wiki page

== Tickets for Milestone A ==
[[TicketQuery(milestone=MilestoneA,status!=closed,group=status,format=table)]]

== Tickets for Milestone B ==
[[TicketQuery(milestone=MilestoneB,status!=closed,group=status,format=table)]]

...

我不知道您是如何定义阶段和章节的，因此您需要调整查询参数以使用适当的工单字段。您还可以使用format 和其他选项来调整输出格式。

这只是一个简单的例子。 TicketQuery 宏能够生成更复杂的报告。有关可能更接近（复杂性）您正在寻找的东西的示例，请查看Trac project's release notes。整个“更改的详细列表”部分是使用单个 TicketQuery 宏生成的（点击“编辑”按钮查看他们是如何做到的）。

---

另一个例子

要重新创建您链接到的图表，您可以执行以下操作：

||= **Id** =||= **Enhancements** =||= **Defects** =||= **Tasks** =||
||[milestone:v1.0 v1.0] || [[TicketQuery(milestone=v1.0,type=enhancement,format=count)]] || [[TicketQuery(milestone=v1.0,type=defect,format=count)]] || [[TicketQuery(milestone=v1.0,type=task,format=count)]] ||
... repeat for each milestone ...

基本上，在宏中使用 format=count 可以为您提供数据库为 Guffa 的答案中的 sum(case ...) 语句之一返回的内容。

使用宏而不是直接 SQL 这样做的最大好处是它变得与数据库无关。您不必担心数据库引擎（mysql、sqlite 等）之间的差异、Trac 数据库布局的变化、处理自定义字段的特殊代码等。

Answer 4

是的，这很难看。您是否必须在一行中获取所有数据？这么多子查询将猛击服务器。你不能做一个标准的分组提取并在客户端做数据透视表吗？

例如

SELECT count(ticket.id) AS Matches, ticket_custom.name, ticket.status
FROM engine.ticket
INNER JOIN engine.ticket_custom ON ticket.id = ticket_custom.ticket
WHERE ticket_custom.name='chapter' AND ticket_custom.value LIKE '%c%' AND type='New material' AND milestone='1.1.12' AND component NOT LIKE 'internal_engine' AND ticket.status='qa' AND (ticket.status IN (........))
GROUP BY ticket.id, ticket_custom.name

然后，在伪代码中：

data = array()
while(row = fetch($result)) {
   data[ticket.id]][ticket.status] = row[ticket.status];
}

这样，您将只运行一个查询，并在获取逻辑中做一些小的额外工作，以重新创建原始查询正在执行的操作。

Answer 5

我最终在php 中编写了自己的跟踪报告系统。这样可以更灵活地创建我需要的报告类型，并且不涉及荒谬的mysql queries。将脚本与 trac 项目集成在一起，报告项目即第 XX 章中 QA 阶段的工单数量是指向门票自己在 trac：http://myhost.co.uk/trac-project/query?id=10&id=15。此外，NavAddPluggin 允许自定义主要的 trac 导航栏，因此可以添加链接到脚本的菜单选项。