使用大型数据集在 MYSQL 中模拟完全连接

Question

我有三个表，我需要根据一个公共字段连接它们的数据。

示例伪表定义：

barometer_log（设备、压力浮动、采样时间时间戳）

温度日志（设备整数、温度浮点数、采样时间时间戳）

ma​​gnitude_log（设备整数、幅度浮点、utcTime 时间戳）

每个表最终将包含数十亿行，但目前每个表包含大约 500,000 行。

我需要能够将表​​中的数据（FULL JOIN）组合起来，以便将 sampleTime 合并为一列（COALESE），从而为我提供如下行： 设备、采样时间、压力、温度、幅度

我需要能够通过指定设备以及开始和结束日期来查询数据，例如 选择 .... 其中 device=1000 和 sampleTime 在 '2011-10-11' 和 '2011-10-17' 之间

我尝试了使用 RIGHT 和 LEFT 连接的不同 UNION ALL 技术 正如MySql full join (union) and ordering on multiple date columns 和MySql full join (union) and ordering on multiple date columns 中所建议的那样，但是查询花费的时间太长，我必须停止它或在运行几个小时后抛出有关临时文件大小的错误。 对我来说，查询这三个表并在可接受的时间范围内合并它们的输出的最佳方法是什么？

这是建议的完整表定义。 注意：未包含设备表。

magnitude_log

CREATE TABLE magnitude_log (
  device int(11) NOT NULL,
  magnitude float not NULL,
  sampleTime timestamp NOT NULL,  
  PRIMARY KEY  (device,sampleTime),
  CONSTRAINT magnitudeLog_device 
    FOREIGN KEY (device) 
      REFERENCES device (id) 
      ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

气压计日志

CREATE TABLE barometer_log (
  device int(11) NOT NULL,
  pressure float not NULL,  
  sampleTime timestamp NOT NULL,  
  PRIMARY KEY  (device,sampleTime),
  CONSTRAINT barometerLog_device 
    FOREIGN KEY (device) 
      REFERENCES device (id) 
      ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

温度日志

CREATE TABLE temperature_log (
  device int(11) NOT NULL,
  sampleTime timestamp NOT NULL,  
  temperature float default NULL,
  PRIMARY KEY  (device,sampleTime),
  CONSTRAINT temperatureLog_device 
    FOREIGN KEY (device) 
      REFERENCES device (id) 
      ON DELETE CASCADE
)  ENGINE=InnoDB DEFAULT CHARSET=utf8;

Answer 1

首先，在需要的时间段内，从所有 3 个表中获取 (device, sampleTime) 的所有组合：

-------- Q --------
    SELECT device, sampleTime
    FROM magnitude_log
    WHERE device = 1000
      AND sampleTime >= '2011-10-11' 
      AND sampleTime <  '2011-10-18'
UNION
    SELECT device, sampleTime
    FROM barometer_log
    WHERE device = 1000
      AND sampleTime >= '2011-10-11' 
      AND sampleTime <  '2011-10-18'
UNION
    SELECT device, sampleTime
    FROM temperature_log
    WHERE device = 1000
      AND sampleTime >= '2011-10-11' 
      AND sampleTime <  '2011-10-18'

然后用这个LEFT JOIN这3张桌子：

SELECT
    q.device
  , q.sampleTime
  , b.pressure
  , t.temperature
  , m.magnitude
FROM 
    ( Q ) AS q
  LEFT JOIN
    ( SELECT * 
      FROM magnitude_log 
      WHERE device = 1000
        AND sampleTime >= '2011-10-11' 
        AND sampleTime <  '2011-10-18'
    ) AS m
      ON (m.device, m.sampleTime) = (q.device, q.sampleTime)
  LEFT JOIN
    ( SELECT * 
      FROM barometer_log 
      WHERE device = 1000
        AND sampleTime >= '2011-10-11' 
        AND sampleTime <  '2011-10-18'
    ) AS b
      ON (b.device, b.sampleTime) = (q.device, q.sampleTime)
  LEFT JOIN
    ( SELECT * 
      FROM temperature_log_log 
      WHERE device = 1000
        AND sampleTime >= '2011-10-11' 
        AND sampleTime <  '2011-10-18'
    ) AS t
      ON (t.device, t.sampleTime) = (q.device, q.sampleTime)

您拥有的时间越长，查询与UNION 子查询的冲突就越长。您可以考虑将Q 作为一个单独的表，可能通过触发器使用其他三个表中唯一的(device, sampleTime) 组合来填充它。

Answer 2

如果您查询的是小时间范围和大量设备，您可能需要考虑反转 PK 索引以使其成为 (timeRange,device)。

您可能希望在设备或 (device,timeRange) 上使用二级索引。

Answer 3

假设表 device 包含所有您并不真正需要正确完全连接的设备，您只需要离开加入 device 上的其他表并像这样按采样时间分组：

SELECT
    d.id AS device,
    COALESCE(m.sampleTime, b.sampleTime, t.sampleTime) AS sampleTime,
    m.magnitude,
    b.pressure,
    t.temperature
FROM device AS d
    LEFT JOIN magnitude_log AS m ON d.id = m.device
    LEFT JOIN barometer_log AS b ON d.id = b.device
    LEFT JOIN temperature_log AS t ON d.id = t.device
WHERE d.id = 1000
GROUP BY device, sampleTime
HAVING sampleTime BETWEEN '2011-10-11' AND '2011-10-17'

但这可能会很慢，因为它会在时间跨度上实际匹配之前进行分组，但如果单个设备本身不会有数百万行，那应该不是问题。但是，如果是这样，我建议将 sampleTime 放在每个连接上：

SELECT
    d.id AS device,
    COALESCE(m.sampleTime, b.sampleTime, t.sampleTime) AS sampleTime,
    m.magnitude,
    b.pressure,
    t.temperature
FROM device AS d
    LEFT JOIN magnitude_log AS m ON d.id = m.device AND m.sampleTime BETWEEN '2011-10-11' AND '2011-10-17'
    LEFT JOIN barometer_log AS b ON d.id = b.device AND b.sampleTime BETWEEN '2011-10-11' AND '2011-10-17'
    LEFT JOIN temperature_log AS t ON d.id = t.device AND t.sampleTime BETWEEN '2011-10-11' AND '2011-10-17'
WHERE d.id = 1000
GROUP BY device, sampleTime
HAVING sampleTime IS NOT NULL

希望有帮助！