多次迭代 mysqli 无缓冲查询结果

Question

问题：

我有一个返回大型结果集的查询。它太大而不能带入 PHP。我收到致命的内存最大错误，无法增加内存限制。 Unbuffered Queries

我需要多次迭代数组，但 mysqli_data_seek 不适用于无缓冲查询。 mysqli_result::data_seek

//I have a buffered result set
$bresult = $mysql->query("SELECT * FROM Small_Table");

//And a very large unbuffered result set
$uresult = $mysqli->query("SELECT * FROM Big_Table", MYSQLI_USE_RESULT);

//The join to combine them takes too long and is too large
//The result set returned by the unbuffered query is too large itself to store in PHP
//There are too many rows in $bresult to re-execute the query or even a subset of it for each one

foreach($bresult as &$row) {
    //My solution was to search $uresult foreach row in $bresult to get the values I need
    $row['X'] = searchResult($uresult, $row['Key']);
    //PROBLEM: After the first search, $uresult is at its and and cannot be reset with mysqli_result::data_seek
}

function searchResult($uresult, $val)
    while($row = $uresult->fetch_assoc()){
        if($row['X'] == $val) {
            return $row['X'];
        }
    }
}

如果您有其他满足这些要求的解决方案，我会接受： - 不尝试在单个查询中加入结果（花费太长时间） - 不对另一个查询中的每个结果运行任何查询（查询太多、耗时太长、降低系统速度）

如果您需要更多信息，请发表评论。

谢谢。

Answer 1

如果您正在尝试处理大数据集，您是否考虑过使用 Hadoop 之类的中介？您可以设置一个小型 hadoop 集群，进行处理，然后让您的 php 代码向 hadoop 输出请求处理后的数据。