将请求分成 1 MB 的块（和 cursor.close()）后仍然会发生 SQLiteBlobTooBigException答案

【问题标题】：SQLiteBlobTooBigException still occurs after dividing the request in chunks of 1 MB (and cursor.close())将请求分成 1 MB 的块（和 cursor.close()）后仍然会发生 SQLiteBlobTooBigException
【发布时间】：2019-12-15 22:55:13
【问题描述】：

可以将 6MB 的文本文件直接导入我的数据库。但是，由于 CursorWindow 的限制为 2MB，因此无法提取文本。（我应该使用文件，但有些用户已经遇到了这个问题，我需要阅读整个文本才能将其放入文件中）我使用 substr（一个特殊的 SQL 函数）只请求 1 MB 并且它有效。但是，以下 while 循环在第二次迭代后不起作用（这意味着即使我调用 cursor.close()，CursorWindow 也没有清空，所以对于第一次迭代它只有 1MB，但在第二次之后它有 2MB 并且抛出异常 SQLiteBlobTooBigException）：

        //Load in chunks
        BookDbHelper bookDbHelper = new BookDbHelper(GlobalContext.get());
        SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
        //Query length
        int chunk_size = (int) Math.pow(2, 20);//mb
        String query_length = "SELECT _id, length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
        Cursor cursor_length = readableDatabase.rawQuery(query_length, null);
        cursor_length.moveToFirst();
        int length = cursor_length.getInt(1);
        cursor_length.close();
        bookDbHelper.close();
        readableDatabase.close();
        //Query text
        int numSteps = length / chunk_size + 1;
        int i = 0;
        while(i < numSteps) {
            BookDbHelper bookDbHelper2 = new BookDbHelper(GlobalContext.get());
            SQLiteDatabase readableDatabase2 = bookDbHelper2.getReadableDatabase();
            int from = i * chunk_size + 1;
            int to = (i + 1) * chunk_size + 1;
            //L.v(from + ", " + to);
            String query = "SELECT _id, substr(text," + from + "," + to + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
            Cursor cursor = readableDatabase2.rawQuery(query, null);
            //Read
            cursor.moveToFirst();
            String string = cursor.getString(1);
            cursor.close();
            bookDbHelper2.close();
            readableDatabase2.close();
            //stringBuilder.append(string);
            i++;
        }

相关的列是_id和text（其中包含一个非常大的字符串），相关的sql函数是length()（知道需要的迭代次数）和substr()（这样就不会发生SQLiteBlobTooBigException因为没有达到 2MB 的限制，所以立即）。

我尝试关闭 bookDbHelper 和 readableDatabase，但没有帮助。

如何强制关闭 CursorWindow，以便我发出 1MB 的请求，清空 CursorWinow，然后继续发出其他请求？

【问题讨论】：

标签： android sqlite exception android-cursor

【解决方案1】：

如何强制关闭 CursorWindow 以便我发出 1MB 的请求，清空 CursorWinow 并继续发出其他请求？

我不认为关闭光标是您的问题，就好像不关闭光标会附加到光标并展开一样。

您的问题在于您构建的查询。

总之substr函数不是from to，它是from for（for 是返回字符串的大小/长度。您的计算基于第二个值是字符的偏移量）。因此，提取的字符串的长度会增加块大小，直到它超过字符串的末尾（在此之前吹过 CursorWindow）。

见substr(X,Y,Z) substr(X,Y)

因此，使用 1MB 的第二个块（如果被视为使用偏移量）在第二次运行时注定会失败，因为它实际上是要提取的长度 (2MB)。减少到小于 1MB 将允许一些余地，但可能会破坏 CursorWindow（但会获得额外的数据）。

但是，作为替代方案，它使用单个游标，每个块作为提取行。解决方案可能是：-

    //Load in chunks
    BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
    SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
    //Query length
    StringBuilder wholeBookText = new StringBuilder();
    int chunk_size = (int) Math.pow(2, 20);//mb
    String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
    Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
    int length = 0;
    if (cursor.moveToFirst()) {
        length = cursor.getInt(0);
    }
    int numSteps = length / chunk_size + 1;
    int i = 0;
    Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);

    StringBuilder sb = new StringBuilder();
    for (i=1; i < length + 1; i+= chunk_size) {
        if (sb.length() > 1) sb.append(" UNION ALL ");
        sb.append("SELECT substr(text,")
                .append(String.valueOf(i)).append(",").append(String.valueOf(chunk_size))
                .append(") FROM ").append(BookContract.TABLE_NAME)
                .append(" WHERE _id=").append(String.valueOf(id));

    }
    sb.append(";");
    Log.d("BOOKINFOV2","SQL generated :-\n\t" + sb.toString());
    cursor = readableDatabase.rawQuery(sb.toString(),null);
    wholeBookText = new StringBuilder();
    while (cursor.moveToNext()) {
        wholeBookText.append(cursor.getString(0));
        Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
    }

而不是单个查询循环运行。这会生成一个查询，将每个块提取为一行。也就是说，它在所有查询之间建立了一个 UNION。例如

SELECT substr(text,1,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 
    UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;

取自上述测试运行。
可以看出，to（应该是for）是块大小。最后一个chunk会根据剩余数据截断。

测试运行的完整输出：-

2019-12-16 14:21:35.546 D/BOOKINFOV2: SQL generated :-
        SELECT substr(text,1,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
2019-12-16 14:21:35.555 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.585 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 1048576
2019-12-16 14:21:35.599 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.616 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 2097152
2019-12-16 14:21:35.653 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 3145728
2019-12-16 14:21:35.654 D/BOOKINFO: Obtained String who's length is 51
        Total Extracted = 3145779

如您所见，CursorWindow 会溢出，但未添加该行，下次添加并访问它。

当然，您可以采用多查询方法，在这种情况下，代码可以是：-

    //Load in chunks
    BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
    SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
    //Query length
    StringBuilder wholeBookText = new StringBuilder();
    int chunk_size = (int) Math.pow(2, 19);//mb
    chunk_size = (1024 * 1024);
    String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
    Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
    int length = 0;
    if (cursor.moveToFirst()) {
        length = cursor.getInt(0);
    }
    int numSteps = length / chunk_size + 1;
    int i = 0;
    Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);

    int from = 1, to = chunk_size;
    while (i < numSteps && length > 0) {
        if (to > length) to = length;
        String query = "SELECT substr(text," + from + "," + (chunk_size) + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
        Log.d("BOOKINFOSQL",query);
        cursor.close();
        cursor = readableDatabase.rawQuery(query, new String[]{String.valueOf(id)});
        //Read
        if (cursor.moveToFirst()) {
            wholeBookText.append(cursor.getString(0));
            Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
        }
        cursor.close();
        i++;
        from = (i * chunk_size) + 1;
        to = from + chunk_size;
    }
    if (!cursor.isClosed()) {
        cursor.close();
    }
    Log.d("BOOKINFO", "The length of the extracted data is " + wholeBookText.length());

以上结果：-

2019-12-16 14:16:15.336 D/BOOKINFO: Length of Text is 3145779 Number of Chunks = 4 Chunk Size = 1048576
2019-12-16 14:16:15.336 D/BOOKINFOSQL: SELECT substr(text,1,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.358 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 1048576
2019-12-16 14:16:15.358 D/BOOKINFOSQL: SELECT substr(text,1048577,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.382 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 2097152
2019-12-16 14:16:15.383 D/BOOKINFOSQL: SELECT substr(text,2097153,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.409 D/BOOKINFO: Obtained String who's length is 1048576
        Total Extracted = 3145728
2019-12-16 14:16:15.409 D/BOOKINFOSQL: SELECT substr(text,3145729,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.418 D/BOOKINFO: Obtained String who's length is 51
        Total Extracted = 3145779
2019-12-16 14:16:15.418 D/BOOKINFO: The length of the extracted data is 3145779

【讨论】：

不确定我是否理解“from for”中的 for 来自何处。文本大约是 6.3MB，如果我从 6MB+1 到 7MB+1 它可以工作，所以查询本身很好。当我连续执行两个 1MB 的此类查询时，就会出现问题。我不确定为什么使用“UNION”会有所不同（CursorWindow 不应该仍然超过 2MB 吗？），但我会尝试的。谢谢，
("所以第二个使用 1Mb 的块在第二次运行时注定会失败。" 为什么？ cursor.close() 不应该释放/清空 CursorWindow 吗？)
substr(2,4) 是从字符 2 到 4 个字符（4 是大小而不是偏移量），所以从字符 2 到字符 6。当您计算偏移量时，第二个是 2MB，第三个本来是3Mb 等等。所以第二个substr(1048576,2097153)（即`2 * 1048576 + 1`根据int to = (i + 1) * chunk_size + 1;）注定会超过2MB。该问题和解决方案与使用 UNION 或关闭光标无关。块中的数据逐渐增加。也许用上面的日志运行你的，你会得到 1,1048577 然后 1048577, 2097153 ..
（哦，我明白了，你是对的。谢谢你的帮助！）
在 Android 上，光标窗口限制为 2 MB。如果这些是文本文件，只需链接它们，以跳过所有处理。多本书、作者和出版日期的索引更适合 SQLite。