目标数据库大小在合并时翻倍

Question

每次使用 concat 调用 coalesce 时，与输入大小相比，1.db 的大小会增加一倍。插入 20MB 数据后，1.db 变成 30MB 大小。

输入：Lines 现在有 10MB 的数据

File file = new File("input.txt");
BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8));
char[] buffer = new char[10 * 1000 * 1000];    
StringBuilder lines = new StringBuilder();    
lines.append(String.valueOf(buffer, 0, reader.read(buffer)));

第一轮插入 -

Connection connection = DriverManager.getConnection("jdbc:sqlite:target/1.db");
Settings settings = new Settings();
settings.setExecuteLogging(false);
File reader2 = new File("target/1.db");

try (DSLContext dsl = DSL.using(connection,settings)) {

    dsl.createTable("TABLE1")
            .column("COL1", SQLiteDataType.CLOB)
            .column("COL2", SQLiteDataType.CLOB)
            .execute();


    dsl.insertInto(TABLE1)
            .columns(TABLE1.COL1, TABLE1.COL2)
            .values("ABB", null)
            .execute();

    Field<String> coalesce = DSL.coalesce(TABLE1.COL2, "");
    dsl.update(TABLE1).set(TABLE1.COL2, DSL.concat(coalesce, DSL.val(lines.toString())))
            .where(TABLE1.COL1.eq("ABB"))
            .execute();

    Result<Record2<String, String>> result = dsl.select(TABLE1.COL1, TABLE1.COL2)
            .from(TABLE1)
            .where(TABLE1.COL1.eq("ABB"))
            .fetch();

    System.out.println("input length: " + lines.length());
    System.out.println("File size:" + reader2.length());
    if (result.getValue(0, TABLE1.COL2) != null) {
        System.out.println("Col1 length: " + result.getValue(0, TABLE1.COL1).length());
        System.out.println("Col2 length: " + result.getValue(0, TABLE1.COL2).length());
    }

输出：

input length: 10000000 //10 mb payload
File size:10113024  // file size increased by 10 MB
Col1 length: 3
Col2 length:10000000 // col2 length 

再插入 10MB，但使用“coalesce”和“concat”将文件大小扩大到 30MB。无法确定此处额外添加了 10 MB 的位置。

第 2 轮插入 -

dsl.update(TABLE1).set(TABLE1.COL2, DSL.concat(coalesce, DSL.val(lines.toString())))
    .where(TABLE1.COL1.eq("ABB"))
    .execute();


result = dsl.select(TABLE1.COL1, TABLE1.COL2)
    .from(TABLE1)
    .where(TABLE1.COL1.eq("ABB"))
    .fetch();

System.out.println("input length: " + lines.length());
System.out.println("File size:" + reader2.length());
if (result.getValue(0, TABLE1.COL2) != null) {
    System.out.println("Col1 length: " + result.getValue(0, TABLE1.COL1).length());
    System.out.println("Col2 length: " + result.getValue(0, TABLE1.COL2).length());
}

输出：

input length: 10000000 //input size
File size:30322688  // increased from 10MB to 30MB for additional 10MB input 
Col1 length: 3
Col2 length:20000000 //col 2 length

和Unicode编码有关系吗？

Answer 1

开始了解 1 个 sqlite 解决方法 -

dsl.execute("VACUUM; ");

在插入和更新后运行上述操作会将文件压缩回 20MB。
我们可以使用 jooq 为 sqlite 启用自动清理吗？

EDIT - 使用事务和 VACUUM

try (DSLContext dsl = DSL.using(dbUrl)) {
    dsl.transaction((ctx) -> {
        dsl.createTable("TABLE1")
            .column("COL1", SQLiteDataType.CLOB)
            .column("COL2", SQLiteDataType.CLOB)
            .execute();

        dsl.insertInto(TABLE1)
            .columns(TABLE1.COL1, TABLE1.COL2)
            .values("ABB", null)
            .execute();

        Field<String> coalesce = DSL.coalesce(TABLE1.COL2, "");
        dsl.update(TABLE1).set(TABLE1.COL2, DSL.concat(coalesce, DSL.val(lines.toString())))
            .where(TABLE1.COL1.eq("ABB"))
            .execute();         

    });
    dsl.execute("VACUUM; ");
}