避免 OutOfMemory 问题的编写大 JSON 文件的最佳方式答案

【问题标题】：Best way of writing Big JSON files avoiding OutOfMemory issues避免 OutOfMemory 问题的编写大 JSON 文件的最佳方式
【发布时间】：2014-02-28 11:16:01
【问题描述】：

首先，请注意今天是我与GSON 的第一天。我正在尝试使用 GSON 库编写 Json 文件。我在ArrayList 中有数千个JsonObjects。写入 Json 文件时，它应该看起来像这样。

[
    {
        "hash_index": "00102x05h06l0aj0dw",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 0,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    },
    {
        "hash_index": "00102x05h06l0aj0dw0iz0kn0l@0t#0",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 1,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    }
]

现在，我使用以下代码编写 JSOn。

 private void writeNewJsonFile() throws IOException
    {
        System.out.println("Starting to write the JSON File");
        //Add everything into a JSONArray
        JsonArray jsonArrayNew = new JsonArray();

        for(int i=0;i<jsonObjectHolder.size();i++)
        {
            System.out.println("inside array");
            jsonArrayNew.add(jsonObjectHolder.get(i));
        }


        //Write it to the File
    /*  File file= new File("items_Articles_4_1.json");

        FileWriter fw = new FileWriter(file);;
        fw.write(jsonArrayNew.toString());
        fw.flush();
        fw.close();*/

        System.out.println("outside array");

        ByteArrayInputStream input = new ByteArrayInputStream(jsonArrayNew.toString().getBytes());

        Long contentLength = Long.valueOf(jsonArrayNew.toString().getBytes().length);

        ObjectMetadata metaData = new ObjectMetadata();
        metaData.setContentLength(contentLength);

        s3.putObject(outputBucket,outputFile,input,metaData);


    }

在这里，我将JsonArray 转换为String 并进行写作。我担心这会很快与 Big Json 数组崩溃并给我OutOfMemoryException。就像我使用 GSON 部分读取 Json 文件一样，有什么方法可以逐个编写 Json 文件或其他东西，可以避免OutOfMemoryException 问题？

【问题讨论】：

为对象编写对象并自己添加数组荣誉是一种选择。
@Gimby：我不明白。

标签： java json amazon-s3 out-of-memory gson

【解决方案1】：

我正在使用下一个代码：

WriteJsonArrayByParts<Cache> write = new WriteJsonArrayByParts<Cache>(fileNameTest, " ");
write.writeStart();
for(Cache cache : listOfObjects()) {
    write.writeObject(cache, Cache.class);
}
write.writeEnd();
write.close();

...

public static class WriteJsonArrayByParts<T> {
    Gson gson = new Gson();
    JsonWriter writer;

    public WriteJsonArrayByParts(String fileNameWithPath, String indent) throws Exception {
        OutputStream os = new FileOutputStream(fileNameWithPath, false);
        BufferedOutputStream osb = new BufferedOutputStream(os, 8 * 1024);

        writer = new JsonWriter(new OutputStreamWriter(osb, StringUtil.UTF_8));
        writer.setIndent(indent);
    }

    public void writeStart() throws IOException {
        writer.beginArray();
    }

    @SuppressWarnings("unchecked")
    public void writeObject(T t, Class<?> resultClass) throws IOException {
        ((TypeAdapter<Object>) gson.getAdapter(resultClass)).write(writer, t);
    }

    public void writeEnd() throws IOException {
        writer.endArray();
    }

    public void close() throws IOException {
        writer.close();
    }
}

【讨论】：