Android：java.lang.OutOfMemoryError：在使用 gson.toJson() 时无法分配空闲字节和 70MB，直到 OOM

Question

在我的 Android 应用程序中，当我尝试将我的数据同步到大型服务器时，我遇到了异常。我认为当数据大小超过 20 MB 时出现此异常。我将位图图像保存为字符串，使用 base64 编码在缩小图像大小后生成如此庞大的数据。

04-18 13:51:51.957  16199-16816/com.example.myproject.app E/art﹕ Throwing OutOfMemoryError "Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM"
04-18 13:51:52.037  16199-16816/com.example.myproject.app E/AndroidRuntime﹕ FATAL EXCEPTION: Thread-4482
Process: com.example.myproject.app, PID: 16199
java.lang.OutOfMemoryError: Failed to allocate a 128887990 byte allocation with 16777216 free bytes and 70MB until OOM
    at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:95)
    at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:146)
    at java.lang.StringBuffer.append(StringBuffer.java:219)
    at java.io.StringWriter.write(StringWriter.java:167)
    at com.google.gson.stream.JsonWriter.string(JsonWriter.java:570)
    at com.google.gson.stream.JsonWriter.value(JsonWriter.java:419)
    at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:426)
    at com.google.gson.internal.bind.TypeAdapters$16.write(TypeAdapters.java:410)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:112)
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:239)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:97)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:61)
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)

如何解决这个问题？我知道这是当我使用 Gson 将数据从类转换为 json 时出现的。以下是我的代码：

SimpleDateFormat dtf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss",Locale.ENGLISH);
GsonBuilder builder = new GsonBuilder();
builder.registerTypeAdapter(Date.class, new JsonDeserializer<Date>() {

        @Override
        public Date deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
            String frStr = json.getAsJsonPrimitive().getAsString();
            Date retDate =null;
            try {
                retDate = dtf.parse(frStr);
            } catch (ParseException e) {
                e.printStackTrace();
            }
            return retDate;
        }
    });
    builder.registerTypeAdapter(Date.class, new JsonSerializer<Date>() {
            @Override
            public JsonElement serialize(Date src, Type typeOfSrc, JsonSerializationContext context) {
                String jsDate = dtf.format(src);
                return new JsonPrimitive(jsDate);
            }
    });
    builder.registerTypeAdapter(byte[].class, new JsonDeserializer<byte[]>() {
            @Override
            public byte[] deserialize(JsonElement json, Type type, JsonDeserializationContext deserializationContext) throws JsonParseException {
                return Base64.decode(json.getAsString(), Base64.NO_WRAP);
            }
    });
    gson = builder.create();


    attDataAcc.setAttList(attList);
    String jsonAttAccts = gson.toJson(attDataAcc, AttachmentDataList.class);
        HttpEntity<String> entityAtt = new HttpEntity<String>(jsonAttAccts,headers);
        ResponseEntity<String> restResA = restTemplate.exchange(strUrl+"/saveAttToServer", HttpMethod.POST, entityAtt, String.class);

public class Attachment implements Serializable {

            @DatabaseField(columnName = "id",id = true)
            private String id;

            @DatabaseField(columnName = "user_id")
            private Integer userId;

            @DatabaseField(columnName = "attachment_id")
            private String attachmentId;

            @DatabaseField(columnName = "file_name")
            private String fileName;

            @DatabaseField(columnName = "file_data")
            private String fileData;

            @DatabaseField(columnName = "date",dataType=DataType.DATE)
            private Date date;

            public Attachment() {
                super();
                // TODO Auto-generated constructor stub
            }

            public Attachment(String id, Integer userId, String attachmentId, String fileName, String fileData, Date date) {
                this.id = id;
                this.userId = userId;
                this.attachmentId = attachmentId;
                this.fileName = fileName;
                this.fileData = fileData;
                this.date = date;
            }

            public String getId() {
                return id;
            }

            public void setId(String id) {
                this.id = id;
            }

            public Integer getUserId() {
                return userId;
            }

            public void setUserId(Integer userId) {
                this.userId = userId;
            }

            public String getAttachmentId() {
                return attachmentId;
            }

            public void setAttachmentId(String attachmentId) {
                this.attachmentId = attachmentId;
            }

            public String getFileName() {
                return fileName;
            }

            public void setFileName(String fileName) {
                this.fileName = fileName;
            }

            public String getFileData() {
                return fileData;
            }

            public void setFileData(String fileData) {
                this.fileData = fileData;
            }

            public Date getDate() {
                return date;
            }

            public void setDate(Date date) {
                this.date = date;
            }

            @Override
            public boolean equals(Object o) {
                if (this == o) return true;
                if (o == null || getClass() != o.getClass()) return false;

                Attachment that = (Attachment) o;

                if (id != null ? !id.equals(that.id) : that.id != null) return false;
                if (userId != null ? !userId.equals(that.userId) : that.userId != null) return false;
                if (attachmentId != null ? !attachmentId.equals(that.attachmentId) : that.attachmentId != null) return false;
                if (fileName != null ? !fileName.equals(that.fileName) : that.fileName != null) return false;
                if (fileData != null ? !fileData.equals(that.fileData) : that.fileData != null) return false;
                if (date != null ? !date.equals(that.date) : that.date != null) return false;

            }

            @Override
            public int hashCode() {
                int result = id != null ? id.hashCode() : 0;
                result = 31 * result + (userId != null ? userId.hashCode() : 0);
                result = 31 * result + (attachmentId != null ? attachmentId.hashCode() : 0);
                result = 31 * result + (fileName != null ? fileName.hashCode() : 0);
                result = 31 * result + (fileData != null ? fileData.hashCode() : 0);
                result = 31 * result + (date != null ? date.hashCode() : 0);
                return result;
            }

            @Override
            public String toString() {
                return userFileName;
            }

        }

        public class AttachmentDataList implements Serializable {
            private ArrayList<Attachment> attList;

            public ArrayList<Attachment> getAttList() {
                return attList;
            }

            public void setAttList(ArrayList<Attachment> attList) {
                this.attList = attList;
            }
        }

Answer 1

您遇到了OutOfMemoryError，因为您使用的是低效且非常消耗内存的 Base64 转换。这里的另一个亮点是 Gson：它没有为 JsonWriter 和 JsonReader 类提供任何原始写入方法：您在这里最多可以做的是写入/读取 single 字符串值。为单个字符串收集大量输入是另一个非常消耗内存的操作：检查您的堆栈跟踪以确保在底层使用了字符串构建器实例——这只是将单个值写入输出流。简而言之，它看起来像这样（如果我没记错你的代码，因为它似乎缺少真正重要的部分，所以我只是试图重建你的场景）：

• 获取一个字节数组（这将是一个新对象，可能是另一个字节数组的克隆）；
• 将字节数组转换为 Base64 编码字符串（这也会影响性能，因为会克隆字节数组以创建防御性副本）；
• 将 ALL 转换为字符串 gson.toJson(attDataAcc, AttachmentDataList.class); -- 又一个巨大的成功。

所有这些都非常消耗内存。如果 Gson 可以支持对输出流的原始写入，那就太好了，但目前它缺少任何一个。

理论上，您可以通过仅写入底层流（可能直接从您的字节数组源中写入而无需任何大量转换，因为 Base64 也可以流式传输从而消耗最少的内存）来克服这个问题。您提到了 Gson 2.6.2，但我正在使用 Gson 2.8.0，因此以下解决方案可以 100% 使用 仅使用 Gson 2.8.0，甚至可能不适用于任何其他次要 Gson版本，因为它使用反射来“破解”JsonWriter 类。

final class ByteArrayTypeAdapter
        extends TypeAdapter<byte[]> {

    // These two methods and one field from the super class privates are necessary to make it all work  
    private static final Method writeDeferredNameMethod;
    private static final Method beforeValueMethod;
    private static final Field writerField;

    static {
        try {
            writeDeferredNameMethod = JsonWriter.class.getDeclaredMethod("writeDeferredName");
            writeDeferredNameMethod.setAccessible(true);
            beforeValueMethod = JsonWriter.class.getDeclaredMethod("beforeValue");
            beforeValueMethod.setAccessible(true);
            writerField = JsonWriter.class.getDeclaredField("out");
            writerField.setAccessible(true);
        } catch ( final NoSuchMethodException | NoSuchFieldException ex ) {
            throw new RuntimeException(ex);
        }
    }

    // This type adapter is effectively a singleton having no any internal state
    private static final TypeAdapter<byte[]> byteArrayTypeAdapter = new ByteArrayTypeAdapter();

    private ByteArrayTypeAdapter() {
    }

    // But making the constructor private and providing access to the instance via the method, we make sure that the only instance exists and it's safe
    static TypeAdapter<byte[]> getByteArrayTypeAdapter() {
        return byteArrayTypeAdapter;
    }

    @Override
    public void write(final JsonWriter out, final byte[] bytes)
            throws IOException {
        try {
            // Since we're writing a byte[] array, that's probably a field value, make sure that the corresponding property name has been written to the output stream
            writeDeferredNameAndFlush(out);
            // Now simulate JsonWriter.value(byte[]) if such a method could exist
            writeRawBase64ValueAndFlush(bytes, (Writer) writerField.get(out));
        } catch ( IllegalAccessException | InvocationTargetException ex ) {
            throw new IOException(ex);
        }
    }

    @Override
    public byte[] read(final JsonReader in) {
        // If necessary, requires more hacks...
        // And this is crucial for the server-side:
        // In theory, the client can generate HUGE Base64 strings,
        // So the server could crash with OutOfMemoryError too
        throw new UnsupportedOperationException();
    }

    private static void writeDeferredNameAndFlush(final Flushable out)
            throws IOException, IllegalAccessException, InvocationTargetException {
        writeDeferredNameMethod.invoke(out);
        beforeValueMethod.invoke(out);
        // Flush is necessary: the JsonWriter does not know that we're using its private field intruding to its privates and may not flush
        out.flush();
    }

    private static void writeRawBase64ValueAndFlush(final byte[] bytes, final Writer writer)
            throws IOException {
        // Writing leading "
        writer.write('\"');
        // This comes from Google Guava
        final BaseEncoding baseEncoding = BaseEncoding.base64();
        final OutputStream outputStream = baseEncoding.encodingStream(writer);
        // This too
        // Note that we just r_e_d_i_r_e_c_t streams on fly not making heavy transformations
        ByteStreams.copy(new ByteArrayInputStream(bytes), outputStream);
        // This is necessary too
        outputStream.close();
        // Writing trailing "
        writer.write('\"');
        // Flush again to keep it all in sync
        writer.flush();
    }

}

我知道这是一个 hack，但总比不断地获取OutOfMemoryError 要好。

现在，让它与 Spring RestTemplates 一起工作：

// Gson is thread-safe and can be re-used
private static final Gson gson = new GsonBuilder()
        // SimpleDateFormat may be NOT thread-safe so you should not share the single SimpleDateFormat between threads
        // However Gson supports date/time formats out of box
        .setDateFormat("yyyy-MM-dd HH:mm:ss")
        // Registering byte[] to the type adapter
        .registerTypeAdapter(byte[].class, getByteArrayTypeAdapter())
        .create();

private static final RestTemplate restTemplate = new RestTemplate();
private static final String URL = "http://localhost";

public static void main(final String... args) {
    sendPostRequest("hello world".getBytes(), byte[].class);
}

private static void sendPostRequest(final Object object, final Type type) {
    // This is where we're binding the output stream I was asking in the question comments
    final RequestCallback requestCallback = request -> gson.toJson(object, type, new OutputStreamWriter(request.getBody()));
    // Spring RestTemplates stuff here...
    final SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    final ResponseExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(String.class, restTemplate.getMessageConverters());
    restTemplate.setRequestFactory(requestFactory);
    // Let it fly
    restTemplate.execute(URL, POST, requestCallback, responseExtractor);
}

请注意，您可能会为可以直接写入输出流的特殊类型编写专门的类型适配器，因此您根本无法摆脱byte[]。您也可以在官方 Gson 问题跟踪器上为这个问题投票：https://github.com/google/gson/issues/971，并且在 Gson 的未来版本中可能不需要使用任何 Java Reflection API hack。