JsonConvert 反序列化对象内存不足异常

Question

我正在尝试反序列化通过 http Web 响应接收到的对象，并且当对象中的 byte[] 较小时它工作正常。但是当大小增加时，JsonConvert 反序列化会抛出内存异常。输入是 http web 响应而不是文件。这是在反序列化一个太大的对象。

var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    using (var streamReader = new StreamReader(response.GetResponseStream()))
    {
        return JsonConvert.DeserializeObject<SomeObjectReply<TResponse>>(streamReader.ReadToEnd());
    }
}

我尝试了以下方法，但仍然是同样的问题

if (response.StatusCode == HttpStatusCode.OK)
{
    using (var streamReader = new StreamReader(response.GetResponseStream()))
    using (var jsonReader = new JsonTextReader(streamReader))
    {
        JsonSerializer serializer = new JsonSerializer();
        return (SomeObjectReply<TResponse>)serializer.Deserialize(jsonReader, typeof(SomeObjectReply<TResponse>));
    }
}   

JSON 看起来像：

{
    "Id":"81a130d2-502f-4cf1-a376-63edeb000e9f",
    // The following is MUCH larger in cases where the exception is thrown.
    "Document":"AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8gISIjJCUmJygpKissLS4vMDEyMzQ1Njc4OTo7PD0+P0BBQkNERUZHSElKS0xNTk9QUVJTVFVWV1hZWltcXV5fYGFiY2RlZmdoaWprbG1ub3BxcnN0dXZ3eHl6e3x9fn+AgYKDhIWGh4iJiouMjY6PkJGSk5SVlpeYmZqbnJ2en6ChoqOkpaanqKmqq6ytrq+wsbKztLW2t7i5uru8vb6/wMHCw8TFxsfIycrLzM3Oz9DR0tPU1dbX2Nna29zd3t/g4eLj5OXm5+jp6uvs7e7v8PHy8/T19vf4+fr7/P3+/wABAgMEBQYHCAkKCwwNDg8QERITFBUWFxgZGhscHR4fICEiIyQlJicoKSorLC0uLzAxMjM0NTY3ODk6Ozw9Pj9AQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVpbXF1eX2BhYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ent8fX5/gIGCg4SFhoeIiYqLjI2Oj5CRkpOUlZaXmJmam5ydnp+goaKjpKWmp6ipqqusra6vsLGys7S1tre4ubq7vL2+v8DBwsPExcbHyMnKy8zNzs/Q0dLT1NXW19jZ2tvc3d7f4OHi4+Tl5ufo6err7O3u7/Dx8vP09fb3+Pn6+/z9/v8="
}

反序列化后，对象将

public class Doc
{
    public Guid Id { get; set; }
    public byte[] Document { get; set; }
}

即我正在反序列化具有大字节数组的单个对象。 当Document 字节数组属性变得太大时，我得到了异常：

Message: Exception of type 'System.OutOfMemoryException' was thrown.
Stack Trace:
   at Newtonsoft.Json.Utilities.BufferUtils.RentBuffer(IArrayPool`1 bufferPool, Int32 minSize)
   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)
   at Newtonsoft.Json.JsonTextReader.ReadAsBytes()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)

Answer 1

您的回溯表明 Json.NET 在尝试实现 单个属性（即对应于单个 Base64 字符串的 byte [] 属性）时内存不足。如果您的 JSON 包含的字符串值无法在内存不足的情况下实现为 .Net 类型，那么您不能使用 Json.NET 解析您的 JSON，因为 JsonTextReader always fully materializes each property - 即使那些否则会跳过属性。

作为替代方案，您可以考虑使用JsonReaderWriterFactory.CreateJsonReader() 返回的阅读器来手动解析您的 JSON。这个工厂返回一个XmlDictionaryReader，它可以从JSON动态转码为XML，因此支持通过XmlReader.ReadContentAsBase64(Byte[], Int32, Int32)增量读取Base64属性。使用这种方法，您可以手动将Document 属性复制到Stream 中，从而允许大量数据，例如FileStream 或RecyclableMemoryStream。

首先，定义如下数据模型：

public class StreamedDocument : IDisposable
{
    public Guid Id { get; set; }
    public Stream Document { get; init; }

    public void Dispose() => Document?.Dispose();
}

然后定义以下工厂用于创建上面的数据模型：

public static class DocumentFactory
{
    const int BufferSize = 8192;
    private static readonly Microsoft.IO.RecyclableMemoryStreamManager manager = new ();

    public static Stream CreateTemporaryStream() => 
        // Create some temporary stream to hold the document.  
        // Could be a FileStream created with FileOptions.DeleteOnClose or a Microsoft.IO.RecyclableMemoryStream
        //File.Create(Path.GetTempFileName(), BufferSize, FileOptions.DeleteOnClose);
        manager.GetStream();
    
    public static StreamedDocument CreateStreamedDocument(Stream inputStream) =>
        DocumentFactory.PopulateStreamedDocument(new StreamedDocument{ Document = CreateTemporaryStream() }, inputStream);
    
    public static StreamedDocument PopulateStreamedDocument(StreamedDocument doc, Stream inputStream)
    {
        if (doc == null)
            throw new ArgumentNullException();
        if (doc.Document == null)
            throw new ArgumentException("null doc.Document");
        using (var reader = JsonReaderWriterFactory.CreateJsonReader(inputStream, XmlDictionaryReaderQuotas.Max))
        {
            while (!reader.EOF)
            {
                if (reader.NodeType == XmlNodeType.Element && reader.LocalName == nameof(doc.Id))
                {
                    doc.Id = reader.ReadElementContentAsGuid();
                    // reader should now be positioned PAST the EndElement.
                    Debug.Assert(reader.NodeType != XmlNodeType.EndElement, $"reader.NodeType {reader.NodeType} != XmlNodeType.EndElement");
                }
                else if (reader.NodeType == XmlNodeType.Element && reader.LocalName == nameof(doc.Document))
                {
                    byte[] buffer = new byte[BufferSize];
                    int readBytes = 0;
                    while ((readBytes = reader.ReadElementContentAsBase64(buffer, 0, buffer.Length)) > 0)
                        doc.Document.Write(buffer, 0, readBytes);
                    // reader should now be positioned ON the EndElement
                    Debug.Assert(reader.NodeType == XmlNodeType.EndElement, "reader.NodeType == XmlNodeType.EndElement");
                    if (doc.Document.CanSeek)
                        doc.Document.Position = 0;
                }
                else
                {
                    reader.Read();
                }
            }
        }
        return doc;
    }
}

现在你应该可以做到了：

var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    return DocumentFactory.CreateStreamedDocument(response.GetResponseStream());
}               

注意事项：

• 您必须最终处置 StreamedDocument 才能删除临时的 Stream Document。

• 有关JsonReaderWriterFactory 返回的阅读器如何从 JSON 转码为 XML 的讨论，请参阅 Efficiently replacing properties of a large JSON using System.Text.Json 和 Mapping Between JSON and XML。

• JsonReaderWriterFactory 被DataContractJsonSerializer 用于从 JSON 序列化数据协定类和向 JSON 序列化数据协定类。虽然不建议用于新开发（Microsoft 建议改用 system.text.json），但此序列化程序及其读取器已完全移植到 .NET Core / .NET 5，并且可以在情况合理时使用，就像他们在这里所做的那样。

• 您还可以检查System.Text.Json 中的JsonSerializer 是否能够反序列化您的 JSON 而不会耗尽内存：

  var doc = await JsonSerializer.DeserializeAsync<Doc>(responseStream);
  

  请注意System.Text.Json 仅支持async deserialization from streams。

  演示小提琴here.

• RecyclableMemoryStream 来自 Microsoft.IO.RecyclableMemoryStream nuget 包。它是一个池化的 MemoryStream 分配器，用于减少 GC 负载并提高高度可扩展系统的性能。 除了其他优点之外，它还可以替代 MemoryStream，它不会尝试将流内容分配为单个巨大的数组。

演示小提琴here.

Answer 2

在处理大型对象时，内存中的序列化/反序列化确实会产生这些类型的问题。作为我自己的一种解决方法，我总是使用文件系统作为中间存储。

将您的 json 保存到临时文件，然后使用以下通用方法对其进行反序列化：

public static class Helper
{
    public static T CreateObjectFromJsonFile<T>(string filePath) where T : class
    {
        T obj;
        using (var file = File.OpenText(filePath))
        {
            var serializer = new JsonSerializer { NullValueHandling = NullValueHandling.Ignore };
            obj = (T)serializer.Deserialize(file, typeof(T));
        }

        return obj;
    }

    public static void CreateJsonFileFromObject(object input, string fileName)
    {
        using (var file = File.CreateText(fileName))
        {
            var serializer = new JsonSerializer { NullValueHandling = NullValueHandling.Ignore };
            serializer.Serialize(file, input);
        }
    }
}

这是一个完整的测试示例：

//create object
var list = new List<TvShow>
{
    new TvShow { Id = 0, Rating = 10, Title = "t1"},
    new TvShow { Id = 1, Rating = 20, Title = "t2"},
    new TvShow { Id = 2, Rating = 30, Title = "t3"}
};

//serialize to JSON and save to file system
const string filePath = "C://temp//test.txt";
Helper.CreateJsonFileFromObject(list, filePath);

//deserialize back to object from file system
var result = Helper.CreateObjectFromJsonFile<List<TvShow>>(filePath);

使用您的特定代码：

const string filePath = "C://temp//test.txt";

//save JSON to file
var response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    using (var fileStream = new FileStream(filePath, FileMode.Create, FileAccess.Write))
    {
        response.Content.ReadAsStream().CopyTo(fileStream);
    }
}

//deserialize object from file
var result = Helper.CreateObjectFromJsonFile<SomeObjectReply<TResponse>>(filePath);