protobuf-net 集合序列化抛出 StackOverflowException

Question

我有一个使用 protobuf-net 序列化的嵌入式 C# 应用程序。 当序列化大约 50 个条目的集合时，它会引发 StackOverflowException，只能在运行 WinCE 的设备上重现。 堆栈大约有 600 个条目，并以如下方式结束：

at ProtoBuf.Meta.RuntimeTypeModel.GetKey(Type type, Boolean demand, Boolean getBaseKey)
   at ProtoBuf.Meta.ValueMember.TryGetCoreSerializer(RuntimeTypeModel model, DataFormat dataFormat, Type type, WireType& defaultWireType, Boolean asReference, Boolean dynamicType, Boolean overwriteList, Boolean allowComplexTypes)
   at ProtoBuf.Meta.ValueMember.BuildSerializer()
   at ProtoBuf.Meta.ValueMember.get_Serializer()
   at ProtoBuf.Meta.MetaType.BuildSerializer()
   at ProtoBuf.Meta.MetaType.get_Serializer()
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.FieldDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.Serializers.SubItemSerializer.ProtoBuf.Serializers.IProtoSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TagDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.ListDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.PropertyDecorator.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Serializers.TypeSerializer.Write(Object value, ProtoWriter dest)
   at ProtoBuf.Meta.RuntimeTypeModel.Serialize(Int32 key, Object value, ProtoWriter dest)
   at ProtoBuf.ProtoWriter.WriteObject(Object value, Int32 key, ProtoWriter writer)
   at ProtoBuf.BclHelpers.WriteNetObject(Object value, ProtoWriter dest, Int32 key, NetObjectOptions options)
   at ProtoBuf.Serializers.NetObjectSerializer.Write(Object value, ProtoWriter dest)

列表定义如下：

[ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        //[ProtoMember(2, AsReference = true)]
        public NodeList<T> Nodes { get; private set; }

（我尝试了两个版本，有和没有 DataFormat.Group）

[ProtoContract(ImplicitFields = ImplicitFields.AllFields)]        
public class NodeList<T> : Collection<GraphNode<T>> where T : new()

在具有更多资源的 Windows 7 上运行应用程序时不会引发异常。

有没有可能优化序列化？ 我是不是做错了什么？

为什么调用栈这么长？我认为一个列表应该被视为一个数组，而不是递归调用？

谢谢， 戴安娜

Answer 1

您的问题中没有包含GraphNode<T> 的示例。但是，从名称上看，它似乎是一个通用的图形节点元素，类似于 Microsoft 技术文章 An Extensive Examination of Data Structures Using C# 2.0: Part 5: From Trees to Graphs 中的 GraphNode<T>、NodeList<T> 和 Graph<T>。

在这种情况下，如果您的节点图非常深并且根节点位于集合的开头，那么很有可能超过序列化节点列表的最大堆栈深度。原因在Protobuf-net: the unofficial manual中有解释：

引用的工作原理

当序列化一个应用AsReference或AsReferenceDefault的类Foo时，protocol buffer中的字段类型由Foo变为bcl.NetObjectProxy，在protobuf-net的源码中定义如下（Tools/bcl.原型）：

message NetObjectProxy {
  // for a tracked object, the key of the **first** 
  // time this object was seen
  optional int32 existingObjectKey = 1; 
  // for a tracked object, a **new** key, the first 
  // time this object is seen
  optional int32 newObjectKey = 2;
  // for dynamic typing, the key of the **first** time 
  // this type was seen
  optional int32 existingTypeKey = 3; 
  // for dynamic typing, a **new** key, the first time 
  // this type is seen
  optional int32 newTypeKey = 4;
  // for dynamic typing, the name of the type (only 
  // present along with newTypeKey) 
  optional string typeName = 8;
  // the new string/value (only present along with 
  // newObjectKey)
  optional bytes payload = 10;
}

看来

• 第一次遇到对象，写入 newObjectKey 和有效载荷字段；据推测，有效载荷被存储为 它的类型是 Foo。
• 再次遇到对象时，只写入existingObjectKey。

因此，如果图很深，并且遇到了根作为列表中的第一个元素，protobuf-net 将深度优先递归遍历图，而不是遍历数组广度优先，从而溢出堆栈。

为了避免这个问题，您可以利用这样一个事实，即当AsReference = true 时，protobuf-net 仅在第一次遇到对象时迭代对象的成员，以分两个阶段序列化您的NodeList<T>：

1. 序列化节点列表没有他们的邻居信息。
2. 然后序列化一个邻居信息表。

例如，使用上面文章中的GraphNode<T>、NodeList<T>和Graph<T>的简化版本作为基础，可以如下实现：

[ProtoContract]
public class GraphNode<T> where T : new()
{
    readonly NodeList<T> neighbors = new NodeList<T>();

    public GraphNode()
    {
        this.Value = new T();
    }

    public GraphNode(T value)
        : this()
    {
        this.Value = value;
    }

    [ProtoMember(1, AsReference = true)]
    public T Value { get; set; }

    // Do not serialize the list of neighbors directly!  
    // Instead this will be serialized by the NodeList<T> owned by the Graph<T>
    [ProtoIgnore]
    public NodeList<T> Neighbors { get { return neighbors; } }
}

[ProtoContract(IgnoreListHandling = true)]
public class NodeList<T> : Collection<GraphNode<T>> where T : new()
{
    [ProtoContract]
    class GraphNodeNeighborsProxy
    {
        [ProtoMember(1, AsReference = true)]
        public GraphNode<T> Node { get; set; }

        [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
        public ICollection<GraphNode<T>> Neighbors
        {
            get
            {
                return Node == null ? null : Node.Neighbors;
            }
        }
    }

    [ProtoMember(1, AsReference = true, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNode<T>> Nodes
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNode<T>>(this, n => n, (c, n) => c.Add(n)); 
        }
    }

    [ProtoMember(2, DataFormat = DataFormat.Group)]
    IEnumerable<GraphNodeNeighborsProxy> NeighborsTable
    {
        get
        {
            return new SerializationCollectionWrapper<GraphNode<T>, GraphNodeNeighborsProxy>(
                this,
                n => new GraphNodeNeighborsProxy { Node = n },
                (c, proxy) => {}
                );
        }
    }
}

[ProtoContract]
public class Graph<T> where T : new()
{
    readonly private NodeList<T> nodeSet = new NodeList<T>();

    public Graph() { }

    public GraphNode<T> AddNode(GraphNode<T> node)
    {
        // adds a node to the graph
        nodeSet.Add(node);
        return node;
    }

    public GraphNode<T> AddNode(T value)
    {
        // adds a node to the graph
        return AddNode(new GraphNode<T>(value));
    }

    public void AddDirectedEdge(GraphNode<T> from, GraphNode<T> to)
    {
        from.Neighbors.Add(to);
    }

    [ProtoMember(2, AsReference = true, DataFormat = DataFormat.Group)]
    public NodeList<T> Nodes
    {
        get
        {
            return nodeSet;
        }
    }
}

public class SerializationCollectionWrapper<TFrom, TTo> : ICollection<TTo>
{
    readonly ICollection<TFrom> collection;
    readonly Func<TFrom, TTo> mapTo;
    readonly Action<ICollection<TFrom>, TTo> add;

    public SerializationCollectionWrapper(ICollection<TFrom> collection, Func<TFrom, TTo> mapTo, Action<ICollection<TFrom>, TTo> add)
    {
        if (collection == null || mapTo == null || add == null)
            throw new ArgumentNullException();
        this.collection = collection;
        this.mapTo = mapTo;
        this.add = add;
    }

    ICollection<TFrom> Collection { get { return collection; } }

    #region ICollection<TTo> Members

    public void CopyTo(TTo[] array, int arrayIndex)
    {
        foreach (var item in this)
            array[arrayIndex++] = item;
    }

    public int Count
    {
        get { return Collection.Count; }
    }

    public bool IsReadOnly
    {
        get { return Collection.IsReadOnly; }
    }

    public void Add(TTo item)
    {
        add(Collection, item);
    }

    public void Clear()
    {
        throw new NotImplementedException();
    }

    public bool Contains(TTo item)
    {
        throw new NotImplementedException();
    }

    public bool Remove(TTo item)
    {
        throw new NotImplementedException();
    }

    #endregion

    #region IEnumerable<TTo> Members

    public IEnumerator<TTo> GetEnumerator()
    {
        foreach (var item in Collection)
            yield return mapTo(item);
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

请注意，这仅在根节点列表包含图中的所有节点时才有效。如果您没有图表中所有节点的表，则需要计算图表的transitive closure 以预先序列化所有节点。