Cosmos DB ReadNextAsync 引发 JSON 序列化错误答案

【问题标题】：Cosmos DB ReadNextAsync throwing JSON Serialization errorCosmos DB ReadNextAsync 引发 JSON 序列化错误
【发布时间】：2021-08-09 19:52:18
【问题描述】：

我正在使用 Microsoft.Azure.Cosmos 3.20.1。我正在创建一个具有托管标识的 cosmos 客户端，

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Azure.Identity;
using Business.CosmosDB;
using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.Linq;
using Newtonsoft.Json;

namespace ConsoleApp2
{
    public class FlattenedProduct
    {
        [JsonProperty(PropertyName = "id")]
        public string id
        {
            get;
            set;
        }
        public string userType_language { get; set; } // partition Key
        public string Name { get; set; }
        public string Description { get; set; }
        public bool IsEnabled { get; set; }
        public bool IsFeatured { get; set; }

        [JsonProperty(PropertyName = "PartitionKey")]
        public string PartitionKey
        {
            get
            {
                return userType_language;
            }
        }
    }

    public class GridResults<T>
    {
        public int Total { get; set; }
        public List<T> Data { get; set; } = new List<T>();

        public long QueryDuration { get; set; }
    }
    class Program
    {
        static async Task Main(string[] args)
        {
            string endpoint = "https://xxx.documents.azure.com:443/";
            CosmosClient client = new CosmosClient(endpoint, new DefaultAzureCredential(), GetCosmosClientOptionsClientOptions());
            string databaseId = "xxx";
            var database = client.GetDatabase(databaseId);
            string collectionId = "FlattenedProduct";
            var container = database.GetContainer(collectionId);
            
            GridResults<FlattenedProduct> results = new GridResults<FlattenedProduct>();

            var requestOptions = new QueryRequestOptions();
            using (var setIterator = container.GetItemLinqQueryable<FlattenedProduct>(requestOptions: requestOptions).Where( x => x.IsEnabled).ToFeedIterator())
            {
                Microsoft.Azure.Cosmos.FeedResponse<FlattenedProduct> items = null;
                while (setIterator.HasMoreResults)
                {
                    try
                    {
                        items = await setIterator.ReadNextAsync();
                        foreach (var item in items)
                        {
                            results.Data.Add(item);
                        }

                        items = null;
                    }
                    catch (Exception e)
                    {
                    }
                }
            }
        }

        protected static CosmosClientOptions GetCosmosClientOptionsClientOptions(int maxRetryAttemptsOnThrottledRequests = 3, int maxRetryWaitTimeInSeconds = 180)
        {
            return new CosmosClientOptions()
            {
                ConnectionMode = Microsoft.Azure.Cosmos.ConnectionMode.Direct,
                RequestTimeout = new TimeSpan(0, 1, 0),
                MaxTcpConnectionsPerEndpoint = 1000, 
                MaxRetryAttemptsOnRateLimitedRequests = maxRetryAttemptsOnThrottledRequests,
                MaxRetryWaitTimeOnRateLimitedRequests = TimeSpan.FromSeconds(maxRetryWaitTimeInSeconds),
                AllowBulkExecution = true
            };
        }
    }
}

我在 ReadNextAsync 上收到以下异常：

{"A different value already has the Id '1'."}
{"Error reading object reference '1'. Path '[37].userType_language (this is the partition key)', line 1, position 53422."}

我检查了所有记录的 id = 1。

完整的堆栈跟踪

   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.AddReference(JsonReader reader, String id, Object value)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Microsoft.Azure.Cosmos.CosmosJsonDotNetSerializer.FromStream[T](Stream stream)
   at Microsoft.Azure.Cosmos.CosmosJsonSerializerWrapper.FromStream[T](Stream stream)
   at Microsoft.Azure.Cosmos.CosmosSerializerCore.FromFeedStream[T](Stream stream)
   at Microsoft.Azure.Cosmos.Serializer.CosmosElementSerializer.GetResourcesHelper[T](IReadOnlyList`1 cosmosElements, CosmosSerializerCore serializerCore, CosmosSerializationFormatOptions cosmosSerializationOptions)
   at Microsoft.Azure.Cosmos.Serializer.CosmosElementSerializer.GetResources[T](IReadOnlyList`1 cosmosArray, CosmosSerializerCore serializerCore)
   at Microsoft.Azure.Cosmos.QueryResponse`1..ctor(HttpStatusCode httpStatusCode, IReadOnlyList`1 cosmosElements, CosmosQueryResponseMessageHeaders responseMessageHeaders, CosmosDiagnostics diagnostics, CosmosSerializerCore serializerCore, CosmosSerializationFormatOptions serializationOptions)
   at Microsoft.Azure.Cosmos.QueryResponse`1.CreateResponse[TInput](QueryResponse cosmosQueryResponse, CosmosSerializerCore serializerCore)
   at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateQueryFeedResponseHelper[T](ResponseMessage cosmosResponseMessage)
   at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateQueryFeedUserTypeResponse[T](ResponseMessage responseMessage)
   at Microsoft.Azure.Cosmos.FeedIteratorCore`1.<ReadNextAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.ClientContextCore.<RunWithDiagnosticsHelperAsync>d__38`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.ClientContextCore.<OperationHelperWithRootTraceAsync>d__29`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at ConsoleApp2.Program.<Main>d__0.MoveNext() in C:\Users\...\ConsoleApp2\ConsoleApp2\Program.cs:line 65

唯一需要注意的是，它在抛出异常之前运行了很长时间（> 1 分钟），所以我们可能需要一个大集合。

【问题讨论】：

你能分享错误的完整堆栈跟踪吗？ dbFilter 的价值是什么？
错误中提到的id = 1不是数据库ID，是Json.NET人为添加的$id属性，用于引用跟踪。见：Preserving Object References。您的 JSON 是否包含具有多个 "$id" : "1" 属性的对象？
我在代码或数据文档中都没有看到任何“$id”。这是运行时属性吗？
是的，它是在序列化为 JSON 期间由 Json.NET 本身添加的，并在反序列化期间由 Json.NET 在内部使用，当PreserveReferencesHandling（或[DataContract(IsReference=true)]）设置在某处时。你永远不会在你的模型上看到它，它只是在 JSON 字符串本身。
我确实在代码中看到了这一点：JsonConvert.DefaultSettings = () => new JsonSerializerSettings { ReferenceLoopHandling = ReferenceLoopHandling.Ignore, PreserveReferencesHandling = PreserveReferencesHandling.Objects };但是，也许其他一些代码可能正在使用它。

标签： json serialization json.net azure-cosmosdb

【解决方案1】：

数据中有一些我以前没有见过的 $id。我通过将元数据属性处理设置为忽略来修复它：

new CosmosClientOptions()
{
    Serializer = new CosmosJsonNetSerializer(new JsonSerializerSettings()
    {
        MetadataPropertyHandling = MetadataPropertyHandling.Ignore
    })
}

CosmosJsonNetSerializer 来源可以在这里找到： https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs

【讨论】：