对一列求和但不断超时 LINQ答案

【问题标题】：Doing a sum of a column but keep getting timeout LINQ对一列求和但不断超时 LINQ
【发布时间】：2021-05-10 12:32:43
【问题描述】：

来自 postgresql：

select client_id, sum(net_amount) from purchases
left join accounts on accounts.id= contracts.account_id
where purchase_date <= '2020-05-08'
and purchase_date > '2019-05-08'
group by client_id

这段代码只用了大约 10 秒的时间来执行，但是下面的 C# 代码失败了：

来自 C#/.Net：

var clientPurchases = _context.Purchases
            .Where(m => accountIds.Contains(m.AccountId) &&
                        m.Date >= resultingStartDate &&
                        m.Date <= resultingEndDate)
            .ToList();
var sum = clientPurchases.Sum(m=>m.NetAmount);

这是抛出的消息：

Unhandled Exception: System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. ---> Npgsql.NpgsqlException: Exception while reading from stream ---> System.IO.IOExcept
ion: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Socke
ts.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   --- End of inner exception stack trace ---
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass31_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlConnector.<>c__DisplayClass161_0.<<ReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Npgsql.NpgsqlDataReader.<NextResult>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.<ExecuteDbDataReader>d__100.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Common.DbCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.<>c.<Reader>b__6_0(DbCommand t, DbCommandInterceptionContext`1 c)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Internal.InterceptableDbCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Common.DbCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__1()
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass41_0.<GetResults>b__0()
   at System.Data.Entity.Infrastructure.DefaultExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__31_0()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.<>c__14`1.<GetElementFunction>b__14_3(IEnumerable`1 sequence)
   at System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.ExecuteSingle[TResult](IEnumerable`1 query, Expression queryRoot)
   at System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.System.Linq.IQueryProvider.Execute[TResult](Expression expression)
   at System.Data.Entity.Internal.Linq.DbQueryProvider.Execute[TResult](Expression expression)
   at System.Linq.Queryable.Sum[TSource](IQueryable`1 source, Expression`1 selector)

我正在尝试对大约 10000 个对象的净数量进行总和。有没有其他替代方案可以做到这一点？

【问题讨论】：

默认超时时间为 30 秒。增加超时时间：docs.microsoft.com/en-us/dotnet/api/…
@jdweng 除了增加超时限制，还有其他选择吗？我的意思是在 C# 代码级别。
您的 c# 版本正在加载购买的所有列，其中 sql 版本只有客户端 ID 和总和 - 您可以更改 c# 以在数据库端计算总和，这应该会提高查询执行时间。
尝试对数据库进行碎片整理。时间取决于数据库的大小、服务器中的内存量、服务器的速度和返回的数据量。

标签： c# .net linq timeout

【解决方案1】：

尝试在第一个查询中删除ToList

var clientPurchases = _context.Purchases
            .Where(m => accountIds.Contains(m.AccountId) &&
                        m.Date >= resultingStartDate &&
                        m.Date <= resultingEndDate);
var sum = clientPurchases.Sum(m=>m.NetAmount);

解释：在where 子句之后链接ToList() 之前，sql server 上没有执行任何查询。通过添加 Sum 而不添加 ToList，您将在 sql server 上计算总和，而无需等待 C# 内存空间上所有符合条件的购买，然后在 IEnumerable 上计算总和。

在deferred linq to database execution上阅读更多信息

【讨论】：

var sum = _context.Purchases.Where(m => accountIds.Contains(m.AccountId) && m.Date >= resultingStartDate && m.Date <= resultingEndDate).Sum(m=>m.NetAmount);我刚试过，效果不好。
accountids 集合中有多少个 id？
集合中大约 50 个
尝试在没有左连接的情况下执行原始查询。执行计划似乎首先通过了命运过滤器
如果时间过长，可以试试linq的left join方案。

【解决方案2】：

您的 linq 代码生成的 SQL 与您的 SQL 代码不同。您可以使用此 linq 生成正确的 SQL 查询。

var result = (from purchase in dbContext.Purchases
               from contract in dbContext.Contracts
               join account in dbContext.Accounts on contract.AccountId equals account.Id 
               into temp
               from account in temp.DefaultIfEmpty()
               where purchase.Date <= DateTime.Parse("2020-05-08")
                  && purchase.Date > DateTime.Parse("2019-05-08")
               group purchase by purchase.ClientId
               into g
               select new
                      {
                              ClientId = g.Key,
                              TotalNetAmount = g.Sum(x => x.NetAmount)
                      }).ToList();

Linq 方法链版本

var result = dbContext.Purchases
                       .SelectMany(purchase => dbContext.Contracts,
                                   (purchase, contract) => new
                                                           {
                                                                   purchase,
                                                                   contract
                                                           })
                       .GroupJoin(dbContext.Accounts, pc => pc.contract.AccountId, account => account.Id,
                                  (pc, temp) => new
                                                {
                                                        pc,
                                                        temp
                                                })
                       .SelectMany(pcTemp => pcTemp.temp.DefaultIfEmpty(),
                                   (pcTemp, account) => new
                                                         {
                                                                 pcTemp,
                                                                 account
                                                         })
                       .Where(pcTempAcc => pcTempAcc.pcTemp.pc.purchase.Date <= DateTime.Parse("2020-05-08") &&
                                           pcTempAcc.pcTemp.pc.purchase.Date > DateTime.Parse("2019-05-08"))
                       .GroupBy(pcTempAcc => pcTempAcc.pcTemp.pc.purchase.ClientId, pcTempAcc => pcTempAcc.pcTemp.pc.purchase)
                       .Select(g => new
                                    {
                                            ClientId = g.Key,
                                            TotalNetAmount = g.Sum(x => x.NetAmount)
                                    }).ToList();

这种方法生成这样的 SQL

SELECT p.client_id AS "ClientId", COALESCE(SUM(p.net_amount), 0.0) AS "TotalNetAmount"
FROM "Purchase" AS p
CROSS JOIN "Contract" AS c
LEFT JOIN "Account" AS a ON c.account_id = a."Id"
WHERE (p.purchase_date <= DATE '2020-05-08') AND (p.purchase_date > DATE '2019-05-08')
GROUP BY p.client_id

【讨论】：