是什么导致 OWIN 管道中的 OperationCanceledException？答案

【问题标题】：What is causing OperationCanceledException in OWIN pipeline?是什么导致 OWIN 管道中的 OperationCanceledException？
【发布时间】：2017-12-27 20:27:29
【问题描述】：

我在我的 API 日志中发现了很多 OperationCanceledExceptions。我认为这与高负载条件有关，但这可能只是因为在高负载下有更多的调用，因此有更多的异常。异常的堆栈跟踪没有到达任何控制器代码；它通过我的一些 OWIN 中间件，但在到达任何控制器之前就死了。对不同 API 端点的许多调用都是相同的堆栈跟踪：

System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MyDomain.Api.Handlers.ApiCultureHandler.<SendAsync>d__0.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\Api\Handlers\ApiCultureHandler.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.AuditLogMiddleware.<Invoke>d__1.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\AuditLogMiddleware.cs:line 23
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.ShardingMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ShardingMiddleware.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.StructureMapMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\StructureMapMiddleware.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DependencyResolution.Api.Middleware.ExceptionHandlerMiddleware.<Invoke>d__3.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ExceptionHandlerMiddleware.cs:line 33

这很令人不安，因为我不知道如何调试它。

其他可能有用的信息：在发生这些错误时，我的一个 API 服务器在 CPU 使用率方面表现出我称之为“Bart Simpson”的模式。 CPU 会在大约 20-30 秒内达到 95-100% 的范围，然后在 20-30 秒内下降到 5-10%，然后再次恢复。循环无限重复。因为我们在生产中，所以我没有时间进一步调查，我杀死了那个服务器，让我们的自动缩放规则启动一个新服务器，然后它工作正常。但我不能保证 Bart Simpson 模式不会影响我们启动的任何其他 API 服务器，我也不知道是什么原因造成的。

有什么想法吗？

【问题讨论】：

在堆栈跟踪中包含的中间件中，哪些是自定义的，哪些是第三方的？不是您或您的团队编写的任何内容都是第三方。
@Nkosi 任何以DependencyResolution 或MyDomain 开头的东西（显然是经过消毒的）都是在内部编写的。剩下的是微软。
可能在某个内部模块中的某个地方，您正在混合异步等待和阻塞调用（.Result 或 .Wait()），这可能会导致死锁，从而导致操作超时并取消。堆栈跟踪已经为您提供了嫌疑人列表。
@Nkosi 这可能是一个解决方案。想要将其发布为答案？
@ShaulBehr 你能解决这个问题吗？我也一样，它只是淹没了服务器

标签： asp.net-web-api owin

【解决方案1】：

由于调用堆栈中的代码中没有其他错误，因此这可能是误报。如果客户端在服务器响应之前关闭连接，则操作将被取消。例如，假设您的客户端有一个心跳端点，它每 5 秒调用一次，调用需要 250 毫秒。每次客户端关闭时，有 5% 的机会取消操作。有许多正常情况下，客户端可能会在收到响应之前切断连接。 OperationCanceledException 本身不应引起警报。但是，请务必考虑与客户数量和您的具体应用相关的频率。

【讨论】：

【解决方案2】：

正如所讨论的，在堆栈跟踪中列出的其中一个自定义中间件中，有可能某处混合了 async await 和阻塞调用（.Result 或 @ 987654322@) 这可能会导致死锁，从而导致操作挂起、超时和取消。

堆栈跟踪 sn-p 已经提供了可能的嫌疑人列表。

建议首先检查列出的处理请求管道的中间件的Invoke。

【讨论】：

这是一个好主意，我在一次重大重构中根除了所有潜在的阻塞代码，但是很遗憾，我们仍然收到错误...