在 Visual Studio 中调试 Web 应用程序时出现 Hangfire DistributedLockTimeoutException

Question

我在调试使用 Hangfire 处理排队的 ASP.NET MVP 应用程序时遇到问题。当我在 Visual Studio 中开始调试时（数据库远程部署在 Azure 上），我得到一堆 DistributedLockTimeoutExceptions 和 RecurringJobScheduler:Debug 错误。它只发生在调试期间。当应用程序部署和运行时，或者如果我在没有通过 VS 调试的情况下启动它，问题不会出现。

发生的情况是出现错误，然后应用程序一遍又一遍地重试连接至少一个小时（通常大约 2-3 小时）。在应用程序最终加载后，对于执行的每个操作，整个过程都会重复。

我真的很想调试应用程序，但是由于需要时间，这几乎是不可能的，而且它经常崩溃。问题越来越大。一开始只需要几分钟就可以加载应用程序，并且越来越糟糕（一开始我什至没有看到任何错误，因为加载是完全可以接受的）。

错误出现多次，但都基本相同。经常出现的错误如下：

Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:locks:schedulepoller' resource.
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
   at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context)
   at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)
Hangfire.SqlServer.ExpirationManager:Debug: Background process 'Hangfire.SqlServer.ExpirationManager' started.
Hangfire.SqlServer.CountersAggregator:Debug: Background process 'Hangfire.SqlServer.CountersAggregator' started.
Hangfire.SqlServer.ExpirationManager:Debug: Removing outdated records from the 'AggregatedCounter' table...
Hangfire.SqlServer.CountersAggregator:Debug: Aggregating records in 'Counter' table...
Hangfire.SqlServer.ExpirationManager:Debug: Removing outdated records from the 'Hash' table...
Hangfire.Server.Worker:Debug: Background process 'Worker #ea15664b' started.
Hangfire.SqlServer.CountersAggregator:Debug: Aggregating records in 'Counter' table...
Hangfire.Server.Worker:Debug: Background process 'Worker #eb7f9473' started.
Hangfire.Server.DelayedJobScheduler:Debug: Error occurred during execution of 'DelayedJobScheduler' process. Execution will be retried (attempt #1) in 00:00:01 seconds.


Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'HangFire:locks:schedulepoller' resource.
   at Hangfire.SqlServer.SqlServerDistributedLock.Acquire(IDbConnection connection, String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
   at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
   at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
   at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context)
   at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)
Hangfire.Server.DelayedJobScheduler:Information: Error occurred during execution of 'DelayedJobScheduler' process. Execution will be retried (attempt #2) in 00:00:03 seconds.


    System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
       at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
       at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
       at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
       at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
       at System.Data.SqlClient.SqlConnection.Open()
       at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()
       at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)
       at Hangfire.SqlServer.SqlServerConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
       at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock(JobStorage storage, Action`1 action)
       at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)
       at Hangfire.Server.AutomaticRetryProcess.Execute(BackgroundProcessContext context)
    Hangfire.Server.RecurringJobScheduler:Debug: Error occurred during execution of 'RecurringJobScheduler' process. Execution will be retried (attempt #1) in 00:00:01 seconds.

我已经尝试重新部署应用程序，但这并没有太大帮助。我试图在网上查看一些类似错误的提及，但它们并没有提供很多见解，并且提供的解决方案对我来说也不起作用。我检查了锁表，但不包括在内。

HangFire 上也几乎没有运行任何作业，除了每 30 分钟 1 个，但它不需要太多资源。否则处理作业的问题很少。

非常欢迎任何帮助！

Answer 1

首先检查您的查询是否以最佳方式编写。

您是否制作了必要的索引？

在ConnectionString中添加Connection Timeout=240。（240 秒 => 4 分钟）

示例： "Data Source=.;Initial Catalog=Hangfire;Persist Security Info=True;User ID=sa;Password=123;MultipleActiveResultSets=True;Application Name=hang-fire;Connection Timeout=240"

如果您使用ADO.NET 执行查询，请在SqlCommand 中设置CommandTimeout = 30000000;。

在这种情况下，请尝试将 WITH(NOLOCK) 或 WITH(READUNCOMMITTED) 添加到您的 Select 语句中。

示例： select * from person WITH(NOLOCK) where name='amin'