Azure Redis 缓存 - 多个错误 TimeoutException：执行 GET {key} 超时答案

【问题标题】：Azure Redis Cache - Multiple errors TimeoutException: Timeout performing GET {key}Azure Redis 缓存 - 多个错误 TimeoutException：执行 GET {key} 超时
【发布时间】：2015-06-16 16:16:33
【问题描述】：

我们将应用程序部署到 Azure。它正在使用 Azure Redis 缓存，我们遇到了很多超时。即：

[TimeoutException: Timeout performing GET textobjectDetails__23290_TextObject, inst: 1, mgr: Inactive, queue: 5, qu=0, qs=5, qc=0, wr=0/0, in=56864/0]

[TimeoutException: Timeout performing GET featured_series_CachedSeries, inst: 1, mgr: Inactive, queue: 4, qu=0, qs=4, qc=0, wr=0/0, in=44470/0]

[TimeoutException: Timeout performing GET SeriesByFranchiseId_1_CachedSeries, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=11252/0]

[TimeoutException: Timeout performing GET media_silo-1-1-0_Media, inst: 1, mgr: Inactive, queue: 3, qu=0, qs=3, qc=0, wr=0/0, in=15188/0]

[TimeoutException: Timeout performing GET textobjectDetails__3092_TextObject, inst: 3, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectbytype_104__TextObject, inst: 11, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_2_14_TextObject, inst: 1, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archived_news_by_group_13586_1_TextObject, inst: 2, mgr: Inactive, queue: 7, qu=0, qs=7, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET textobjectDetails__24404_TextObject, inst: 11, mgr: Inactive, queue: 12, qu=0, qs=12, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET standings_232_lcds_fixtures, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET player_name1099_Player, inst: 4, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET groupnews_1_6_TextObject, inst: 4, mgr: Inactive, queue: 9, qu=0, qs=9, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET archivednewscount__20789_TextObject, inst: 2, mgr: Inactive, queue: 11, qu=0, qs=11, qc=0, wr=0/0, in=65536/0]

[TimeoutException: Timeout performing GET media_id3648_Media, inst: 1, mgr: Inactive, queue: 10, qu=0, qs=10, qc=0, wr=0/0, in=65536/0]

所有异常的主体都是相同的：

StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):509
    StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server):25
    StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags):16
    AB.SiteCaching.Providers.RedisDataSource+<>c__DisplayClasse`1.<RetrieveCacheObject>b__b():0
    Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func):115
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCacheObject[T](String fullCacheKey):56
    AB.SiteCaching.Providers.RedisDataSource.RetrieveCached[T](String key, Func`1 onNotCached, TimeSpan timeOut):61
    DataAccess.Data.Caching.CachedSeries.GetSeriesByFranchiseId(Int32 franchiseId):64
    Shared.Services.SeriesService.LatestYearByFranchiseId(Int32 franchiseId):0
    AllBlacksdotcom.Controllers.FixturesController._MostRecentYearOfFixtures(Int32 franchiseId):0
    (unknown).lambda_method(Closure , ControllerBase , Object[] ):-1
    System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters):0
    System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters):87
    System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d():20
    System.Web.Mvc.Async.AsyncControllerActionInvoker+AsyncInvocationWithFilters+<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f():134
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21+<>c__DisplayClass2b.<BeginInvokeAction>b__1c():0
    System.Web.Mvc.Async.AsyncControllerActionInvoker+<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult):65
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult):0
    System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1.End():41
    System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult):0
    System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper+<>c__DisplayClassa.<EndProcessRequest>b__9():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper+<>c__DisplayClass4.<Wrap>b__3():0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap[TResult](Func`1 func):0
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper.Wrap(Action action):25
    System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerAsyncWrapper.EndProcessRequest(IAsyncResult result):32
    System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride):772

请查看我们的超时设置：

retryTimeoutInMilliseconds = "5000" 
connectionTimeoutInMilliseconds = "5000" 
operationTimeoutInMilliseconds = "1000"

如何处理这些超时？增加 operationTimeoutInMilliseconds 会起作用吗？我还了解到 g-zip 压缩有助于减少从 Redis 读取数据所需的时间。

所有与 Redis 相关的 nuget 包都在其最新版本上。我们正在使用 C1 版本的 Azure Redis 缓存（增加到 C2 会有帮助吗？）。

【问题讨论】：

作为一个起点，请您阅读这篇文章，看看是否有帮助azure.microsoft.com/blog/2015/02/10/…
@pranavrastogi 是的，我看过那个。很遗憾，我现在没有 Azure 提供的 Radis 信息，但会在星期一查看。
@pranavrastogi 嘿，我们检查了带宽，没问题。

标签： caching azure redis stackexchange.redis azure-redis-cache

【解决方案1】：

改善我们情况的几点：

Protobuf-net 代替 BinaryFormatter

我建议使用protobuf-net，因为它会减少您想要存储在缓存中的值的大小。

public interface ICacheDataSerializer
    {
        byte[] Serialize(object o);
        T Deserialize<T>(byte[] stream);
    }

public class ProtobufNetSerializer : ICacheDataSerializer
    {
        public byte[] Serialize(object o)
        {
            using (var memoryStream = new MemoryStream())
            {
                Serializer.Serialize(memoryStream, o);

                return memoryStream.ToArray();
            }
        }

        public T Deserialize<T>(byte[] stream)
        {
            var memoryStream = new MemoryStream(stream);

            return Serializer.Deserialize<T>(memoryStream);
        }
    }

实施重试策略

实现这个 RedisCacheTransientErrorDetectionStrategy 来处理超时问题。

using Microsoft.Practices.TransientFaultHandling;

public class RedisCacheTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        /// <summary>
        /// Custom Redis Transient Error Detenction Strategy must have been implemented to satisfy Redis exceptions.
        /// </summary>
        /// <param name="ex"></param>
        /// <returns></returns>
        public bool IsTransient(Exception ex)
        {
            if (ex == null) return false;

            if (ex is TimeoutException) return true;

            if (ex is RedisServerException) return true;

            if (ex is RedisException) return true;

            if (ex.InnerException != null)
            {
                return IsTransient(ex.InnerException);
            }

            return false;
        }
    }

像这样实例化：

private readonly RetryPolicy _retryPolicy;

// CODE
var retryStrategy = new FixedInterval(3, TimeSpan.FromSeconds(2));
            _retryPolicy = new RetryPolicy<RedisCacheTransientErrorDetectionStrategy>(retryStrategy);

这样使用：

var cachedString = _retryPolicy.ExecuteAction(() => dataCache.StringGet(fullCacheKey));

查看您的代码，以尽量减少缓存调用和您存储在缓存中的值。通过更有效地存储值，我减少了很多错误。

如果这些都没有帮助。移动到更高的缓存（我们最终使用 C3 而不是 C1）。

【讨论】：

【解决方案2】：

在所有异常消息中，您都有类似“in=65536/0”的内容。这表示已在本地（在本地计算机上的内核套接字缓冲区中）接收到 65536 个字节，但尚未由 StackExchange.Redis 处理。这将指向客户端应用程序方面的问题，该问题使系统无法处理您的响应。我能想到的两种可能会导致这种情况：

您的系统没有足够的空闲线程池线程来足够快地处理数据。如果您更新到构建 1.0.450 或更高版本的 StackExchange.Redis，超时消息将包括线程池繁忙程度的统计信息，这将有助于诊断此问题。或者，您可以使用https://github.com/JonCole/SampleCode/blob/master/ThreadPoolMonitor/ThreadPoolLogger.cs 之类的代码在遇到异常时自行记录相同的信息。
您正在从缓存中接收一个非常大的对象，并且在 StackExchange.Redis 上配置的超时时间内未收到该对象的大小。发生这种情况时，排队到大项后面的同一连接上的所有调用将在大项超时时超时。

无论如何，希望这会有所帮助。

【讨论】：

【解决方案3】：

转到 Azure 并获取更大的 Redis 实例。那个

【讨论】：

漂亮的答案 :)）请参阅下面的答案，说明一切。
@JakubHolovsky 你的回答是 9:30 发布的，作者的回答是 9:25 :)