春天用 Spark 创建 Redis 集群客户端

Question

我有一个项目通过客户端创建连接到独立 redis：

public RedisConnectionFactory clientCreation() {

        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(5);
        poolConfig.setTestOnBorrow(true);
        poolConfig.setTestOnReturn(true);

        JedisConnectionFactory client = new JedisConnectionFactory(poolConfig);
        client.setHostName(redisEastEndpoint);
        client.setPort(redisEastPort);
        client.setTimeout(redisTimeout);
        return client;

jedis 和 spring-data-redis 使用的库版本是：

[INFO] +- redis.clients:jedis:jar:2.9.3:compile
[INFO] |  \- org.apache.commons:commons-pool2:jar:2.6.2:compile
[INFO] +- org.springframework.data:spring-data-redis:jar:2.1.8.RELEASE:compile
[INFO] |  +- org.springframework.data:spring-data-keyvalue:jar:2.1.8.RELEASE:compile
[INFO] |  +- org.springframework:spring-oxm:jar:5.1.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-aop:jar:5.1.7.RELEASE:compile
[INFO] |  \- org.springframework:spring-context-support:jar:5.1.7.RELEASE:compile

现在我需要迁移到集群redis，并将客户端创建更改为

public RedisConnectionFactory clientCreation() {

        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(5);
        poolConfig.setTestOnBorrow(true);
        poolConfig.setTestOnReturn(true);


        RedisClusterConfiguration redisClusterConfiguration = new RedisClusterConfiguration(redisEastEndpoint);
        if(StringUtils.isNotBlank(redisPassword)) {
            redisClusterConfiguration.setPassword(redisPassword);
        }

        JedisClientConfiguration.JedisClientConfigurationBuilder jedisClientConfiguration = JedisClientConfiguration.builder();
        jedisClientConfiguration.connectTimeout(Duration.ofSeconds(redisTimeout));
        jedisClientConfiguration.usePooling().poolConfig(poolConfig);

        return new JedisConnectionFactory(redisClusterConfiguration,
                jedisClientConfiguration.build());

通过此代码更改，我在集群中得到 No reachable node as ：

Caused by: org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
    at org.springframework.data.redis.connection.jedis.JedisClusterConnection.convertJedisAccessException(JedisClusterConnection.java:760)
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.convertJedisAccessException(JedisClusterStringCommands.java:556)
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.setNX(JedisClusterStringCommands.java:186)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.setNX(DefaultedRedisConnection.java:288)
    at org.springframework.data.redis.core.DefaultValueOperations.lambda$setIfAbsent$10(DefaultValueOperations.java:296)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
    at org.springframework.data.redis.core.DefaultValueOperations.setIfAbsent(DefaultValueOperations.java:296)
    at org.springframework.data.redis.support.atomic.RedisAtomicLong.initializeIfAbsent(RedisAtomicLong.java:147)
    at org.springframework.data.redis.support.atomic.RedisAtomicLong.<init>(RedisAtomicLong.java:92)
    at org.springframework.data.redis.support.atomic.RedisAtomicLong.<init>(RedisAtomicLong.java:62)
    at com.capitalone.api.customers.customerapi.service.config.RedisConfig.westESCID(RedisConfig.java:126)
    at com.capitalone.api.customers.customerapi.service.config.RedisConfig$$EnhancerBySpringCGLIB$$88e1d043.CGLIB$westESCID$4(<generated>)
    at com.capitalone.api.customers.customerapi.service.config.RedisConfig$$EnhancerBySpringCGLIB$$88e1d043$$FastClassBySpringCGLIB$$7889c704.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
    at com.capitalone.api.customers.customerapi.service.config.RedisConfig$$EnhancerBySpringCGLIB$$88e1d043.westESCID(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 39 more
Caused by: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
    at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:61)
    at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:78)
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:113)
    at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:58)
    at redis.clients.jedis.BinaryJedisCluster.setnx(BinaryJedisCluster.java:273)
    at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.setNX(JedisClusterStringCommands.java:184)


    ... 59 more

}

由于我们有一个在 spark-2.1.3 中运行的 spark 应用程序，由于版本依赖性，我需要坚持使用相同的 spring-data-redis。 不升级jedis和spring-data-redis的版本，如何连接集群redis？

Answer 1

瓶颈是 spark 2.1.3。一旦 spark 升级到 2.4.4，它就可以正常工作了。