配置 Hazelcast CPSubsystem 重试超时答案

【问题标题】：Configure Hazelcast CPSubsystem Retries Timeout配置 Hazelcast CPSubsystem 重试超时
【发布时间】：2019-08-16 14:19:24
【问题描述】：

目前我在CPSubsystem 中注册了三个实例。

      ----- 
     | I1* | * Leader
      ----- 

 ----       ---- 
| I2 |     | I3 |
 ----       ----

当所有实例都运行起来时，所有实例都已注册并在CPSubsystem 上看到彼此，一切都按预期工作。以下调用用于在所有实例之间执行分布式锁：

getHazelcastInstance().getCpSubsystem().getLock(lockDefinition.getLockEntryName())

当其中两个实例死亡并且没有领导者或其他实例可用于执行领导者选举时，我注意到了一个问题：

      ----- 
     | XXX | * DEAD
      ----- 

 ----       ----- 
| I2 |     | XXX | * DEAD
 ----       -----

正在运行的实例然后尝试获取分布式锁，但是请求冻结执行getLock方法，导致请求排队几分钟（当实例成为唯一的实例时需要配置超时子系统）。

我还注意到以下日志被永久打印：

2019-08-16 10:56:21.697  WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:23.737  WARN 1337 --- [cached.thread-8] c.h.nio.tcp.TcpIpConnectionErrorHandler  : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 106
2019-08-16 10:56:23.927  WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:26.006  WARN 1337 --- [onMonitorThread] c.h.s.i.o.impl.Invocation                : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1468, invocationTime=1565963786004 (2019-08-16 10:56:26.004), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=130, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.core.MemberLeftException: Member [127.0.0.1]:5702 - ab45ea09-c8c9-4f03-b3db-42b7b440d161 this has left cluster!
2019-08-16 10:56:26.232  WARN 1337 --- [cached.thread-8] c.h.nio.tcp.TcpIpConnectionErrorHandler  : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 107
2019-08-16 10:56:26.413  WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:27.143  WARN 1337 --- [onMonitorThread] c.h.s.i.o.impl.Invocation                : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1479, invocationTime=1565963787142 (2019-08-16 10:56:27.142), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=140, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5703, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.spi.exception.TargetNotMemberException: Not Member! target: CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, partitionId: 81, operation: com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp, service: hz:core:raft
2019-08-16 10:56:28.835  WARN 1337 --- [cached.thread-6] c.h.nio.tcp.TcpIpConnectionErrorHandler  : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 108
2019-08-16 10:56:28.941  WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:31.038  WARN 1337 --- [cached.thread-3] c.h.nio.tcp.TcpIpConnectionErrorHandler  : [127.0.0.1]:5702 [dev] [3.12.1] Removing connection to endpoint [127.0.0.1]:5701 Cause => java.net.SocketException {Connection refused to address /127.0.0.1:5701}, Error-Count: 109
2019-08-16 10:56:31.533  WARN 1337 --- [ration.thread-1] Impl$LeaderFailureDetectionTask(default) : [127.0.0.1]:5702 [dev] [3.12.1] We are FOLLOWER and there is no current leader. Will start new election round...
2019-08-16 10:56:31.555  WARN 1337 --- [.async.thread-3] c.h.s.i.o.impl.Invocation                : [127.0.0.1]:5702 [dev] [3.12.1] Retrying invocation: Invocation{op=com.hazelcast.cp.internal.operation.ChangeRaftGroupMembershipOp{serviceName='hz:core:raft', identityHash=1295439737, partitionId=81, replicaIndex=0, callId=1493, invocationTime=1565963791554 (2019-08-16 10:56:31.554), waitTimeout=-1, callTimeout=60000, groupId=CPGroupId{name='default', seed=0, commitIndex=6}, membersCommitIndex=0, member=CPMember{uuid=4792972d-d430-48f5-93ed-cb0e1fd8aed2, address=[127.0.0.1]:5703}, membershipChangeMode=REMOVE}, tryCount=250, tryPauseMillis=500, invokeCount=150, callTimeoutMillis=60000, firstInvocationTimeMs=1565963740657, firstInvocationTime='2019-08-16 10:55:40.657', lastHeartbeatMillis=0, lastHeartbeatTime='1969-12-31 21:00:00.000', target=[127.0.0.1]:5702, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}, Reason: com.hazelcast.cp.exception.NotLeaderException: CPMember{uuid=ab45ea09-c8c9-4f03-b3db-42b7b440d161, address=[127.0.0.1]:5702} is not LEADER of CPGroupId{name='default', seed=0, commitIndex=6}. Known leader is: N/A

有没有办法识别实例现在单独运行，如果是，在获取新锁期间不要阻塞应用程序？

我正在寻找某种机制以不以任何方式阻止应用程序的流动，即使应用程序单独运行，我也会使用常规的 j.u.c.l.ReentrantLock 而不是 FencedLock。

【问题讨论】：

标签： java spring spring-boot hazelcast

【解决方案1】：

CP 子系统被设计为在没有足够的可用成员首先形成 CP 子系统时阻止对属于 CP 子系统族的数据结构的所有操作。该属性由CPSubsystemConfig.setCPMemberCount(int)控制

hazelcastInstance.getCPSubsystem().getCPSubsystemManagementService().getCPMembers() 会给你集群中的 CP 成员。

要确定集群成员数，您可以使用 hazelcastInstance.getCluster().getMembers() 和/或使用 MembershipListener 来让成员加入或离开活动。

【讨论】：

您好，我已经通过对getMembers 调用的更多处理解决了这个问题，问题就在那里，将回答解释该方法的问题

【解决方案2】：

经过几天的测试，我得出以下结论：

但是CPSubsystem 至少需要三个模块才能开始工作，但可以运行两个实例
在我提出的最灾难性的可能场景中（只有 一个实例在运行），没有什么可做的，您的环境可能正处于闲置时间，某种干预或关注将是需要解决此中断问题

我决定阻止请求被执行，以防这种情况发生，以保持模块之间所有操作的一致性。

这个决定是在阅读大量材料后做出的（here、here、here、here、here 并模拟了here 上的场景。

所以方法如下：

try {
    if( !hz.isCpInstanceAvailable() ) {
        throw new HazelcastUnavailableException("CPSubsystem is not available");
    }
    ... acquires the lock ...
} catch (HazelcastUnavailableException e) {
    LOG.error("Error retrieving Hazelcast Distributed Lock :( Please check the CPSubsystem health among all instances", e);
    throw e;
}

isCpInstanceAvailable 方法将执行三个验证：

如果当前应用程序注册在CPSubsystem
如果CPSubsystem 已启动
如果CPSubsystem 中的可用成员最少

所以这里是解决方案：

protected boolean isCpInstanceAvailable() {
    try {
        return getCPLocalMember() != null && getCPMembers().get(getMemberValidationTimeout(), TimeUnit.SECONDS).size() > ONE_MEMBER;
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
        LOG.error("Issue retrieving CP Members", e);
    }

    return false;
}

protected ICompletableFuture<Collection<CPMember>> getCPMembers() {
    return Optional.ofNullable(getCPSubsystemManagementService().getCPMembers()).orElseThrow(
            () -> new HazelcastUnavailableException("CP Members not available"));
}

protected CPMember getCPLocalMember() {
    return getCPSubsystemManagementService().getLocalCPMember();
}

问题来了，简单地调用getCPMembers().get() 会导致我遇到长时间的停顿（默认超时）。

所以我使用了getCPMembers().get(getMemberValidationTimeout(), TimeUnit.SECONDS)，如果调用超过预期的超时时间会抛出错误。

【讨论】：