Hazelcast 在成员重启后终止客户端连接答案

【问题标题】：Hazelcast kills client connections after member restartHazelcast 在成员重启后终止客户端连接
【发布时间】：2017-05-10 10:53:09
【问题描述】：

我使用的是 3.8.1，我注意到会员被退回后客户端周围的重新连接行为存在一些问题。尽管设置了重新连接尝试，但客户端经常在记录有关无法获得集群连接的重复警告后无论如何都会断开连接 - 尽管始终至少有一个幸存成员。

那么为什么会发生断开连接呢？我相信，如果至少有一个成员还活着，那么任何客户端都不应该断开连接。为什么客户在会员退会后反复登录并发出警告？

测试：我有 2 个成员加入。我有 3 个客户端通过 tcp 连接到两个成员。客户端配置为重新连接 1000 次。客户端 1 循环并简单地将随机值放入同一个键。客户端 2 和 3 是此地图的侦听器并记录更新。

步骤：启动成员 1，成员 2。启动客户端 2、3（侦听器）。启动客户端 1（作者）。此时一切正常。日志中没有警告。

弹跳成员 1，等待它开始。弹跳成员 2。弹跳成员 1。

客户并不总是，但经常会这样报告：

Put 29: d16b0acd-d0d6-4722-b511-7fd975774f8c
May 10, 2017 11:41:10 AM com.hazelcast.client.spi.impl.ClusterListenerSupport
WARNING: hz.client_0 [dev] [3.8.1] Unable to get alive cluster connection, try in 0 ms later, attempt 4 of 1000.
May 10, 2017 11:41:10 AM com.hazelcast.client.spi.impl.ClusterListenerSupport
INFO: hz.client_0 [dev] [3.8.1] Trying to connect to [127.0.0.1]:5701 as owner member
Put 30: f91ec949-19bd-4039-95d3-28c7abd0f241
May 10, 2017 11:41:15 AM com.hazelcast.client.spi.impl.ClusterListenerSupport
INFO: hz.client_0 [dev] [3.8.1] Trying to connect to [127.0.0.1]:5702 as owner member
Put 31: 2af9fb36-d501-4b0b-9fc8-6b36d467a929
May 10, 2017 11:41:20 AM com.hazelcast.client.spi.impl.ClusterListenerSupport
INFO: hz.client_0 [dev] [3.8.1] Trying to connect to [127.0.0.1]:5701 as owner member
Put 32: f08bd6ca-a79f-44f4-9064-42e47953c37a

客户端仍然可以操作并监听事件，但一段时间后，他们经常断开连接：

May 10, 2017 11:25:04 AM com.hazelcast.core.LifecycleService
INFO: hz.client_0 [dev] [3.8.1] HazelcastClient 3.8.1 (20170411 - f1e9264) is SHUTTING_DOWN
May 10, 2017 11:25:04 AM com.hazelcast.client.connection.ClientConnectionManager
INFO: hz.client_0 [dev] [3.8.1] Removed connection to endpoint: [localhost]:5701, connection: ClientConnection{alive=false, connectionId=5, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[closed]}, remoteEndpoint=[localhost]:5701, lastReadTime=2017-05-10 11:25:04.021, lastWriteTime=2017-05-10 11:25:04.084, closedTime=2017-05-10 11:25:04.084, lastHeartbeatRequested=2017-05-10 11:25:03.834, lastHeartbeatReceived=2017-05-10 11:25:03.834, connected server version=3.8.1}
May 10, 2017 11:25:04 AM com.hazelcast.client.connection.ClientConnectionManager
INFO: hz.client_0 [dev] [3.8.1] Removed connection to endpoint: [localhost]:5702, connection: ClientConnection{alive=false, connectionId=6, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[closed]}, remoteEndpoint=[localhost]:5702, lastReadTime=2017-05-10 11:24:59.513, lastWriteTime=2017-05-10 11:25:04.084, closedTime=2017-05-10 11:25:04.084, lastHeartbeatRequested=2017-05-10 11:17:43.840, lastHeartbeatReceived=2017-05-10 11:17:43.840, connected server version=3.8.1}
May 10, 2017 11:25:04 AM com.hazelcast.core.LifecycleService
INFO: hz.client_0 [dev] [3.8.1] HazelcastClient 3.8.1 (20170411 - f1e9264) is SHUTDOWN

客户端断开连接：

INFO: hz.client_0 [dev] [3.8.1] Removed connection to endpoint: [localhost]:5701, connection: ClientConnection{alive=false, connectionId=8, socketChannel=DefaultSocketChannelWrapper{socketChannel=java.nio.channels.SocketChannel[closed]}, remoteEndpoint=[localhost]:5701, lastReadTime=2017-05-10 11:45:31.791, lastWriteTime=2017-05-10 11:45:31.791, closedTime=2017-05-10 11:45:31.791, lastHeartbeatRequested=2017-05-10 11:45:23.604, lastHeartbeatReceived=2017-05-10 11:45:23.605, connected server version=3.8.1}
May 10, 2017 11:45:31 AM com.hazelcast.core.LifecycleService
INFO: hz.client_0 [dev] [3.8.1] HazelcastClient 3.8.1 (20170411 - f1e9264) is CLIENT_DISCONNECTED

会员报告：

May 10, 2017 11:47:32 AM com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask
WARNING: [localhost]:5701 [dev] [3.8.1] Member having uuid 8847c2e3-2fcb-428f-a827-d0e24f5624a1 is not part of the cluster. Client Authentication rejected.
May 10, 2017 11:47:32 AM com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask
WARNING: [localhost]:5701 [dev] [3.8.1] Received auth from Connection[id=222, /127.0.0.1:5701->/127.0.0.1:53883, endpoint=null, alive=true, type=NONE] with principal ClientPrincipal{uuid='f1e5d928-23b2-4fdf-bd3f-9db1778a5a8c', ownerUuid='8847c2e3-2fcb-428f-a827-d0e24f5624a1'} , authentication failed

主要成员：

public class HzNodeTest {
    private HazelcastInstance service;

    @Before
    public void setUp() throws Exception {
        Config config = new Config();
        JoinConfig join = config.getNetworkConfig().setPort(5701).getJoin();
        join.getMulticastConfig().setEnabled(false);
        join.getAwsConfig().setEnabled(false);
        join.getTcpIpConfig().addMember("localhost:5701").addMember("localhost:5702").setEnabled(true);
        service = Hazelcast.newHazelcastInstance(config);
    }

    @After
    public void tearDown() throws Exception {
        service.shutdown();
    }

    @Test
    public void testStart() throws InterruptedException {
        Thread.sleep(1000000000);
    }
}

Secondary member:
public class HzNodeSecondaryTest {
    private HazelcastInstance service;

    @Before
    public void setUp() throws Exception {
        Config config = new Config();
        JoinConfig join = config.getNetworkConfig().setPort(5702).getJoin();
        join.getMulticastConfig().setEnabled(false);
        join.getAwsConfig().setEnabled(false);
        join.getTcpIpConfig().addMember("localhost:5701").addMember("localhost:5702").setEnabled(true);
        service = Hazelcast.newHazelcastInstance(config);
    }

    @After
    public void tearDown() throws Exception {
        service.shutdown();
    }

    @Test
    public void testStart() throws InterruptedException {
        Thread.sleep(1000000000);
    }
}

Listener:

public class HzListenerTest {
    private HazelcastInstance service;
    private AtomicLong counter = new AtomicLong();

    @Before
    public void setUp() throws Exception {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.getNetworkConfig().addAddress("localhost:5701").addAddress("localhost:5702").setConnectionAttemptLimit(1000);
        service = HazelcastClient.newHazelcastClient(clientConfig);
    }

    @After
    public void tearDown() throws Exception {
        service.shutdown();
    }

    @Test
    public void testListen() throws InterruptedException {
        service.getMap("TEST").addEntryListener(new Listener(), true);

        Thread.sleep(1000000);
    }

    private class Listener implements EntryAddedListener, EntryUpdatedListener, EntryRemovedListener,
            EntryEvictedListener {

        @Override
        public void entryAdded(EntryEvent event) {
            System.out.println("onAdded " + counter.getAndIncrement() + ": " + event);
        }

        @Override
        public void entryEvicted(EntryEvent event) {
            System.out.println("onEvicted " + counter.getAndIncrement() + ": " + event);
        }

        @Override
        public void entryRemoved(EntryEvent event) {
            System.out.println("onRemoved " + counter.getAndIncrement() + ": " + event);
        }

        @Override
        public void entryUpdated(EntryEvent event) {
            System.out.println("onUpdated " + counter.getAndIncrement() + ": "+ event);
        }
    }
}

Updater:

public class HzUpdaterTest {
    private HazelcastInstance service;

    @Before
    public void setUp() throws Exception {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.getNetworkConfig().addAddress("localhost:5701").addAddress("localhost:5702").setConnectionAttemptLimit(1000);
        service = HazelcastClient.newHazelcastClient(clientConfig);
        service.getMap("TEST").put("1", UUID.randomUUID().toString());
    }

    @After
    public void tearDown() throws Exception {
        service.shutdown();
    }

    @Test
    public void testSpin() {
        for (int i = 0; i < 10000; i++) {
            try {
                String value = UUID.randomUUID().toString();
                service.getMap("TEST").put("1", value);
                System.out.println("Put " + i + ": " + value);
                Thread.sleep(5000);
            } catch (Exception ex) {
                System.out.println(ex);
            }
        }
    }
}

【问题讨论】：

标签： cluster-computing hazelcast hazelcast-imap

【解决方案1】：

每当 Hazelcast 客户端连接到集群时，该客户端可以建立连接的第一个成员（这是CLIENT_CONNECTED 被触发时）成为该客户端的所有者成员。客户端从所有者成员那里获取集群信息和其他成员的地址。之后，客户端知道所有成员并根据需要直接连接到他们。

当您关闭的成员恰好是该客户端的所有者成员时，CLIENT_DISCONNECTED 事件将被触发。但是在很短的时间内（同一秒左右），客户端应该重新建立与剩余成员的连接。

【讨论】：

【解决方案2】：

我在你的日志中看到

Unable to get alive cluster connection, try in 0 ms later, attempt 4 of 1000.

可以试试这个属性

<propertyname="hazelcast.invalidation.reconciliation.interval.seconds">5</property>

【讨论】：