Cassandra 脑裂分区答案

【问题标题】：Cassandra split brain partitionCassandra 脑裂分区
【发布时间】：2015-11-24 23:36:13
【问题描述】：

我们正在跨两个 AWS 可用区（ap-southeast-1 和 ap-southeast-2）运行一个 6 节点 cassandra 集群。

在愉快地运行了几个月后，集群被滚动重启以修复挂起的修复，现在每个组都认为对方宕机了。

Cluster Information:
    Name: MegaportGlobal
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        220727fa-88d2-366f-9473-777e32744c37: [10.5.13.117, 10.5.12.245, 10.5.13.93]

        UNREACHABLE: [10.4.0.112, 10.4.0.169, 10.4.2.186]

Cluster Information:
    Name: MegaportGlobal
    Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
    Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
    Schema versions:
        3932d237-b907-3ef8-95bc-4276dc7f32e6: [10.4.0.112, 10.4.0.169, 10.4.2.186]

        UNREACHABLE: [10.5.13.117, 10.5.12.245, 10.5.13.93]

来自悉尼的“nodetool status”报告大多数新加坡节点已关闭：

Datacenter: ap-southeast-2
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns    Host ID                               Rack
UN  10.4.0.112   9.04 GB    256     ?       b9c19de4-4939-4112-bf07-d136d8a57b57  2a
UN  10.4.0.169   9.34 GB    256     ?       2d7c3ac4-ae94-43d6-9afe-7d421c06b951  2a
UN  10.4.2.186   10.72 GB   256     ?       4dc8b155-8f9a-4532-86ec-d958ac207f40  2b
Datacenter: ap-southeast-1
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns    Host ID                               Rack
UN  10.5.13.117  9.45 GB    256     ?       324ee189-3e72-465f-987f-cbc9f7bf740b  1a
DN  10.5.12.245  10.25 GB   256     ?       bee281c9-715b-4134-a033-00479a390f1e  1b
DN  10.5.13.93   12.29 GB   256     ?       a8262244-91bb-458f-9603-f8c8fe455924  1a

但是来自新加坡，所有悉尼节点都报告为宕机：

ap-southeast-2
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns    Host ID                               Rack
DN  10.4.0.112   8.91 GB    256     ?       b9c19de4-4939-4112-bf07-d136d8a57b57  2a
DN  10.4.0.169   ?          256     ?       2d7c3ac4-ae94-43d6-9afe-7d421c06b951  2a
DN  10.4.2.186   ?          256     ?       4dc8b155-8f9a-4532-86ec-d958ac207f40  2b
Datacenter: ap-southeast-1
==========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens  Owns    Host ID                               Rack
UN  10.5.13.117  9.45 GB    256     ?       324ee189-3e72-465f-987f-cbc9f7bf740b  1a
UN  10.5.12.245  10.25 GB   256     ?       bee281c9-715b-4134-a033-00479a390f1e  1b
UN  10.5.13.93   12.29 GB   256     ?       a8262244-91bb-458f-9603-f8c8fe455924  1a

更令人困惑的是，在悉尼执行的“nodetool gossipinfo”报告状态一切正常 - 正常：

/10.5.13.117
  generation:1440735653
  heartbeat:724504
  SEVERITY:0.0
  DC:ap-southeast-1
  LOAD:1.0149565738E10
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:1a
  STATUS:NORMAL,-1059943672916788858
  RELEASE_VERSION:2.1.6
  NET_VERSION:8
  RPC_ADDRESS:10.5.13.117
  INTERNAL_IP:10.5.13.117
  HOST_ID:324ee189-3e72-465f-987f-cbc9f7bf740b
/10.5.12.245
  generation:1440734497
  heartbeat:728014
  SEVERITY:0.0
  DC:ap-southeast-1
  LOAD:1.100647505E10
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:1b
  STATUS:NORMAL,-1029869455226513030
  RELEASE_VERSION:2.1.6
  NET_VERSION:8
  RPC_ADDRESS:10.5.12.245
  INTERNAL_IP:10.5.12.245
  HOST_ID:bee281c9-715b-4134-a033-00479a390f1e
/10.4.0.112
  generation:1440973751
  heartbeat:4135
  SEVERITY:0.0
  DC:ap-southeast-2
  LOAD:9.70297176E9
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:2a
  RELEASE_VERSION:2.1.6
  STATUS:NORMAL,-1016623069114845926
  NET_VERSION:8
  RPC_ADDRESS:10.4.0.112
  INTERNAL_IP:10.4.0.112
  HOST_ID:b9c19de4-4939-4112-bf07-d136d8a57b57
/10.5.13.93
  generation:1440734532
  heartbeat:727909
  SEVERITY:0.0
  DC:ap-southeast-1
  LOAD:1.3197536002E10
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:1a
  STATUS:NORMAL,-1021689296016263011
  RELEASE_VERSION:2.1.6
  NET_VERSION:8
  RPC_ADDRESS:10.5.13.93
  INTERNAL_IP:10.5.13.93
  HOST_ID:a8262244-91bb-458f-9603-f8c8fe455924
/10.4.0.169
  generation:1440974511
  heartbeat:1832
  SEVERITY:0.0
  DC:ap-southeast-2
  LOAD:1.0023502338E10
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:2a
  RELEASE_VERSION:2.1.6
  STATUS:NORMAL,-1004223692762353764
  NET_VERSION:8
  RPC_ADDRESS:10.4.0.169
  INTERNAL_IP:10.4.0.169
  HOST_ID:2d7c3ac4-ae94-43d6-9afe-7d421c06b951
/10.4.2.186
  generation:1440734382
  heartbeat:730171
  SEVERITY:0.0
  DC:ap-southeast-2
  LOAD:1.1507595081E10
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RACK:2b
  STATUS:NORMAL,-10099894685483463
  RELEASE_VERSION:2.1.6
  NET_VERSION:8
  RPC_ADDRESS:10.4.2.186
  INTERNAL_IP:10.4.2.186
  HOST_ID:4dc8b155-8f9a-4532-86ec-d958ac207f40

在新加坡执行的相同命令不包括悉尼任何节点的状态：

/10.5.12.245
  generation:1440974710
  heartbeat:1372
  SEVERITY:0.0
  LOAD:1.100835806E10
  RPC_ADDRESS:10.5.12.245
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  STATUS:NORMAL,-1029869455226513030
  DC:ap-southeast-1
  RACK:1b
  INTERNAL_IP:10.5.12.245
  HOST_ID:bee281c9-715b-4134-a033-00479a390f1e
/10.5.13.117
  generation:1440974648
  heartbeat:1561
  SEVERITY:0.0
  LOAD:1.0149992022E10
  RPC_ADDRESS:10.5.13.117
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  STATUS:NORMAL,-1059943672916788858
  DC:ap-southeast-1
  RACK:1a
  HOST_ID:324ee189-3e72-465f-987f-cbc9f7bf740b
  INTERNAL_IP:10.5.13.117
/10.4.0.112
  generation:1440735420
  heartbeat:23
  SEVERITY:0.0
  LOAD:9.570546197E9
  RPC_ADDRESS:10.4.0.112
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  DC:ap-southeast-2
  RACK:2a
  INTERNAL_IP:10.4.0.112
  HOST_ID:b9c19de4-4939-4112-bf07-d136d8a57b57
/10.5.13.93
  generation:1440734532
  heartbeat:729862
  SEVERITY:0.0
  LOAD:1.3197536002E10
  RPC_ADDRESS:10.5.13.93
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  STATUS:NORMAL,-1021689296016263011
  DC:ap-southeast-1
  RACK:1a
  INTERNAL_IP:10.5.13.93
  HOST_ID:a8262244-91bb-458f-9603-f8c8fe455924
/10.4.0.169
  generation:1440974511
  heartbeat:15
  SEVERITY:0.5076141953468323
  RPC_ADDRESS:10.4.0.169
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  DC:ap-southeast-2
  RACK:2a
  INTERNAL_IP:10.4.0.169
  HOST_ID:2d7c3ac4-ae94-43d6-9afe-7d421c06b951
/10.4.2.186
  generation:1440734382
  heartbeat:15
  SEVERITY:0.0
  RPC_ADDRESS:10.4.2.186
  NET_VERSION:8
  SCHEMA:7bf335ee-61ae-36c6-a902-c70d785ec7a3
  RELEASE_VERSION:2.1.6
  DC:ap-southeast-2
  RACK:2b
  INTERNAL_IP:10.4.2.186
  HOST_ID:4dc8b155-8f9a-4532-86ec-d958ac207f40

重启时，每个节点都能看到远程DC一小会儿：

INFO  [GossipStage:1] 2015-08-31 10:53:07,638 OutboundTcpConnection.java:97 - OutboundTcpConnection using coalescing strategy DISABLED
INFO  [HANDSHAKE-/10.4.2.186] 2015-08-31 10:53:08,267 OutboundTcpConnection.java:485 - Handshaking version with /10.4.2.186
INFO  [HANDSHAKE-/10.4.0.169] 2015-08-31 10:53:08,287 OutboundTcpConnection.java:485 - Handshaking version with /10.4.0.169
INFO  [HANDSHAKE-/10.5.12.245] 2015-08-31 10:53:08,391 OutboundTcpConnection.java:485 - Handshaking version with /10.5.12.245
INFO  [HANDSHAKE-/10.5.13.93] 2015-08-31 10:53:08,498 OutboundTcpConnection.java:485 - Handshaking version with /10.5.13.93
INFO  [GossipStage:1] 2015-08-31 10:53:08,537 Gossiper.java:987 - Node /10.5.12.245 has restarted, now UP
INFO  [HANDSHAKE-/10.5.13.117] 2015-08-31 10:53:08,537 OutboundTcpConnection.java:485 - Handshaking version with /10.5.13.117
INFO  [GossipStage:1] 2015-08-31 10:53:08,656 StorageService.java:1642 - Node /10.5.12.245 state jump to normal
INFO  [GossipStage:1] 2015-08-31 10:53:08,820 Gossiper.java:987 - Node /10.5.13.117 has restarted, now UP
INFO  [GossipStage:1] 2015-08-31 10:53:08,852 Gossiper.java:987 - Node /10.5.13.93 has restarted, now UP
INFO  [SharedPool-Worker-33] 2015-08-31 10:53:08,907 Gossiper.java:954 - InetAddress /10.5.12.245 is now UP
INFO  [GossipStage:1] 2015-08-31 10:53:08,947 StorageService.java:1642 - Node /10.5.13.93 state jump to normal
INFO  [GossipStage:1] 2015-08-31 10:53:09,007 Gossiper.java:987 - Node /10.4.0.169 has restarted, now UP
WARN  [GossipTasks:1] 2015-08-31 10:53:09,123 FailureDetector.java:251 - Not marking nodes down due to local pause of 7948322997 > 5000000000
INFO  [GossipStage:1] 2015-08-31 10:53:09,192 StorageService.java:1642 - Node /10.4.0.169 state jump to normal
INFO  [HANDSHAKE-/10.5.12.245] 2015-08-31 10:53:09,199 OutboundTcpConnection.java:485 - Handshaking version with /10.5.12.245
INFO  [GossipStage:1] 2015-08-31 10:53:09,203 Gossiper.java:987 - Node /10.4.2.186 has restarted, now UP
INFO  [GossipStage:1] 2015-08-31 10:53:09,206 StorageService.java:1642 - Node /10.4.2.186 state jump to normal
INFO  [SharedPool-Worker-34] 2015-08-31 10:53:09,215 Gossiper.java:954 - InetAddress /10.5.13.93 is now UP
INFO  [SharedPool-Worker-33] 2015-08-31 10:53:09,259 Gossiper.java:954 - InetAddress /10.5.13.117 is now UP
INFO  [SharedPool-Worker-33] 2015-08-31 10:53:09,259 Gossiper.java:954 - InetAddress /10.4.0.169 is now UP
INFO  [SharedPool-Worker-33] 2015-08-31 10:53:09,259 Gossiper.java:954 - InetAddress /10.4.2.186 is now UP
INFO  [GossipStage:1] 2015-08-31 10:53:09,296 StorageService.java:1642 - Node /10.4.0.169 state jump to normal
INFO  [GossipStage:1] 2015-08-31 10:53:09,491 StorageService.java:1642 - Node /10.5.12.245 state jump to normal
INFO  [HANDSHAKE-/10.5.13.117] 2015-08-31 10:53:09,509 OutboundTcpConnection.java:485 - Handshaking version with /10.5.13.117
INFO  [GossipStage:1] 2015-08-31 10:53:09,511 StorageService.java:1642 - Node /10.5.13.93 state jump to normal
INFO  [HANDSHAKE-/10.5.13.93] 2015-08-31 10:53:09,538 OutboundTcpConnection.java:485 - Handshaking version with /10.5.13.93

然后，没有任何错误，节点被标记为关闭：

INFO  [GossipTasks:1] 2015-08-31 10:53:34,410 Gossiper.java:968 - InetAddress /10.5.13.117 is now DOWN
INFO  [GossipTasks:1] 2015-08-31 10:53:34,411 Gossiper.java:968 - InetAddress /10.5.12.245 is now DOWN
INFO  [GossipTasks:1] 2015-08-31 10:53:34,411 Gossiper.java:968 - InetAddress /10.5.13.93 is now DOWN

我们尝试了多次重启，但行为仍然相同。

*编辑

看起来和Gossip协议有关……打开extra debug显示PHI值在稳步增加：

TRACE [GossipTasks:1] 2015-08-31 16:46:44,706 FailureDetector.java:262 - PHI for /10.4.0.112 : 2.9395029255
TRACE [GossipTasks:1] 2015-08-31 16:46:45,727 FailureDetector.java:262 - PHI for /10.4.0.112 : 3.449690761
TRACE [GossipTasks:1] 2015-08-31 16:46:46,728 FailureDetector.java:262 - PHI for /10.4.0.112 : 3.95049114
TRACE [GossipTasks:1] 2015-08-31 16:46:47,730 FailureDetector.java:262 - PHI for /10.4.0.112 : 4.451317456
TRACE [GossipTasks:1] 2015-08-31 16:46:48,732 FailureDetector.java:262 - PHI for /10.4.0.112 : 4.952114357
TRACE [GossipTasks:1] 2015-08-31 16:46:49,733 FailureDetector.java:262 - PHI for /10.4.0.112 : 5.4529339645
TRACE [GossipTasks:1] 2015-08-31 16:46:50,735 FailureDetector.java:262 - PHI for /10.4.0.112 : 5.953951289
TRACE [GossipTasks:1] 2015-08-31 16:46:51,737 FailureDetector.java:262 - PHI for /10.4.0.112 : 6.4547808165
TRACE [GossipTasks:1] 2015-08-31 16:46:52,738 FailureDetector.java:262 - PHI for /10.4.0.112 : 6.955600038
TRACE [GossipTasks:1] 2015-08-31 16:46:53,740 FailureDetector.java:262 - PHI for /10.4.0.112 : 7.456422601
TRACE [GossipTasks:1] 2015-08-31 16:46:54,742 FailureDetector.java:262 - PHI for /10.4.0.112 : 7.957303284
TRACE [GossipTasks:1] 2015-08-31 16:46:55,751 FailureDetector.java:262 - PHI for /10.4.0.112 : 8.461658576
TRACE [GossipTasks:1] 2015-08-31 16:46:56,755 FailureDetector.java:262 - PHI for /10.4.0.112 : 8.9636610545
TRACE [GossipTasks:1] 2015-08-31 16:46:57,763 FailureDetector.java:262 - PHI for /10.4.0.112 : 9.4676926445

重启后 PHI 值稳定增加，直到超过故障阈值并标记为 DOWN。

对如何进行有什么建议吗？

【问题讨论】：

nodetool gossipinfo 显示的状态仅表示各个节点通过 gossip 宣布的引导/正常/离开生命周期状态。它对应于nodetool status 中的“State=Normal/Leaving/Joining/Moving”字段，但不指示所有其他节点是否可以访问该节点（“Status Up/Down”）。在您的情况下，似乎存在连接问题可能是由网络问题引起的节点之间的问题。你应该在两个节点上运行tcpdump，试图在彼此之间进行八卦，看看是否是这种情况。

标签： split cassandra partition

【解决方案1】：

对于滞后的网络，将 phi 故障检测阈值提高到 12 或 15。这在 AWS 中通常是必需的，尤其是跨区域。

【讨论】：

根据文档，我们已经将其增加到 12 个。 PHI 值开始时很低，但随着时间的推移快速线性增长，并在大约 30 秒内超过阈值。它很快就达到了 1000，因此即使是更大的值也无济于事。目前的想法是清除八卦状态并重新启动整个集群......
您是否进行了 tcpdump 以确保它们真正在通信？每次错过心跳时，phi 值应该继续增加 - 如果它无限期地攀升，它是否无法正常通信？想必安全组一定是正确的，但是 tcpdump 在 7000 上的节点之间查看它是否正在丢弃 tcp 流？
我们正在运行 tcdump，我们可以看到通信。在 C* 日志 (TRACE) 中，我可以看到 GossipDigestSynMessage，然后是 GossipDigestAckMessage，但没有 GossipDigestAck2Message... 也许是 MTU 问题？需要花更多时间在 tcpdump 上...

【解决方案2】：

问题出在 AWS 的网络链接上，特别是网络 MTU。由于我们的路由配置存在细微问题，悉尼和新加坡 AWS 之间的数据路径变得不对称。

我想吸取的教训是，如果模式在 DC 内而不是 DC 之间是正常的，那么很可能是网络，MTU 之类的东西很重要，即使 ping 和 telnet 等看起来没问题。

感谢 Jeff 和 Stefan 的投入 - 如果你在布里斯班找到自己，我会请你喝啤酒！

【讨论】：