引导或修复节点时损坏的管道答案

【问题标题】：Broken pipe when bootstrapping or repair node引导或修复节点时损坏的管道
【发布时间】：2015-12-31 22:05:39
【问题描述】：

我在两个数据中心有 2 个节点“A”和“B”，

A 在 DC1，B 在 DC2。

DC1 = 3，DC2 = 1 的复制因子。

我尝试将新节点 "C" 加入 DC1，但我在节点 "A" 上收到此错误消息：

ERROR [STREAM-OUT-/X.X.XX.XXX] 2015-12-31 02:39:05,252 StreamSession.java:524 - [Stream #a74ecdc0-af67-11e5-8429-a94b3a1b0e16] Streaming error occurred
org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.4.jar:2.2.4]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:427) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:492) ~[na:1.8.0_65]
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:607) ~[na:1.8.0_65]
        at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.4.jar:2.2.4]
        ... 11 common frames omitted

我尝试在节点 A 上运行 nodetool repair，但我得到：

修复范围 xx xx 的会话 xxx 失败，错误流失败

[2015-12-31 14:36:12,722] Repair session 75e93060-af4c-11e5-8429-a94b3a1b0e16 for range (-4171974372789419026,-4027636102743839484] failed with error Stream failed (progress: 9%)
[2015-12-31 16:48:26,913] Repair session 75f76130-af4c-11e5-8429-a94b3a1b0e16 for range (-1477833652511894033,-1464467586067376161] finished (progress: 12%)
[2015-12-31 17:19:40,353] Repair session 75f82483-af4c-11e5-8429-a94b3a1b0e16 for range (4136345906437366778,4261334469649078197] failed with error Stream failed (progress: 12%)

在日志中出现同样的Broken pipe错误

我发现一些信息表明这可能是网络问题。我用mtr检查节点之间的连接，但这够了吗？

是否有任何其他可能仅在失败的范围内重试修复？

【问题讨论】：

标签： cassandra broken-pipe

【解决方案1】：

CompressedStreamReader 遇到大于 2.14GB 的部分时似乎存在错误（因为使用 int 而不是 long 来计算部分长度...）

请查看此错误https://issues.apache.org/jira/browse/CASSANDRA-10961，已附加一个修补的 JAR，并将包含在 Cassandra 2.2.5 中

【讨论】：