【问题标题】:apache cassandra 3.11.2 nodetool cleanup error java.lang.AssertionError: nullapache cassandra 3.11.2 nodetool 清理错误 java.lang.AssertionError: null
【发布时间】:2019-09-02 05:55:19
【问题描述】:

apache cassandra 3.11.2, 12 个节点,运行 nodetool cleanup 时在一个节点出现此错误:

ERROR [CompactionExecutor:4996] 2019-04-10 16:09:43,938 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:4996,1,main]
java.lang.AssertionError: null
        at org.apache.cassandra.io.compress.CompressionMetadata$Chunk.<init>(CompressionMetadata.java:474) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:239) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:163) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:1137) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionManager$6.execute(CompactionManager.java:475) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:322) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,613 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6bfdb59e) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1698322065:/data/cassandra/data_files/path/mc-135-big-Index.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,626 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6105deee) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@604462250:/data/cassandra/data_files/path/mc-135-big-Data.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,626 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@22504cf1) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@359209204:[Memory@[0..44), Memory@[0..4c8)] was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,627 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5ad45619) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1143514085:Memory@[7eaa3bdf4000..7eaa3bdf7200) was not released before the reference was garbage collected

我们需要做什么?用新节点替换这个节点?这个错误是什么意思? 我们在所有节点中成功完成 nodetool 修复后运行清理。

【问题讨论】:

    标签: cassandra nodetool


    【解决方案1】:

    自 Cassandra 的早期版本以来,这一直是一个问题,并且已经 reported 给开发团队。这种情况似乎是无害的,但如果你发现它只发生在同一个节点上,你可能需要更换它。

    【讨论】:

    • 它发生在集群中的 2 个节点中,有办法断开这些节点并从头开始擦除和重新同步数据吗?这会有帮助吗?还是更好地采用新节点,将它们添加到集群中,然后删除这两个有问题的节点?谢谢。
    • 在进行整体更换之前,您是否尝试过对集群进行修复?一次执行nodetool repair 一个节点将确保所有数据的一致性
    • 我们在集群中的所有节点(一次一个节点)中完成 nodetool 修复后运行 nodetool cleanup。修复完成没有错误.... nodetool repair 没有解决我们运行 nodetool cleanup 时出现的这个问题。无论如何,我们将首先添加 2 个新节点并再次运行修复,然后删除有问题的节点。删除后,我们将再次运行 nodetool repair。听起来好吗?谢谢!
    • 是的,它可能会解决您的问题;只需确保在节点替换完成后一次运行nodetool cleanup 一个节点,因为节点中数据的所有权将会发生变化。
    • 谢谢!替换 2 个节点后,此错误并没有在 2 个新节点上再次出现,而是出现在集群中的另一个节点中,之前没有出现过......可能是 cassandra 错误?
    猜你喜欢
    • 2014-08-24
    • 1970-01-01
    • 1970-01-01
    • 2019-11-12
    • 1970-01-01
    • 2015-05-14
    • 2016-02-02
    • 1970-01-01
    • 2018-06-20
    相关资源
    最近更新 更多