为什么 cassandra 在 sstables 压缩期间因 OutOfMemoryError 失败答案

【问题标题】：Why cassandra fails with OutOfMemoryError during sstables compaction为什么 cassandra 在 sstables 压缩期间因 OutOfMemoryError 失败
【发布时间】：2014-10-01 16:11:29
【问题描述】：

您好，这可能是一个愚蠢的问题，但我没有通过谷歌找到答案。

所以我有什么：

java 1.7
cassandra 1.2.8 在单节点中运行，带有 -Xmx1G 和 -Xms1G 且未对 yaml 文件进行任何更改

我已经创建了下一个测试列族：

CREATE COLUMN FAMILY TEST_HUGE_SF
    WITH comparator = UTF8Type
    AND key_validation_class=UTF8Type;

然后我尝试在这个列族中插入行。我使用 astyanax lib 访问 cassandra：

    final long START = 1;
    final long MAX_ROWS_COUNT = 1000000000; // 1 Billion

    Keyspace keyspace = AstyanaxProvider.getAstyanaxContext().getClient();

    ColumnFamily<String, String> cf = new ColumnFamily<>(
        "TEST_HUGE_SF", 
        StringSerializer.get(), 
        StringSerializer.get());

    MutationBatch mb = keyspace.prepareMutationBatch()
            .withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
    for (long i = START; i<MAX_ROWS_COUNT; i++) {
        long t = i % 1000;
        if (t == 0) {
            System.out.println("pushed: " + i);
            mb.execute();
            Thread.sleep(1);
            mb = keyspace.prepareMutationBatch()
                    .withRetryPolicy(new BoundedExponentialBackoff(250, 5000, 20));
        }

        ColumnListMutation<String> clm = mb.withRow(cf, String.format("row_%012d", i));
        clm.putColumn("col1", i);
        clm.putColumn("col2", t);
    }
    mb.execute();

从代码中可以看出，我尝试插入 10 亿行，每行包含两列，每列包含简单的 long 值。

插入约 1.22 亿行后，- cassandra 因 OutOfMemoryError 而崩溃。在日志中有下一个：

 INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,334 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941,].  865 252 169 bytes to 901 723 715 (~104% of original) in 922 963ms = 0,931728MB/s.  26 753 257 total rows, 26 753 257 unique.  Row merge counts were {1:26753257, 2:0, 3:0, 4:0, }
 INFO [CompactionExecutor:1571] 2014-08-08 08:31:45,337 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1571] 2014-08-08 08:31:46,167 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1571,1,main]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
    at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
    at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
    at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
    at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
    at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
    at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
 INFO [CompactionExecutor:1570] 2014-08-08 08:31:46,994 CompactionTask.java (line 263) Compacted 4 sstables to [\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-3213,].  34 773 524 bytes to 35 375 883 (~101% of original) in 44 162ms = 0,763939MB/s.  1 151 482 total rows, 1 151 482 unique.  Row merge counts were {1:1151482, 2:0, 3:0, 4:0, }
 INFO [CompactionExecutor:1570] 2014-08-08 08:31:47,105 CompactionTask.java (line 106) Compacting [SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2069-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-629-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-2941-Data.db'), SSTableReader(path='\var\lib\cassandra\data\cyodaTest1\TEST_HUGE_SF\cyodaTest1-TEST_HUGE_SF-ib-1328-Data.db')]
ERROR [CompactionExecutor:1570] 2014-08-08 08:31:47,110 CassandraDaemon.java (line 132) Exception in thread Thread[CompactionExecutor:1570,1,main]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.cassandra.io.util.Memory.<init>(Memory.java:52)
    at org.apache.cassandra.io.util.Memory.allocate(Memory.java:60)
    at org.apache.cassandra.utils.obs.OffHeapBitSet.<init>(OffHeapBitSet.java:40)
    at org.apache.cassandra.utils.FilterFactory.createFilter(FilterFactory.java:143)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:137)
    at org.apache.cassandra.utils.FilterFactory.getFilter(FilterFactory.java:126)
    at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:445)
    at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:92)
    at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1958)
    at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:144)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:59)
    at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:62)
    at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:191)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

我看到 cassandra 在 sstables 压缩期间崩溃。

这是否意味着要处理更多行 cassandra 需要更多堆空间？

我预计堆空间不足只会影响性能。谁能描述一下，为什么我的期望是错误的？

【问题讨论】：

我见过由于缺少堆而导致的失败。您有机会升级到更新版本的 cassandra 吗？
是的，我打算用 2.0.x 版本做同样的测试。
另外，请查看 DataStax 的内存调整指南。如果您采纳他们的建议，可能会节省您的时间和麻烦。 datastax.com/documentation/cassandra/2.0/cassandra/operations/…
1GB 是一个很小的堆，尤其是对于写入密集型工作负载。您应该至少使用 4 GB。

标签： cassandra out-of-memory

【解决方案1】：

其他人注意到这一点 - 1GB 堆非常小。使用 Cassandra 2.0，您可以查看此调整指南以获取更多信息： http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html

另一个考虑因素是垃圾收集的处理方式。在 cassandra 日志目录中，还应该有 GC 日志，指示收集的频率和时间。如果需要，您可以使用 jvisualvm 实时监控它们。

【讨论】：