尝试从键空间读取时，Cassandra 复制节点失败

Question

我的 Cassandra (3.0.2) 复制集中有 3 个节点。我的一致性级别是“ONE”。一开始，我所有的键空间的复制因子都等于 1。我通过表更改对其进行了更改，并在所有节点上运行了“nodetool repair”。现在，当我尝试选择一些数据（不是在每个键空间上）时，我会得到这样的结果（从 keyspace.table 中选择 *）：

Traceback（最近一次调用最后一次）： 文件“/usr/bin/cqlsh.py”，第 1258 行，在 perform_simple_statement 结果 = future.result() 文件“cassandra/cluster.py”，第 3781 行，在 cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:73073) 提高 self._final_exception ReadFailure：来自服务器的错误：code=1300 [Replica(s) failed to execute read] message="操作失败 - 收到 0 个响应和 1 个失败" info={'failures': 1, 'received_responses': 0, 'required_responses' : 1, '一致性': 'ONE'}

在“/var/log/cassandra/system.log”中我得到：

WARN [SharedPool-Worker-2] 2017-04-07 12:46:20,036 AbstractTracingAwareExecutorService.java:169 - 线程 Thread[SharedPool-Worker-2,5,main] 上未捕获的异常：{} java.lang.AssertionError: null 在 org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState.updateBlock(AbstractSSTableIterator.java:463) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader.computeNext(SSTableIterator.java:268) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:352) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:426) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:286) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1721) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2375) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121] 在 org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-3.0.2.jar:3.0.2] 在 org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.2.jar:3.0.2] 在 java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] 调试 [SharedPool-Worker-1] 2017-04-07 12:46:20,037 ReadCallback.java:126 - 失败；收到 0 个回复，共 1 个回复

我也明白了：

调试 [SharedPool-Worker-1] 2017-04-07 13:20:30,002 ReadCallback.java:126 - 超时；收到 0 个回复，共 1 个回复

我检查了端口 9042 和 7000 上的节点之间是否存在连接。我更改了“/etc/cassandra/cassandra.yml”中的选项，例如“read_request_timeout_in_ms”、“range_request_timeout_in_ms”、“write_request_timeout_in_ms”或“truncate_request_timeout_in_ms”。我更改了文件“~/.cassandra/cqlshrc”和选项“client_timeout = 3600”。另外，当我这样做时，例如“从 keyspace.table 中选择 *，其中 column1 = 'value' and column2 = value”，我得到：

ReadTimeout：来自服务器的错误：code=1200 [协调节点超时等待副本节点的响应] message="操作超时 - 仅收到 0 个响应。" info={'received_responses': 0, 'required_responses': 1, '一致性': 'ONE'}

有什么想法吗？

Answer 1

这或多或少是一条评论，但因为有太多要说的，它不适合评论。

很高兴说出您更改了值的复制因子。我只是假设它是 3，因为它非常标准。再说一次，因为您只有 3 人的集群，有时将 RF 设置为 2。您还提到您更新了表上的复制因子。据我所知，复制因子是在键空间级别设置的。

如果您发布了对发生错误的键空间的描述，将会非常有帮助。

考虑到select * from something 在您的集群中可能会变得非常密集，尤其是在您有大量数据的情况下。如果您要在 cqlsh 中执行此查询，您可能会返回 10 000，然后您再次只提到 cqlsh 而没有应用程序代码，所以我只是注意这个等。

您能否提供nodetool status 以确保您实际上没有在某些节点关闭的情况下运行查询。因为第一个错误看起来像这样。

通过您发布堆栈跟踪的第二个错误，您看起来好像缺少磁盘上的一些 sstables？是否有其他进程以某种方式操纵了 sstables？

您还更改了 cassandra.yaml 中的许多属性，基本上您将预期响应时间减少了近 50%，我想这也难怪节点没有时间响应...计算整个表通常需要更多时间超过 3.6 秒

根本没有解释为什么这些值会改变。

Answer 2

Marko Švaljek，是的，我将复制因子从 1 更改为 3（因为我的复制中有 3 个节点）。你说得对;您更改键空间上的复制因子，这就是我所做的。在这里你有我的键空间的描述，通常我会遇到一些错误（但当然它会发生在其他键空间）：

soi@cqlsh> desc keyspace engine;

CREATE KEYSPACE engine WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE engine.messages (
    persistence_id text,
    partition_nr bigint,
    sequence_nr bigint,
    timestamp timeuuid,
    timebucket text,
    message blob,
    tag1 text,
    tag2 text,
    tag3 text,
    used boolean static,
    PRIMARY KEY ((persistence_id, partition_nr), sequence_nr, timestamp, timebucket)
) WITH CLUSTERING ORDER BY (sequence_nr ASC, timestamp ASC, timebucket ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50', 'min_threshold': '4', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE MATERIALIZED VIEW engine.eventsbytag1 AS
    SELECT tag1, timebucket, timestamp, persistence_id, partition_nr, sequence_nr, message
    FROM engine.messages
    WHERE persistence_id IS NOT NULL AND partition_nr IS NOT NULL AND sequence_nr IS NOT NULL AND tag1 IS NOT NULL AND timestamp IS NOT NULL AND timebucket IS NOT NULL
    PRIMARY KEY ((tag1, timebucket), timestamp, persistence_id, partition_nr, sequence_nr)
    WITH CLUSTERING ORDER BY (timestamp ASC, persistence_id ASC, partition_nr ASC, sequence_nr ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE engine.config (
    property text PRIMARY KEY,
    value text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

CREATE TABLE engine.metadata (
    persistence_id text PRIMARY KEY,
    deleted_to bigint,
    properties map<text, text>
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

通常我得到代码错误号。 1200 或 1300，您可以在第一篇文章中看到。这里有我的“nodetool status”：

ubuntu@cassandra-db1:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  192.168.1.13  3.94 MB    256          ?       8ebcc3fe-9869-44c5-b7a5-e4f0f5a0beb1  rack1
UN  192.168.1.14  4.26 MB    256          ?       977831cb-98fe-4170-ab15-2b4447559003  rack1
UN  192.168.1.15  4.94 MB    256          ?       7515a967-cbdc-4d89-989b-c0a2f124173f  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

我不认为其他进程可以操作磁盘上的某些数据。我会补充一点，我有类似的复制，我有更多的数据并且我没有这样的问题。

Answer 3

已修复！我将 Cassandra 版本从 3.0.2 更改为 3.0.9 并解决了问题。