Opscenter 不显示我的密钥空间答案

【问题标题】：Opscenter does not show my key spacesOpscenter 不显示我的密钥空间
【发布时间】：2013-12-02 03:19:13
【问题描述】：

在我的 opscenter 网页中，在架构选项卡中，我无法看到我的任何键空间（0 Keyspaces | 0 Column Families）并且在日志中继续说

WARN [rollup-snapshot] 2013-11-18 20:02:47,373 42937 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,373 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,373 42938 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,373 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,373 42939 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,373 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42940 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42941 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42942 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42943 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42944 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42945 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42946 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,374 42947 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42948 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42949 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42950 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42951 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42952 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42953 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 42954 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,375 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,376 42955 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,376 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,376 42956 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-18 20:02:47,376 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-18 20:02:47,376 42957 operations dropped so far.

我重新启动了datastax-agent，但在下面的日志文件中仍然找不到任何错误是agent.log文件

启动日志：

 Starting DataStax agent monitor datastax_agent_monitor[  OK  ]
    log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
     INFO [main] 2013-11-27 01:37:45,191 Loading conf files: /var/lib/datastax-agent/conf/address.yaml
     INFO [main] 2013-11-27 01:37:45,260 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_25
     INFO [main] 2013-11-27 01:37:45,261 Waiting for the config from OpsCenter
     INFO [main] 2013-11-27 01:37:45,262 Attempting to determine Cassandra's broadcast address through JMX
     INFO [main] 2013-11-27 01:37:45,264 Starting Stomp
     INFO [main] 2013-11-27 01:37:45,264 SSL communication is disabled
     INFO [main] 2013-11-27 01:37:45,264 Creating stomp connection to x.x.x.x:61620
     INFO [Initialization] 2013-11-27 01:37:45,266 New JMX connection (127.0.0.1:7199)
     INFO [StompConnection receiver] 2013-11-27 01:37:45,274 Reconnecting in 0s.
     INFO [StompConnection receiver] 2013-11-27 01:37:45,280 Connected to x.x.x.x:61620
     INFO [main] 2013-11-27 01:37:45,313 Starting Jetty server: {:port 61621, :host nil, :ssl? false, :join? false}
     INFO [Jetty] 2013-11-27 01:37:45,511 Jetty server started
     INFO [StompConnection receiver] 2013-11-27 01:37:45,566 Got new config from OpsCenter: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticket_cache true, :rollups60_ttl 604800, :thrift_port 9160, :ec2_metadata_api_host "x.x.x.x", :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_truststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/log/cassandra/system.log", :thrift_rpc_interface "x.x.x.x", :thrift_ssl_truststore_password nil, :jmx_port 7199, :provisioning 0, :use_ssl 0, :kerberos_debug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCenter", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_truststore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth, dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
     INFO [StompConnection receiver] 2013-11-27 01:37:45,567 New JMX connection (127.0.0.1:7199)
     INFO [Initialization] 2013-11-27 01:37:45,633 Using x.x.x.x as the cassandra broadcast address
     INFO [StompConnection receiver] 2013-11-27 01:37:45,662 Starting up agent collection.
     INFO [Initialization] 2013-11-27 01:37:45,714 agent RPC address is  x.x.x.x
     INFO [Initialization] 2013-11-27 01:37:45,715 agent RPC broadcast address is  x.x.x.x
     INFO [StompConnection receiver] 2013-11-27 01:37:45,721 Starting OS metric collectors (Linux)
     INFO [Initialization] 2013-11-27 01:37:45,723 Clearing ssl.truststore
     INFO [Initialization] 2013-11-27 01:37:45,723 Clearing ssl.truststore.password
     INFO [Initialization] 2013-11-27 01:37:45,723 Setting ssl.store.type to JKS
     INFO [Initialization] 2013-11-27 01:37:45,728 Clearing kerberos.service.principal.name
     INFO [Initialization] 2013-11-27 01:37:45,728 Clearing kerberos.principal
     INFO [Initialization] 2013-11-27 01:37:45,728 Setting kerberos.useTicketCache to true
     INFO [Initialization] 2013-11-27 01:37:45,728 Clearing kerberos.ticketCache
     INFO [Initialization] 2013-11-27 01:37:45,729 Setting kerberos.useKeyTab to true
     INFO [Initialization] 2013-11-27 01:37:45,729 Clearing kerberos.keyTab
     INFO [Initialization] 2013-11-27 01:37:45,729 Setting kerberos.renewTGT to true
     INFO [Initialization] 2013-11-27 01:37:45,729 Setting kerberos.debug to false
     INFO [thrift-init] 2013-11-27 01:37:45,733 Connecting to Cassandra cluster: x.x.x.x (port 9160)
     INFO [StompConnection receiver] 2013-11-27 01:37:45,737 Starting Cassandra JMX metric collectors
     INFO [thrift-init] 2013-11-27 01:37:45,749 Downed Host Retry service started with queue size -1 and retry delay 10s
     INFO [StompConnection receiver] 2013-11-27 01:37:45,755 New JMX connection (127.0.0.1:7199)
     INFO [thrift-init] 2013-11-27 01:37:45,757 Registering JMX me.prettyprint.cassandra.service_Agent Cluster:ServiceType=hector,MonitorType=hector
     INFO [pdp-loader] 2013-11-27 01:37:45,834 in execute with client org.apache.cassandra.thrift.Cassandra$Client@67cf1438
     INFO [thrift-init] 2013-11-27 01:37:45,836 Connected to Cassandra cluster: /Test
     INFO [pdp-loader] 2013-11-27 01:37:45,844 Attempting to load stored metric values.
     INFO [thrift-init] 2013-11-27 01:37:45,841 in execute with client org.apache.cassandra.thrift.Cassandra$Client@67cf1438
     INFO [thrift-init] 2013-11-27 01:37:45,845 Using partitioner: org.apache.cassandra.dht.Murmur3Partitioner
     INFO [jmx-metrics-1] 2013-11-27 01:37:50,748 New JMX connection (127.0.0.1:7199)
     INFO [qtp131393312-25] 2013-11-27 01:38:59,902 HTTP: :get /os-metric/disk-space {} - 200
     INFO [qtp131393312-24] 2013-11-27 01:39:04,468 HTTP: :get /os-metric/disk-space {} - 200

WARN [rollup-snapshot] 2013-11-27 01:42:45,841 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,842 1 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,842 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,842 2 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 3 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 4 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,843 5 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,844 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,844 6 operations dropped so far.
 WARN [rollup-snapshot] 2013-11-27 01:42:45,844 Thrift operation queue is full, discarding thrift operation
 WARN [rollup-snapshot] 2013-11-27 01:42:45,844 7 operations dropped so far.

Thrift 正在运行：

tcp        0      0 0.0.0.0:7199                0.0.0.0:*                   LISTEN      498        21333533   15520/java
tcp        0      0 0.0.0.0:9160                0.0.0.0:*                   LISTEN      498        21334831   15520/java

Cassandra 节点已启动并正在运行。

【问题讨论】：

您包含的第一个日志实际上也是代理日志。这表明代理无法通过 thrift 将数据写入集群，但它不应该影响键空间列表。您能否在 OpsCenter 日志中查找错误并将其包括在此处？从 'curl http://://keyspaces' 发送输出也很有用
我更新了日志，奇怪的是现在我可以看到键空间但是每次我重新启动浏览器都需要很长时间，那些 WARN 呢？
WARN 消息意味着代理无法通过 thrift 连接到 cassandra。您的 cassandra 节点是否已启动并且正在运行？您粘贴的其他堆栈跟踪应在更高版本的 OpsCenter 中修复。可以升级到 3.2.2 或 4.0.1 吗？
我升级到 4.0.1 但仍然是同样的警告。
我将这些添加到 address.yaml 文件 thrift_max_conns: 100 async_pool_size: 100。仍然是同样的错误

标签： datastax-enterprise opscenter

【解决方案1】：

本例中的问题与集群中创建的列族数量有关。大量列族可能会减慢获取键空间和列族列表以及备份度量值插入的速度。您可以配置收集哪些列族的指标。见：

http://www.datastax.com/documentation/opscenter/4.0/webhelp/index.html#opsc/configure/../../opsc/configure/../../opsc/configure/opscExcludingKeyspaces_c.html

如果您不想在具有大量列族的集群上禁用监控，您可以在代理配置中调整一些设置。

thrift_max_conns - the max number of concurrent connections to make to the local node
asysnc_pool_size - the size of the threadpool pulling from a queue of inserts and inserting in to cassandra
async_queue_size - the size of the queue of inserts to send to cassandra, if the queue fills up additional operations will be dropped

【讨论】：

我不想从监控中排除我的任何键空间。有什么办法吗？
我已经用适当的设置编辑了我的答案以进行调整。
这些配置选项是否仍然适用于 DSE 4.6.1、OpsC 5.1.0？他们没有在这里记录：datastax.com/documentation/opscenter/5.1/opsc/configure/…。我在节点的 agent.log 中看到很多以下消息：“Cassandra 操作队列已满，正在丢弃 cassandra 操作”。我的键空间中有 400 多个表