DSE Solr：并发查询请求期间的 Netty 错误

Question

我有一个小型（5 节点）集群，我用它来测试具有各种特定索引策略的 DSE 搜索性能（我在 CentOS 上使用 DSE 4.5.2）。当我一次只提交一个查询时，一切正常，但是当我增加负载时，我很快就会看到网络异常。在 SolrJ 中（我的大部分测试都使用 java 客户端），这些表现为

Error with server [MY URL]: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: address already in use by: [id: 0x2d3873c4, local:E:2d3873c4 => local:local]

每当我有多个线程同时查询集群时，无论我实际指向哪个节点，也不管请求是否来自同一台服务器，这似乎总是会发生。在使用其他客户端（例如 PySolr4）时，我也能够引发这种行为，因此这似乎是 DSE 设置问题，而不是我的测试应用程序的一些奇怪问题。

此外，如果我将 dse.yaml 中的节点间通信类型从netty 更改为http，同一个集群将愉快地响应客户端提交的数百个并发请求。我最初在 dse.yaml 中保留了默认的 netty* 设置，但在遇到错误后，我尝试了一些选项（max_connections、acceptor/worker threads 等）；不幸的是，我没有找到很多关于这些值的文档，而且我所做的任何更改似乎都没有达到预期的效果。

从概念上讲，使用 netty 似乎会提供比 http 更好的性能，所以我想尝试找出这里可能出了什么问题。提前感谢您的任何建议。

请参阅下面来自/var/log/cassandra/system.log 的堆栈跟踪，对应于这些错误之一：

ERROR [http-8983-exec-4] 2014-11-19 18:35:51,557 SolrDispatchFilter.java (line 696) Error request exception: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
java.lang.RuntimeException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.tryNewConnection(ShardClient.java:204)
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.sendTo(ShardClient.java:163)
    at com.datastax.bdp.search.solr.handler.shard.netty.NettyShardHandler.submit(NettyShardHandler.java:76)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:287)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1889)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:723)
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:185)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:147)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:891)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:750)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.ChannelException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
    at io.netty.channel.local.LocalChannelRegistry.register(LocalChannelRegistry.java:46)
    at io.netty.channel.local.LocalChannel.doBind(LocalChannel.java:178)
    at io.netty.channel.local.LocalChannel$LocalUnsafe.connect(LocalChannel.java:349)
    at io.netty.channel.DefaultChannelPipeline$HeadHandler.connect(DefaultChannelPipeline.java:1008)
    at io.netty.channel.DefaultChannelHandlerContext.invokeConnect(DefaultChannelHandlerContext.java:495)
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:480)
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:465)
    at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:847)
    at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:199)
    at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:165)
    at io.netty.channel.local.LocalEventLoop.run(LocalEventLoop.java:33)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
    ... 1 more

Answer 1

似乎是 Netty/JVM 的 bug，与：https://github.com/netty/netty/issues/1765 有关。您可以尝试其他操作系统和/或 JVM 版本吗？