Cassandra 2.2.3 中的 SSTableLoader 错误打开的文件太多答案

【问题标题】：SSTableLoader error in Cassandra 2.2.3 Too Many Open FilesCassandra 2.2.3 中的 SSTableLoader 错误打开的文件太多
【发布时间】：2016-03-08 02:52:57
【问题描述】：

我目前正在使用 SSTableLoader 将数据从多个 CSV 文件加载到 cassandra。之前我在 Cassandra 2.1.9 上运行 SSTableLoader 时，运行成功，没有任何错误。

但是，自从我升级到 cassandra 2.2.3 后，我一直遇到 SSTableLoader 的以下错误：

org.apache.cassandra.io.FSReadError: java.io.IOException: An existing connection was forcibly closed by the remote host
    at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:293) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:96) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) ~[na:1.8.0_45]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_45]
    at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[na:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_45]
    at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:483) ~[na:1.8.0_45]
    at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:579) ~[na:1.8.0_45]
    at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:140) ~[apache-cassandra-2.2.3.jar:2.2.3]
    ... 11 common frames omitted


After examining the system.log file in "/var/log/cassandra", I see that the error was caused by "Too many open files".

Here is the error message in the system.log file:

ERROR [STREAM-IN-/169.254.122.80] 2015-12-02 11:37:48,235 StreamSession.java:524 - [Stream #d33ca5f0-98e8-11e5-b1fd-732c31b2403a] Streaming error occurred
java.nio.file.FileSystemException: /var/lib/cassandra/data/pricing/listings-088f7050945411e59780d168f1e8851b/tmp-la-968-big-Data.db: Too many open files
    at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:1.8.0_60]
    at sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source) ~[na:1.8.0_60]
    at java.nio.channels.FileChannel.open(Unknown Source) ~[na:1.8.0_60]
    at java.nio.channels.FileChannel.open(Unknown Source) ~[na:1.8.0_60]
    at org.apache.cassandra.io.util.SequentialWriter.<init>(SequentialWriter.java:131) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:66) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:75) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:106) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:95) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:129) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:78) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:49) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-2.2.3.jar:2.2.3]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

通过之前关于这个主题的问题进行了研究，我尝试过的一件事是使用 ulimit 命令将打开文件的数量设置为 1000000。但是在重新启动 cassandra 服务后，我仍然收到错误。

我还需要做些什么才能使这些更改生效吗？我必须在 cassandra.yaml 文件中设置一个设置来防止这种情况发生吗？非常感谢。

【问题讨论】：

标签： cassandra cassandra-2.1

【解决方案1】：

使用 ulimit 可能没有按您预期的方式应用它。可以检查 cat /proc/<pid>/limits 在你的 limits.conf 中配置它并在那里更新其他内容，例如 nproc 限制。见http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html

【讨论】：

【解决方案2】：

谢谢你，我设法解决了这个问题。

对于其他人的信息，我做了以下：

编辑 /etc/security 中的 limits.conf 文件
添加以下条目：
- 软文件100000
- 硬文件 100000
保存更改
使用“重启”命令重启机器
重启机器后，我运行'ulimit -a'命令看到打开的文件数是100000。
批量加载过程成功。

【讨论】：

【解决方案3】：

这个问题是由于打开文件描述符总数的限制。使用命令检查此限制

ulimit -a  
or  
ulimit -n

使用命令更改它的简单方法 -

ulimit -n 100000

（我以root用户运行此命令，可能可以使用sudo）

【讨论】：