起因是这样的,logstash突然出现内存上涨而没有停下来,由于是进行日志采集用的,可以容忍部分数据丢失。
1. 出现这种情况,首先看下日志,发现似乎是jvm内存不够用,于是调整jvm大小,重新启动logstash
如果这样就解决了,那,是不可能的。观察一段时间后,发现内存还是在缓涨,最后又挂了。
2. 考虑是否是elasticsearch的写入瓶颈
修改es配置:
indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 150mb
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
thread_pool.bulk.queue_size: 300
thread_pool.index.queue_size: 300
indices.fielddata.cache.size: 15%
重启es
修改logstash配置(logstash.yml):
pipeline.batch.size: 1000
pipeline.workers: 4
重启logstash
然而并没有用
3. 再次查看日志
[WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 6291456000, max: 6306136064)
之前光注意OutOfDirectMemoryError这个错误了,可不管怎么调大jvm,内存始终会被消耗殆尽。
但是造成这种现象的原因似乎是由An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.造成的
4.最后解决办法是将logstash版本提升两个版本
问题解决