起因是这样的,logstash突然出现内存上涨而没有停下来,由于是进行日志采集用的,可以容忍部分数据丢失。

1. 出现这种情况,首先看下日志,发现似乎是jvm内存不够用,于是调整jvm大小,重新启动logstash

如果这样就解决了,那,是不可能的。观察一段时间后,发现内存还是在缓涨,最后又挂了。

2. 考虑是否是elasticsearch的写入瓶颈

修改es配置:

indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 150mb
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
thread_pool.bulk.queue_size: 300
thread_pool.index.queue_size: 300
indices.fielddata.cache.size: 15%
重启es

修改logstash配置(logstash.yml):

pipeline.batch.size: 1000

pipeline.workers: 4

重启logstash

然而并没有用

3. 再次查看日志

logstash内存缓慢增长直到内存溢出

[WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 6291456000, max: 6306136064)
之前光注意OutOfDirectMemoryError这个错误了,可不管怎么调大jvm,内存始终会被消耗殆尽。

但是造成这种现象的原因似乎是由An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.造成的

4.最后解决办法是将logstash版本提升两个版本

问题解决

 

 

 

相关文章: