logstash 发送数据到elasticsearch集群的问题答案

【问题标题】：logstash Issue sending data to elasticseach clusterlogstash 发送数据到elasticsearch集群的问题
【发布时间】：2019-10-25 07:37:12
【问题描述】：

我已将 ELK 堆栈升级到 7.4 版本（filebeat、logstash、elasticalert、kibana）。我正在使用弹性搜索云。

升级后，logstash 日志文件中会显示以下错误。但很少有记录可以在 kibana 中看到。

[2019-10-25T15:22:01,578][ERROR][logstash.outputs.elasticsearch][main] 尝试向 elasticsearch 发送批量请求，但 Elasticsearch 似乎无法访问或关闭！ {:error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@:9243/][Manticore::ConnectTimeout] 连接超时", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::池::HostUnreachableError", :will_retry_in_seconds=>2} [2019-10-25T15:22:01,595][WARN][logstash.outputs.elasticsearch][main] 将 url 标记为已死。最后一个错误：[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxx@:9243/][Manticore::ConnectTimeout] 连接超时 {:url= >https://user:xxxxxx@/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@:9243/][Manticore::ConnectTimeout] 连接超时", :error_class=>"LogStash: :Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

我该如何解决这个问题？

【问题讨论】：

标签： elastic-stack elk

【解决方案1】：

通过优化管道解决了这个问题。在我的设置中，每个 filebeat 服务器发送不同的日志。因此根据日志大小配置了单独的管道和不同的工作人员。请参阅下面的 pipeline.yml 文件。

- pipeline.id: intake
  config.string: |
    input { beats { port => 5043 } }
    output {
    if [log][file][path] =~ "request-response-logger" {
        pipeline { send_to => requestResponseLogger }
    } else if [host][name] =~ "wso2telcohubgateway" and [log][file][path] =~ "wso2carbon"  {
        pipeline { send_to => wso2gateway }
    } else if [host][name] =~ "wso2esb" and [log][file][path] =~ "wso2carbon" {
        pipeline { send_to => wso2esb }
    } else if [host][name] =~ "wso2ei" and [log][file][path] =~ "wso2carbon" {
        pipeline { send_to => wso2ei }
    } else if [host][name] =~ "wso2telcomig" and [log][file][path] =~ "wso2carbon" {
        pipeline { send_to => wso2telcomig }
    }
    }


- pipeline.id: requestResponseLogger
  pipeline.workers: 1
  path.config: "/etc/logstash/conf.d/requestResponseLogger.conf"

- pipeline.id: wso2gateway
  pipeline.batch.delay: 20
  pipeline.workers: 15
  path.config: "/etc/logstash/conf.d/wso2gateway.conf"

- pipeline.id: wso2esb
  pipeline.workers: 20
  pipeline.batch.delay: 10
  path.config: "/etc/logstash/conf.d/wso2esb.conf"

- pipeline.id: wso2ei
  pipeline.workers: 2
  path.config: "/etc/logstash/conf.d/wso2ei.conf"

- pipeline.id: wso2telcomig
  pipeline.workers: 10
  path.config: "/etc/logstash/conf.d/wso2telcomig.conf"

【讨论】：