当 kibana 尝试连接时，Elasticsearch 连接被拒绝答案

【问题标题】：Elasticsearch connection refused while kibana is trying to connect当 kibana 尝试连接时，Elasticsearch 连接被拒绝
【发布时间】：2018-12-02 08:01:44
【问题描述】：

我正在尝试使用 docker 容器运行 ELK 堆栈。但是我收到错误消息，kibana 无法与 elasticsearch 建立连接。

kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["error","elasticsearch","admin"],"pid":12,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.18.0.2:9200"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:console@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:elasticsearch@5.6.9","error"],"pid":12,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:metrics@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1  | [2018-06-22T19:31:38,182][INFO ][o.e.d.DiscoveryModule    ] [g8HPieb] using discovery type [zen]
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","plugin:timelion@5.6.9","info"],"pid":12,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["listening","info"],"pid":12,"message":"Server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:38Z","tags":["status","ui settings","error"],"pid":12,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1  | [2018-06-22T19:31:38,634][INFO ][o.e.n.Node               ] initialized
elasticsearch_1  | [2018-06-22T19:31:38,634][INFO ][o.e.n.Node               ] [g8HPieb] starting ...
elasticsearch_1  | [2018-06-22T19:31:38,767][INFO ][o.e.t.TransportService   ] [g8HPieb] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
elasticsearch_1  | [2018-06-22T19:31:38,776][WARN ][o.e.b.BootstrapChecks    ] [g8HPieb] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
logstash_1       | log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
logstash_1       | log4j:WARN Please initialize the log4j system properly.
logstash_1       | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
**logstash_1       | {:timestamp=>"2018-06-22T19:31:40.555000+0000", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}**
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:40Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2018-06-22T19:31:40Z","tags":["warning","elasticsearch","admin"],"pid":12,"message":"No living connections"}

这是我的 docker-comp 的内容

version: "2.0"

services:

  logstash:
    image: logstash:2
    ports:
      - "5044:5044"
    volumes:
      - ./:/config
    command: logstash -f /config/logstash.conf
    links:
      - elasticsearch
    depends_on:
      - elasticsearch

   elasticsearch:
     image: elasticsearch:5.6.9
     ports:
       - "9200:9200"
     volumes:
       - "./es_data/es_data:/usr/share/elasticsearch/data/"

   kibana:
     image: kibana:5
     ports:
       - "5601:5601"
     links:
       - elasticsearch
     environment:
       ELASTICSEARCH_URL: http://elasticsearch:9200
     depends_on:
       - elasticsearch

我的 logstash.conf 的内容

input { beats {      port => 5044    }  }

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
  stdout {
    codec => rubydebug
  }
}

我在 elasticsearch 容器和 kibana 容器上有 curl，我觉得它看起来不错

{
  "name" : "g8HPieb",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "XxH0TcAmQcGqprf6s7TJEQ",
  "version" : {
    "number" : "5.6.9",
    "build_hash" : "877a590",
    "build_date" : "2018-04-12T16:25:14.838Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

curl localhost:9200/_cat/indices?pretty

yellow open .kibana GIBmXdlRQJmI67oq5r4oCg 1 1 1 0 3.2kb 3.2kb

增加虚拟内存大小后

root@sfbp19:~/dockerizing-jenkins# sysctl -p vm.max_map_count = 262144 root@sfbp19:~/dockerizing-jenkins# docker-compose -f docker-compose-elk.yml up

Creating network "dockerizingjenkins_default" with the default driver
Creating dockerizingjenkins_elasticsearch_1
Creating dockerizingjenkins_logstash_1
Creating dockerizingjenkins_kibana_1
Attaching to dockerizingjenkins_elasticsearch_1, dockerizingjenkins_kibana_1, dockerizingjenkins_logstash_1
elasticsearch_1  | [2018-06-26T19:08:19,294][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2018-06-26T19:08:19,363][INFO ][o.e.e.NodeEnvironment    ] [PVmTsqv] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/sfbp19--vg-root)]], net usable_space [671.9gb], net total_space [789.2gb], spins? [possibly], types [ext4]
elasticsearch_1  | [2018-06-26T19:08:19,364][INFO ][o.e.e.NodeEnvironment    ] [PVmTsqv] heap size [1.9gb], compressed ordinary object pointers [true]
elasticsearch_1  | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node               ] node name [PVmTsqv] derived from node ID [PVmTsqv3QnyS3sQarPcJ-A]; set [node.name] to override
elasticsearch_1  | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node               ] version[5.6.9], pid[1], build[877a590/2018-04-12T16:25:14.838Z], OS[Linux/4.4.0-31-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]
elasticsearch_1  | [2018-06-26T19:08:19,369][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [ingest-common]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [lang-expression]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [lang-groovy]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [lang-mustache]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [lang-painless]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [parent-join]
elasticsearch_1  | [2018-06-26T19:08:20,040][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [percolator]
elasticsearch_1  | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [reindex]
elasticsearch_1  | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [transport-netty3]
elasticsearch_1  | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService     ] [PVmTsqv] loaded module [transport-netty4]
elasticsearch_1  | [2018-06-26T19:08:20,041][INFO ][o.e.p.PluginsService     ] [PVmTsqv] no plugins loaded
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:kibana@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:elasticsearch@5.6.9","info"],"pid":13,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["error","elasticsearch","admin"],"pid":13,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.18.0.2:9200"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:console@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:elasticsearch@5.6.9","error"],"pid":13,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:20Z","tags":["status","plugin:metrics@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["status","plugin:timelion@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["listening","info"],"pid":13,"message":"Server running at http://0.0.0.0:5601"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:21Z","tags":["status","ui settings","error"],"pid":13,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
elasticsearch_1  | [2018-06-26T19:08:21,190][INFO ][o.e.d.DiscoveryModule    ] [PVmTsqv] using discovery type [zen]
elasticsearch_1  | [2018-06-26T19:08:21,654][INFO ][o.e.n.Node               ] initialized
elasticsearch_1  | [2018-06-26T19:08:21,654][INFO ][o.e.n.Node               ] [PVmTsqv] starting ...
elasticsearch_1  | [2018-06-26T19:08:21,780][INFO ][o.e.t.TransportService   ] [PVmTsqv] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
logstash_1       | log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
logstash_1       | log4j:WARN Please initialize the log4j system properly.
logstash_1       | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:23Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:23Z","tags":["warning","elasticsearch","admin"],"pid":13,"message":"No living connections"}
logstash_1       | {:timestamp=>"2018-06-26T19:08:23.572000+0000", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
logstash_1       | {:timestamp=>"2018-06-26T19:08:23.790000+0000", :message=>"Pipeline main started"}
elasticsearch_1  | [2018-06-26T19:08:24,837][INFO ][o.e.c.s.ClusterService   ] [PVmTsqv] new_master {PVmTsqv}{PVmTsqv3QnyS3sQarPcJ-A}{coD5A4HyR7-1MedSq8dFUQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)[, ]
elasticsearch_1  | [2018-06-26T19:08:24,869][INFO ][o.e.h.n.Netty4HttpServerTransport] [PVmTsqv] publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch_1  | [2018-06-26T19:08:24,870][INFO ][o.e.n.Node               ] [PVmTsqv] started
elasticsearch_1  | [2018-06-26T19:08:24,989][INFO ][o.e.g.GatewayService     ] [PVmTsqv] recovered [1] indices into cluster_state
elasticsearch_1  | [2018-06-26T19:08:25,148][INFO ][o.e.c.r.a.AllocationService] [PVmTsqv] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:26Z","tags":["status","plugin:elasticsearch@5.6.9","info"],"pid":13,"state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
kibana_1         | {"type":"log","@timestamp":"2018-06-26T19:08:26Z","tags":["status","ui settings","info"],"pid":13,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Elasticsearch plugin is red"}

========================filebeat.yml==================== ==========

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /jenkins/gerrit_volume/logs/*_log
#============================= Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 3
#============================== Kibana =====================================
setup.kibana:
  #host: "localhost:5601"
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.1.9.69:5044"]

logging.level: debug

【问题讨论】：

你确定它还是坏了吗？ Kibana 在最后 2 条日志行中显示状态 == 绿色...只有先前的状态/消息是红色的，这对于启动例程来说是正常的。当您尝试在网络浏览器中导航到 container_host:5601 时会发生什么？
是的，我可以看到 kibana 在网络浏览器中运行。问题是当我将日志（使用 filebeat）转发到 logstash 时，它们没有被索引。我不知道如何得到它
好的，明白了。我看到你有一个节拍输入——节拍客户端在哪里运行？你能分享它的配置和日志吗？
好的，谢谢大家的支持和帮助。我将我的 elk docker 镜像版本升级到 6.3.0（最新），一切看起来都不错。
好的，谢谢大家的支持和帮助。我将我的 elk docker 映像版本升级到 6.3.0（最新），一切看起来都很好。但是，我的弹性搜索没有创建索引。我可以看到 logstash 从 filebeat 接收日志，但它以某种方式没有转发到 elasticsearch。我在我的帖子中包含了 filebeat.yml。

标签： elasticsearch logstash kibana elastic-stack elasticsearch-plugin

【解决方案1】：

看起来这是您的日志记录中的弹性搜索问题，导致 ES 无法初始化。这一行：

elasticsearch_1  | [2018-06-22T19:31:38,776][WARN ][o.e.b.BootstrapChecks    ] [g8HPieb] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

您可以使用以下命令暂时将其提升：

sysctl -w vm.max_map_count=262144

或者，如果您在实时实例上，则将以下行添加到 /etc/sysctl.conf 并运行 sysctl -p 以获取配置来永久设置它：

vm.max_map_count=262144

由于您在 Docker 容器中执行此操作，您可能只想在 /etc/sysctl.conf 中使用后一个选项。

参考：https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

【讨论】：

嗨，保罗，感谢您的输入，但不幸的是，它不起作用。同样的错误
@LifeIsButiool 你能粘贴最近运行的日志吗？现在应该有点不同了，因为如果 max_map_count 太低，ES 将拒绝启动。还是您仍然收到该消息？