【发布时间】：2023-04-03 15:35:02
【问题描述】：
在我的 docker 环境中，弹性搜索（ES-1）暴露在端口 9200 上，这是维护数据库数据的主要 ES，所以为了记录我们正在使用运行新弹性搜索的 fluentd-elasticsearch-kibna 堆栈在端口 9700(ES-2) 上。
我无法将 fluentd 连接到这个新的弹性搜索（ES-2）到端口 9700。让我知道如何将 fluentd 连接到端口“9200”以外的 es
以下是用于 F-E-K 的 docker 配置详细信息
fluentd.config :
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match *.**>
@type copy
<store>
@type elasticsearch
host elasticsearchkibana
port 9200
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
</store>
<store>
@type stdout
</store>
</match>
docker-compose.yml :
version: '2'
services:
web:
image: httpd
ports:
- "80:80"
links:
- fluentd
logging:
driver: "fluentd"
options:
fluentd-address: localhost:24224
tag: httpd.access
elasticsearchkibana:
image: elasticsearch
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
expose:
- 9800
ports:
- "9700:9300"
- "9800:9200"
fluentd:
build: ./fluentd
volumes:
- ./fluentd/conf:/fluentd/etc
links:
- "elasticsearchkibana"
ports:
- "24224:24224"
- "24224:24224/udp"
kibana:
image: kibana
environment:
- ELASTICSEARCH_URL=http://elasticsearchkibana:9200
# - "ES_HOST=192.168.241.82"
# - "ES_PORT=9700"
ports:
- "5601:5601"
以下是我使用上述配置时收到的错误消息
fluentd_1 | 2017-09-25 15:16:36 +0000 fluent.warn: {"next_retry":"2017-09-25 15:16:37 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:37 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:16:37 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:39 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:16:37 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:16:37 +0000 fluent.warn: {"next_retry":"2017-09-25 15:16:39 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:39 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:16:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:43 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:16:39 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:16:39 +0000 fluent.warn: {"next_retry":"2017-09-25 15:16:43 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:43 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:16:43 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:50 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:16:43 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:16:43 +0000 fluent.warn: {"next_retry":"2017-09-25 15:16:50 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:16:50 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:16:50 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:17:04 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:16:50 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:16:50 +0000 fluent.warn: {"next_retry":"2017-09-25 15:17:04 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:17:04 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:17:04 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:17:39 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:17:04 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:17:04 +0000 fluent.warn: {"next_retry":"2017-09-25 15:17:39 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:17:39 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:17:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:18:46 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:17:39 +0000 fluent.warn: {"next_retry":"2017-09-25 15:18:46 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:18:46 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
fluentd_1 | 2017-09-25 15:17:39 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:18:46 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-25 15:20:49 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!" plugin_id="object:2af57e68ced4"
fluentd_1 | 2017-09-25 15:18:46 +0000 [warn]: suppressed same stacktrace
fluentd_1 | 2017-09-25 15:18:46 +0000 fluent.warn: {"next_retry":"2017-09-25 15:20:49 +0000","error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"Can not reach Elasticsearch cluster ({:host=>\"localhost\", :port=>9700, :scheme=>\"http\"})!","plugin_id":"object:2af57e68ced4","message":"temporarily failed to flush the buffer. next_retry=2017-09-25 15:20:49 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"Can not reach Elasticsearch cluster ({:host=>\\\"localhost\\\", :port=>9700, :scheme=>\\\"http\\\"})!\" plugin_id=\"object:2af57e68ced4\""}
root@docker-test:/home/tlirlogin/docker-compose-efk-master2# cd fluentd/conf/
谢谢
【问题讨论】：
-
您的 fluentd 在容器网络中运行，而不是在主机中运行。因此，即使您在主机上映射到 9700，端口仍需要为 9200。因为这种情况下连接依赖于容器端口
-
@tarun Laiwani 更改 fluentd_1 fluent.warn: error_class":"Fluent::ElasticsearchOutput::ConnectionFailure","error":"无法访问 Elasticsearch 集群 ({:host=> \"elasticsearchkibana\", :port=>9200, :scheme=>\"http\"})!","plugin_id":"object:2ac5be462aa4","message":"暂时无法刷新缓冲区。 next_retry=2017-09-26 05:14:24 +0000 error_class=\"Fluent::ElasticsearchOutput::ConnectionFailure\" error=\"无法访问Elasticsearch集群
-
你能分享流利的dockerfile吗？所以我可以试一下你的撰写文件
-
@TarunLalwani 嗨，我用新配置编辑了问题，请找到..谢谢
-
我不是在找配置，我需要你用于 fluentd 构建的 Dockerfile
标签： docker elasticsearch kibana fluentd