【问题标题】:Running 2 separate ELK stacks on docker在 docker 上运行 2 个单独的 ELK 堆栈
【发布时间】:2021-11-05 00:50:29
【问题描述】:

在我的 AWS EC2 linux 服务器上,我正在运行一个 ELK 堆栈,其中 logstash 正在转换 postgress 数据库并导入 Elasticsearch。此设置当前用于我的开发环境。我们已经到了创建登台环境的地步,因此我们可能还需要一个单独的 ELK 堆栈来进行登台,因为我们不想混合来自 2 个独立数据库(开发和舞台)的数据。

我在 ELK 方面的经验相当少,我检查了一些选项,但没有找到任何解决此问题的方法。

我尝试创建另一个具有不同容器名称和端口的docker-compose 文件。当我运行docker-compose.elastic.dev.yml 时,它通常会创建第一个 ELK 堆栈。然后我运行docker-compose.elastic.stage.yml,但它开始运行recreate 现有的ELK 容器。我曾尝试使用 docker-compose 设置,但到目前为止还没有运气。有什么建议吗?

仅供参考,开发中不包含kibana,因为我们不需要它。

docker-compose.elastic.stage.yml

    version: '3.7'
services:
  elasticsearch-stage:
    container_name: elasticsearch-stage
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    ports:
      - 9400:9200
    environment:
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.cors.allow-methods=OPTIONS,HEAD,GET,POST,PUT,DELETE
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - transport.host=127.0.0.1
      - cluster.name=docker-cluster
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch_data_stage:/usr/share/elasticsearch/data
    networks:
      - api_network
  kibana-stage:
    container_name: kibana-stage
    image: docker.elastic.co/kibana/kibana:7.10.2
    ports:
      - 5601:5601
    networks:
      - api_network
    depends_on:
      - elasticsearch-stage
  logstash-stage:
    container_name: logstash-stage
    ports:
      - 5045:5045
    build:
      dockerfile: Dockerfile.logstash
      context: .
    environment:
      LOGSTASH_JDBC_URL: "jdbc:postgresql://serverip:15433/name"
      LOGSTASH_JDBC_USERNAME: "name"
      LOGSTASH_JDBC_PASSWORD: "password"
      LOGSTASH_ELASTICSEARCH_HOST: "http://elasticsearch-stage:9200"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./offers_template.json:/usr/share/logstash/templates/offers_template.json
      - ./offers_query.sql:/usr/share/logstash/queries/offers_query.sql
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "5"
    networks:
      - api_network
    depends_on:
      - elasticsearch-stage
      - kibana-stage
volumes:
  elasticsearch_data_stage:
networks:
  api_network:
    name: name_api_network_stage

docker-compose.elastic.dev.yml

version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    ports:
      - 9200:9200
    environment:
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.cors.allow-methods=OPTIONS,HEAD,GET,POST,PUT,DELETE
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - transport.host=127.0.0.1
      - cluster.name=docker-cluster
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    networks:
      - api_network
  logstash:
    build:
      dockerfile: Dockerfile.logstash
      context: .
    environment:
      LOGSTASH_JDBC_URL: "jdbc:postgresql://serverip:15432/username"
      LOGSTASH_JDBC_USERNAME: "username"
      LOGSTASH_JDBC_PASSWORD: "password"
      LOGSTASH_ELASTICSEARCH_HOST: "http://elasticsearch:9200"
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./offers_template.json:/usr/share/logstash/templates/offers_template.json
      - ./offers_query.sql:/usr/share/logstash/queries/offers_query.sql
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "5"
    networks:
      - api_network
    depends_on:
      - elasticsearch
volumes:
  elasticsearch_data:
networks:
  api_network:
    name: name_api_network

我还找到了this 文章,似乎是类似/相同的问题,不幸的是主题已关闭,没有得到确认的解决方案。

logstash.conf

input {
    jdbc {
        jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/postgresql.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        lowercase_column_names => false
        schedule => "* * * * *"
        statement_filepath => "/usr/share/logstash/queries/offers_query.sql"
    }
}
filter {
    json {
        source => "name"
        target => "name"
    }
    json {
        source => "description"
        target => "description"
    }
    ...
    ...
}
output {
    elasticsearch {
        hosts => ["${LOGSTASH_ELASTICSEARCH_HOST}"]
        index => "offers"
        document_id => "%{id}"
        manage_template => true
        template_name => "offers"
        template => "/usr/share/logstash/templates/offers_template.json"
        template_overwrite => true
    }
    stdout { codec => json_lines }

}

更新: 我在here 中发现,如果不运行默认的logstash 配置,我需要为logstash 设置XPACK_MONITORING_ENABLED: "false" environment,并且logstash 无法连接到elasticsearch 的错误消失了,但logstash 仍然没有这样做正常处理来自数据库的数据的工作。现在发生的事情是在logstash 日志中,每隔几分钟就会从offers_query.sql 加载纯查询文本。当我输入 elasticsearch_server_ip:9400 时,我得到了这个输出(所以它应该正在运行):

{
  "name" : "30ac276f0846",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "14mxQTP7S32o-rIrjYSsXw",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

【问题讨论】:

    标签: docker elasticsearch logstash elastic-stack


    【解决方案1】:

    据我了解,文件中的服务名称仍然相同,这让 docker-compose up -d 感到困惑。

    您的问题是 docker-compose 文件中的服务命名。

    services:
      elasticsearch
      logstash
    

    在 dev 和 staging compose 上是一样的,因为你没有运行 swarm,所以你需要以下内容: 将 docker-compose 分离到不同的文件夹,以便 docker-compose 可以创建不同的容器名称。

    是的,您不能在主机端口上转发相同的端口

      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
        ports:
          - 9200:9200
    

    一个弹性搜索应该有 9400:9200 或类似的东西。

    【讨论】:

    • 我添加了文件,请注意 dev 文件没有容器名称,当它们启动时,它们会获得默认名称,但 staging 有名称,但无论如何它们都会重新创建 dev 容器..
    • @JMP 感谢您提供的信息,我知道我有多需要它,我只是写信获取信息..
    • 谢谢,容器正在运行,但logstash 无法连接到elasticsearch,因为我猜是端口弄错了。我已将我的帖子编辑为当前版本的暂存撰写文件,dev compose 未更改,请查看。
    • 尝试将logstash环境设置为staging到LOGSTASH_ELASTICSEARCH_HOST: "elasticsearch-stage:9200" 为什么? .. 端口的左侧是主机的端口,因此如果您点击 localhost:9400 您将被转发到容器 elasticsearch-stage:9200 ...您没有更改弹性侦听的端口。
    • 将其更改为elasticsearch-stage:9200,但现在出现此错误[2021-09-09T12:38:47,445][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
    猜你喜欢
    • 1970-01-01
    • 2019-05-09
    • 1970-01-01
    • 2012-08-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2018-08-11
    • 2023-03-14
    相关资源
    最近更新 更多