【问题标题】:Distributed secure MinIO in docker-composedocker-compose 中的分布式安全 MinIO
【发布时间】:2021-10-16 01:12:17
【问题描述】:

我在Docker 中有一个相当复杂的系统。一切都在一个大的docker-compose 文件中运行。以前一切都在我的 Docker Swarm 中的一个(管理器)节点上运行,因此我为我的域(使用certbot)生成了一个 CERT,并且我在我的撰写文件中使用了以下MinIO 服务:

  object_storage:
    image: minio/minio:RELEASE.2020-12-10T01-54-29Z
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
    command: server /data
    depends_on:
      - fluentd
    volumes:
      - object_storage_data:/data
      - ./certs/domain.crt:/root/.minio/certs/public.crt
      - ./certs/domain.key:/root/.minio/certs/private.key
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

上述实现按预期工作!但现在我有 2 个独立的服务器来运行 MinIO。这些服务器作为工作节点加入我的Docker SwarmMinIO 不应该在管理节点上运行(仅在两个独立的工作节点上)!

>>> docker node ls
ID                                          HOSTNAME    STATUS      AVAILABILITY   MANAGER STATUS   ENGINE VERSION
mcbkz9m5nzf7oa3fiqk0lf4qo *  manager         Ready           Active                    Leader                    20.10.1
dz4e3k70g8ik2z4bcx8u0ft9ao   minio_1          Ready           Active                                                   20.10.2
r0qpdn2guyy5773vo8vg2trzo    minio_2          Ready           Active                                                   20.10.2

我当前的MinIO 实现在我的docker-compose 文件中:

object_storage_1:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000:9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_1_1:/data1
     - object_storage_data_1_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_1
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

 object_storage_2:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_2_1:/data1
     - object_storage_data_2_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_2
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

如果我检查我的 MinIO 服务实例的日志,我会收到以下错误:

Unable to read 'format.json' from https://object_storage_1:9000/data1: Post "https://object_storage_1:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_1
Unable to read 'format.json' from https://object_storage_2:9000/data1: Post "https://object_storage_2:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_2

但是我可以在9000端口上到达MinIO,只是弹出错误:

我只想通过我的域 (my_domain.app:9000) 访问 MinIO。在这种情况下,MinIO 不使用真实的服务器名称,而是使用“虚拟”Docker 网络(例如:https://object_storage_2:9000)。

我的问题:

  • 如何为“虚拟”Docker 网络(例如:object_storage_1 或 object_storage_2)生成证书?
  • 我应该把生成的证书放在哪里?
  • 是否可以仅使用我的生成(针对我的域)证书来解决?

我愿意接受每一个提示和解决方案!

【问题讨论】:

    标签: docker docker-compose certificate minio object-storage


    【解决方案1】:

    我不得不将(域)CERT 文件放到 minio/certs/CAs 文件夹而不是 /root/.minio/certs 文件夹中。此外,如果我没有将 CERT 复制到服务在工作节点上找不到它的节点,我必须将其复制到工作节点(分离的服务器)。

    正确的volumes参数如下:

    volumes:
      - object_storage_data_1_1:/data1
      - object_storage_data_1_2:/data2
      - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
    

    我的几个 MinIO 服务中的一个工作服务:

      object-storage-1:
        image: minio/minio:RELEASE.2021-08-17T20-53-08Z
        expose:
          - "9000"
          - "9001"
        environment:
          MINIO_ACCESS_KEY_FILE: object_storage_user
          MINIO_SECRET_KEY_FILE: object_storage_password
          MINIO_BROWSER_REDIRECT_URL: https://${SYSTEM_HOST}:9001
          MINIO_SERVER_URL: https://${SYSTEM_HOST}:9000
        command: server --console-address ":9001" http://object-storage-{1...4}/data{1...2}
        hostname: object-storage-1
        depends_on:
          - fluentd
        volumes:
          - object_storage_data_1_1:/data1
          - object_storage_data_1_2:/data2
          - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
        networks:
          - object_storage_net
        secrets:
          - object_storage_user
          - object_storage_password
        deploy:
          restart_policy:
            condition: on-failure
          placement:
            constraints:
              - node.hostname == minio1
        logging:
          driver: "fluentd"
          options:
            fluentd-address: ${SYSTEM_HOST}:24224
            tag: object-storage
    

    我必须创建一个NgInx 配置:

    upstream minio {
        server object-storage-1:9000;
        server object-storage-2:9000;
        server object-storage-3:9000;
        server object-storage-4:9000;
    }
    
    upstream console {
        ip_hash;
        server object-storage-1:9001;
        server object-storage-2:9001;
        server object-storage-3:9001;
        server object-storage-4:9001;
    }
    
    server {
        listen              9000 ssl;
        listen              [::]:9000 ssl;
        server_name         my.server.com;
        ssl_certificate     /ssl/domain.crt;
        ssl_certificate_key /ssl/domain.key;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;
    
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
    
    
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
    
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
    
            proxy_pass http://minio;
    
        }
    }
    
    server {
        listen              9001 ssl;
        listen              [::]:9001 ssl;
        server_name         my.server.com;
        ssl_certificate     /ssl/domain.crt;
        ssl_certificate_key /ssl/domain.key;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;
    
        # To allow special characters in headers
        ignore_invalid_headers off;
        # Allow any size file to be uploaded.
        # Set to a value such as 1000m; to restrict file size to a specific value
        client_max_body_size 0;
        # To disable buffering
        proxy_buffering off;
    
        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-NginX-Proxy true;
    
            # This is necessary to pass the correct IP to be hashed
            real_ip_header X-Real-IP;
    
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
    
            proxy_pass http://console;
    
        }
    }
    

    【讨论】:

      猜你喜欢
      • 2019-10-30
      • 2023-02-18
      • 2020-12-29
      • 2018-07-03
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2023-03-19
      相关资源
      最近更新 更多