[...] 上超过了高磁盘水印 [90%]，分片将从该节点重新定位

【问题标题】：High disk watermark [90%] exceeded on [...], shards will be relocated away from this node[...] 上超过了高磁盘水印 [90%]，分片将从该节点重新定位
【发布时间】：2020-11-09 10:21:49
【问题描述】：

我在 kubernetes 集群（v1.18 - kind）上运行 Elasicsearch (floragunncom/sg-elasticsearch:7.0.1-oss-35.0.0)。

我已经为每个 Elasticsearch 数据节点（即 Pod）分配了 PVC 的 200Mi 作为存储。但是当 elasticseach 集群（3 master, 3 data, 2 ingest）启动时，它会显示以下警告消息：

{"type": "server", "timestamp": "2020-07-20T06:27:14,354+0000", "level": "WARN", "component": "oecraDiskThresholdMonitor", "cluster.名称”：“拓扑-es”，“node.name”：“master-topology-es-1”，“cluster.uuid”：“U0Yf6tiJSHC57R_QHa2qiw”，“node.id”：“tDet2llLQaydoxfYT4TcpQ”，“消息”：“在 [C-TUE5_cR1ihqrhThpWnsg][data-topology-es-1][/usr/share/elasticsearch/data/nodes/0] 上超过高磁盘水印 [90%] 免费：5.5gb [7%]，分片将被重新定位远离这个节点”}

  • 5.5gb[7%] 来自哪里？

我还检查了分片的状态：

$ curl -XGET "https://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason" -u "admin:XXXXX" --insecure
searchguard 0 p STARTED    
searchguard 0 r UNASSIGNED REPLICA_ADDED
searchguard 0 r UNASSIGNED REPLICA_ADDED
  • 如果200Mi还不够，那么data-node-0中怎么分配主shard成功，而data-node-1和data-node-2中没有replica呢？

更新：

$ curl -XGET "https://localhost:9200/_nodes/stats/fs?pretty" -u "admin:XXXX" --insecure
{
  "_nodes" : {
    "total" : 8,
    "successful" : 8,
    "failed" : 0
  },
  "cluster_name" : "topology-es",
  "nodes" : {
    "jYkduRCmQ2GEHMn2DeBrxg" : {
      "timestamp" : 1595231421976,
      "name" : "data-topology-es-0",
      "transport_address" : "10.244.0.59:9300",
      "host" : "10.244.0.59",
      "ip" : "10.244.0.59:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231421989,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77306,
              "read_operations" : 7745,
              "write_operations" : 69561,
              "read_kilobytes" : 230532,
              "write_kilobytes" : 500284
            }
          ],
          "total" : {
            "operations" : 77306,
            "read_operations" : 7745,
            "write_operations" : 69561,
            "read_kilobytes" : 230532,
            "write_kilobytes" : 500284
          }
        }
      }
    },
    "N7ql_AneQoa6w8yvgBhYZg" : {
      "timestamp" : 1595231421977,
      "name" : "client-topology-es-0",
      "transport_address" : "10.244.0.49:9300",
      "host" : "10.244.0.49",
      "ip" : "10.244.0.49:9300",
      "roles" : [
        "ingest"
      ],
      "fs" : {
        "timestamp" : 1595231421981,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 84052,
              "read_operations" : 7853,
              "write_operations" : 76199,
              "read_kilobytes" : 231804,
              "write_kilobytes" : 565340
            }
          ],
          "total" : {
            "operations" : 84052,
            "read_operations" : 7853,
            "write_operations" : 76199,
            "read_kilobytes" : 231804,
            "write_kilobytes" : 565340
          }
        }
      }
    },
    "gse01W_XR7qKvZPHPq7n7g" : {
      "timestamp" : 1595231421984,
      "name" : "master-topology-es-2",
      "transport_address" : "10.244.0.57:9300",
      "host" : "10.244.0.57",
      "ip" : "10.244.0.57:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231421993,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77627,
              "read_operations" : 7747,
              "write_operations" : 69880,
              "read_kilobytes" : 230560,
              "write_kilobytes" : 504608
            }
          ],
          "total" : {
            "operations" : 77627,
            "read_operations" : 7747,
            "write_operations" : 69880,
            "read_kilobytes" : 230560,
            "write_kilobytes" : 504608
          }
        }
      }
    },
    "bBNEuz2pSkWeOUmDIowEDQ" : {
      "timestamp" : 1595231421991,
      "name" : "client-topology-es-1",
      "transport_address" : "10.244.0.51:9300",
      "host" : "10.244.0.51",
      "ip" : "10.244.0.51:9300",
      "roles" : [
        "ingest"
      ],
      "fs" : {
        "timestamp" : 1595231421995,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 82173,
              "read_operations" : 7788,
              "write_operations" : 74385,
              "read_kilobytes" : 231144,
              "write_kilobytes" : 546544
            }
          ],
          "total" : {
            "operations" : 82173,
            "read_operations" : 7788,
            "write_operations" : 74385,
            "read_kilobytes" : 231144,
            "write_kilobytes" : 546544
          }
        }
      }
    },
    "-hwsv3rqQeSVdfOsKqzrwA" : {
      "timestamp" : 1595231422013,
      "name" : "master-topology-es-0",
      "transport_address" : "10.244.0.53:9300",
      "host" : "10.244.0.53",
      "ip" : "10.244.0.53:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231422080,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 79542,
              "read_operations" : 7762,
              "write_operations" : 71780,
              "read_kilobytes" : 230792,
              "write_kilobytes" : 521656
            }
          ],
          "total" : {
            "operations" : 79542,
            "read_operations" : 7762,
            "write_operations" : 71780,
            "read_kilobytes" : 230792,
            "write_kilobytes" : 521656
          }
        }
      }
    },
    "C-TUE5_cR1ihqrhThpWnsg" : {
      "timestamp" : 1595231421984,
      "name" : "data-topology-es-1",
      "transport_address" : "10.244.0.61:9300",
      "host" : "10.244.0.61",
      "ip" : "10.244.0.61:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231422003,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77169,
              "read_operations" : 7736,
              "write_operations" : 69433,
              "read_kilobytes" : 230396,
              "write_kilobytes" : 498544
            }
          ],
          "total" : {
            "operations" : 77169,
            "read_operations" : 7736,
            "write_operations" : 69433,
            "read_kilobytes" : 230396,
            "write_kilobytes" : 498544
          }
        }
      }
    },
    "zlL7F5kRTLWWogzffZXomA" : {
      "timestamp" : 1595231422000,
      "name" : "data-topology-es-2",
      "transport_address" : "10.244.0.63:9300",
      "host" : "10.244.0.63",
      "ip" : "10.244.0.63:9300",
      "roles" : [
        "data"
      ],
      "fs" : {
        "timestamp" : 1595231422001,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77048,
              "read_operations" : 7736,
              "write_operations" : 69312,
              "read_kilobytes" : 230396,
              "write_kilobytes" : 496864
            }
          ],
          "total" : {
            "operations" : 77048,
            "read_operations" : 7736,
            "write_operations" : 69312,
            "read_kilobytes" : 230396,
            "write_kilobytes" : 496864
          }
        }
      }
    },
    "tDet2llLQaydoxfYT4TcpQ" : {
      "timestamp" : 1595231421993,
      "name" : "master-topology-es-1",
      "transport_address" : "10.244.0.55:9300",
      "host" : "10.244.0.55",
      "ip" : "10.244.0.55:9300",
      "roles" : [
        "master"
      ],
      "fs" : {
        "timestamp" : 1595231422020,
        "total" : {
          "total_in_bytes" : 84054204416,
          "free_in_bytes" : 10231291904,
          "available_in_bytes" : 5917450240
        },
        "data" : [
          {
            "path" : "/usr/share/elasticsearch/data/nodes/0",
            "mount" : "/usr/share/elasticsearch/data (/dev/sda3)",
            "type" : "ext4",
            "total_in_bytes" : 84054204416,
            "free_in_bytes" : 10231291904,
            "available_in_bytes" : 5917450240
          }
        ],
        "io_stats" : {
          "devices" : [
            {
              "device_name" : "sda3",
              "operations" : 77766,
              "read_operations" : 7748,
              "write_operations" : 70018,
              "read_kilobytes" : 230576,
              "write_kilobytes" : 505892
            }
          ],
          "total" : {
            "operations" : 77766,
            "read_operations" : 7748,
            "write_operations" : 70018,
            "read_kilobytes" : 230576,
            "write_kilobytes" : 505892
          }
        }
      }
    }
  }
}

【问题讨论】：

  • 运行这个GET _nodes/stats/fs会得到什么？
  • @Val，更新了我的问题。请看一看。
  • @KamolHasan 很长时间以来，您都没有接受并赞成我的回答，如果您的问题得到解决，如果您能接受并赞成答案，那就太好了
  • @opsterelasticsearchninja，非常感谢您的回答。这很有帮助。因此我投了赞成票。但这并没有解决我的问题。我在另一个答案中添加了原因。
  • @KamolHasan 哦，好的，谢谢，很高兴我有帮助

标签： elasticsearch kubernetes search-guard


【解决方案1】：

所以，这是我的问题：我已通过 PVC 将 200Mi 的存储分配给每个 Elasticsearch 数据节点。当我检查_nodes/stats：

 "total" : {
      "total_in_bytes" : 84054204416, <-- ~84Gi
      "free_in_bytes" : 10231291904,
      "available_in_bytes" : 5917450240
    },

因此，预期值应该是 200Mi 而不是 ~84Gi。

说明：

因为我在kind 集群中。默认存储由hostpath 提供，我们从主机上安装目录，而不是单独的文件系统。

而kind使用rancher/local-path-provisioner，目前使用的限制是：

- No support for the volume capacity limit currently.
    - The capacity limit will be ignored for now.

这就是为什么我得到~84Gi（主机）而不是200Mi（通过PVC分配）。

【讨论】：

    【解决方案2】：

    这是主节点C-TUE5_cR1ihqrhThpWnsg 的文件系统统计信息，取自_nodes/stats 输出：

        "total" : {
      "total_in_bytes" : 84054204416,
      "free_in_bytes" : 10231291904,
      "available_in_bytes" : 5917450240
    },

    您可以看到 5917450240 / 84054204416 = ~7%，因此该节点上的存储空间已满 93%，因此您会收到该警告。

    【讨论】：

    • 谢谢，所以这与通过 PVC 为 pod 分配多少存储空间（此处为 200Mi）无关？
    • 这里的数据存储在 sda3 上，显然有 85GB 的存储空间。此外，从数字来看，您的所有节点似乎都共享同一个磁盘
    猜你喜欢
    • 2021-08-17
    • 2016-01-26
    • 1970-01-01
    • 2016-07-28
    • 1970-01-01
    • 2015-07-29
    • 2021-12-04
    • 2023-01-15
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode