一、理解Cluster Map
cluster map由monitor维护，用于跟踪ceph集群状态
当client启动时，会连接monitor获取cluster map副本，发现所有其他组件的位置，然后直接与所需的进程通信，以存储和检索数据
monitor跟踪这些集群组件的状态，也负责管理守护进程和客户端之间的身份验证
cluster map实际是多种map的集群，包含：monitor map、osd map、pg map、mds map、mgr map、service map
1.1 Clisuter Map内容
monitor map：包含集群ID、monitor节点名称、IP以及端口号以及monitor map的版本号
OSD map：包含集群ID、自身的版本号以及存储池相关的信息，包括存储池名称、ID、类型、副本级别和PG。还包括OSD的信息，比如数量、状态、权限以及OSD节点信息
PG map：包含PG的版本、时间戳、OSD map的最新版本号、容量相关的百分比。还记录了每个PG的ID、对象数量、状态、状态时间戳等
MDS map：包含MDS的地址、状态、数据池和元数据池的ID
MGR map：包含MGR的地址和状态，以及可用和已启用模块的列表
service map：跟踪通过librados部署的一些服务的实例，如RGW、rbd-mirror等。service map收集这些服务的信息然后提供给其他服务，如MGR的dashboard插件使用该map报告这些客户端服务的状态
1.2 Cluster Map基本查询
查询mon map
[root@ceph2 ~]# ceph mon dump dumped monmap epoch 1 epoch 1 fsid 35a91e48-8244-4e96-a7ee-980ab989d20d last_changed 2019-03-16 12:39:14.839999 created 2019-03-16 12:39:14.839999 0: 172.25.250.11:6789/0 mon.ceph2 1: 172.25.250.12:6789/0 mon.ceph3 2: 172.25.250.13:6789/0 mon.ceph4
查询osd map
[root@ceph2 ~]# ceph osd dump epoch 281 fsid 35a91e48-8244-4e96-a7ee-980ab989d20d created 2019-03-16 12:39:22.552356 modified 2019-03-26 22:32:15.354383 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 43 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release luminous pool 1 'testpool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 190 flags hashpspool stripe_width 0 application rbd snap 1 'testpool-snap-20190316' 2019-03-16 22:27:34.150433 snap 2 'testpool-snap-2' 2019-03-16 22:31:15.430823 pool 5 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 191 flags hashpspool stripe_width 0 application rbd removed_snaps [1~13] pool 6 'rbdmirror' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 192 flags hashpspool stripe_width 0 application rbd removed_snaps [1~7] pool 7 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 176 flags hashpspool stripe_width 0 application rgw pool 8 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 178 flags hashpspool stripe_width 0 application rgw pool 9 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 180 flags hashpspool stripe_width 0 application rgw pool 10 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 182 flags hashpspool stripe_width 0 application rgw pool 11 'xiantao.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 194 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 12 'xiantao.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 196 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 13 'xiantao.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 198 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 14 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 214 flags hashpspool stripe_width 0 application cephfs pool 15 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 214 flags hashpspool stripe_width 0 application cephfs pool 16 'test' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 32 pgp_num 32 last_change 280 flags hashpspool stripe_width 0 application rbd pool 17 'ssdpool' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 32 pgp_num 32 last_change 281 flags hashpspool stripe_width 0 application rbd max_osd 9 osd.0 up in weight 1 up_from 54 up_thru 264 down_at 53 last_clean_interval [7,51) 172.25.250.11:6800/185671 172.25.250.11:6801/185671 172.25.250.11:6802/185671 172.25.250.11:6803/185671 exists,up 745dce53-1c63-4c50-b434-d441038dafe4 osd.1 up in weight 1 up_from 187 up_thru 258 down_at 184 last_clean_interval [54,186) 172.25.250.13:6809/60269 172.25.250.13:6807/1060269 172.25.250.13:6803/1060269 172.25.250.13:6813/1060269 exists,up a7562276-6dfd-4803-b248-a7cbdb64ebec osd.2 up in weight 1 up_from 258 up_thru 264 down_at 257 last_clean_interval [54,257) 172.25.250.12:6804/59201 172.25.250.12:6810/8059201 172.25.250.12:6811/8059201 172.25.250.12:6815/8059201 exists,up bbef1a00-3a31-48a0-a065-3a16b9edc3b1 osd.3 up in weight 1 up_from 54 up_thru 272 down_at 53 last_clean_interval [13,51) 172.25.250.11:6804/185668 172.25.250.11:6805/185668 172.25.250.11:6806/185668 172.25.250.11:6807/185668 exists,up e934a4fb-7125-4e85-895c-f66cc5534ceb osd.4 up in weight 1 up_from 187 up_thru 267 down_at 184 last_clean_interval [54,186) 172.25.250.13:6805/60272 172.25.250.13:6802/1060272 172.25.250.13:6810/1060272 172.25.250.13:6811/1060272 exists,up e2c33bb3-02d2-4cce-85e8-25c419351673 osd.5 up in weight 1 up_from 261 up_thru 275 down_at 257 last_clean_interval [54,258) 172.25.250.12:6808/59198 172.25.250.12:6806/8059198 172.25.250.12:6807/8059198 172.25.250.12:6814/8059198 exists,up d299e33c-0c24-4cd9-a37a-a6fcd420a529 osd.6 up in weight 1 up_from 54 up_thru 273 down_at 52 last_clean_interval [21,51) 172.25.250.11:6808/185841 172.25.250.11:6809/185841 172.25.250.11:6810/185841 172.25.250.11:6811/185841 exists,up debe7f4e-656b-48e2-a0b2-bdd8613afcc4 osd.7 up in weight 1 up_from 187 up_thru 266 down_at 184 last_clean_interval [54,186) 172.25.250.13:6801/60271 172.25.250.13:6806/1060271 172.25.250.13:6808/1060271 172.25.250.13:6812/1060271 exists,up 8c403679-7530-48d0-812b-72050ad43aae osd.8 up in weight 1 up_from 151 up_thru 265 down_at 145 last_clean_interval [54,150) 172.25.250.12:6800/59200 172.25.250.12:6801/7059200 172.25.250.12:6802/7059200 172.25.250.12:6805/7059200 exists,up bb73edf8-ca97-40c3-a727-d5fde1a9d1d9
查询osd crush map
[root@ceph2 ~]# ceph osd crush dump { "devices": [ { "id": 0, "name": "osd.0", "class": "hdd" }, { "id": 1, "name": "osd.1", "class": "hdd" }, { "id": 2, "name": "osd.2", "class": "hdd" }, { "id": 3, "name": "osd.3", "class": "hdd" }, { "id": 4, "name": "osd.4", "class": "hdd" }, { "id": 5, "name": "osd.5", "class": "hdd" }, { "id": 6, "name": "osd.6", "class": "hdd" }, { "id": 7, "name": "osd.7", "class": "hdd" }, { "id": 8, "name": "osd.8", "class": "hdd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "chassis" }, { "type_id": 3, "name": "rack" }, { "type_id": 4, "name": "row" }, { "type_id": 5, "name": "pdu" }, { "type_id": 6, "name": "pod" }, { "type_id": 7, "name": "room" }, { "type_id": 8, "name": "datacenter" }, { "type_id": 9, "name": "region" }, { "type_id": 10, "name": "root" }, { "type_id": 11, "name": "aaa" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 10, "type_name": "root", "weight": 8649, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 2883, "pos": 0 }, { "id": -5, "weight": 2883, "pos": 1 }, { "id": -7, "weight": 2883, "pos": 2 } ] }, { "id": -2, "name": "default~hdd", "type_id": 10, "type_name": "root", "weight": 5898, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 1966, "pos": 0 }, { "id": -6, "weight": 1966, "pos": 1 }, { "id": -8, "weight": 1966, "pos": 2 } ] }, { "id": -3, "name": "ceph2", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 3, "weight": 983, "pos": 0 }, { "id": 6, "weight": 983, "pos": 1 } ] }, { "id": -4, "name": "ceph2~hdd", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 3, "weight": 983, "pos": 0 }, { "id": 6, "weight": 983, "pos": 1 } ] }, { "id": -5, "name": "ceph4", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 983, "pos": 0 }, { "id": 7, "weight": 983, "pos": 1 } ] }, { "id": -6, "name": "ceph4~hdd", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 983, "pos": 0 }, { "id": 7, "weight": 983, "pos": 1 } ] }, { "id": -7, "name": "ceph3", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 983, "pos": 0 }, { "id": 8, "weight": 983, "pos": 1 } ] }, { "id": -8, "name": "ceph3~hdd", "type_id": 1, "type_name": "host", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 983, "pos": 0 }, { "id": 8, "weight": 983, "pos": 1 } ] }, { "id": -9, "name": "dc1", "type_id": 10, "type_name": "root", "weight": 11796, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -10, "weight": 2949, "pos": 0 }, { "id": -11, "weight": 8847, "pos": 1 }, { "id": -12, "weight": 0, "pos": 2 } ] }, { "id": -10, "name": "rack1", "type_id": 3, "type_name": "rack", "weight": 2949, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 2949, "pos": 0 } ] }, { "id": -11, "name": "rack2", "type_id": 3, "type_name": "rack", "weight": 2949, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -7, "weight": 2949, "pos": 0 } ] }, { "id": -12, "name": "rack3", "type_id": 3, "type_name": "rack", "weight": 2949, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -5, "weight": 2949, "pos": 0 } ] }, { "id": -13, "name": "rack3~hdd", "type_id": 3, "type_name": "rack", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -6, "weight": 1966, "pos": 0 } ] }, { "id": -14, "name": "rack2~hdd", "type_id": 3, "type_name": "rack", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -8, "weight": 1966, "pos": 0 } ] }, { "id": -15, "name": "rack1~hdd", "type_id": 3, "type_name": "rack", "weight": 1966, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 1966, "pos": 0 } ] }, { "id": -16, "name": "dc1~hdd", "type_id": 10, "type_name": "root", "weight": 5898, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -15, "weight": 1966, "pos": 0 }, { "id": -14, "weight": 1966, "pos": 1 }, { "id": -13, "weight": 1966, "pos": 2 } ] }, { "id": -17, "name": "ceph2-ssd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 983, "pos": 0 } ] }, { "id": -18, "name": "ceph3-ssd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 983, "pos": 0 } ] }, { "id": -19, "name": "ceph4-ssd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 983, "pos": 0 } ] }, { "id": -20, "name": "ssd-root", "type_id": 10, "type_name": "root", "weight": 2949, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -17, "weight": 983, "pos": 0 }, { "id": -18, "weight": 983, "pos": 1 }, { "id": -19, "weight": 983, "pos": 2 } ] }, { "id": -21, "name": "ceph2-ssd~hdd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 983, "pos": 0 } ] }, { "id": -22, "name": "ssd-root~hdd", "type_id": 10, "type_name": "root", "weight": 2949, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -21, "weight": 983, "pos": 0 }, { "id": -24, "weight": 983, "pos": 1 }, { "id": -23, "weight": 983, "pos": 2 } ] }, { "id": -23, "name": "ceph4-ssd~hdd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 1, "weight": 983, "pos": 0 } ] }, { "id": -24, "name": "ceph3-ssd~hdd", "type_id": 1, "type_name": "host", "weight": 983, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 983, "pos": 0 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated1_rule", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 11, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "indc1", "ruleset": 3, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -9, "item_name": "dc1" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] }, { "rule_id": 4, "rule_name": "ssdrule", "ruleset": 4, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -20, "item_name": "ssd-root" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }, "choose_args": {} }