带有副本集的 mongodb 的 Statefulset答案

【问题标题】：Statefulset for mongodb with replica set带有副本集的 mongodb 的 Statefulset
【发布时间】：2020-03-14 14:29:02
【问题描述】：

我想使用副本集 (mongo) 从 mongo 映像创建 Statefulset。

为此，我有我的 statefulset：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-test
spec:
  selector:
    matchLabels:
      app: mongo-test
  serviceName: "mongo-test"
  replicas: 1
  template:
    metadata:
      labels:
        app: mongo-test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongodb
        image: mongo:4.2.1
        ports:
        - containerPort: 27017
          name: web
        command: ["/bin/sh","-c"]
        args: ["mongod --oplogSize 128 --replSet rs0 --bind_ip_all && mongo --eval \"rs.initiate()\""]
        volumeMounts:
        - name: mongo-persistent-storage
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi

Statefulset 和创建的 Pod 都可以，但是副本集没有初始化。我正在发送命令：mongo --eval "rs.initiate()" 以创建副本，但由于某种原因未创建副本。

我还使用了命令：sleep 30 before the rs.initiate() 命令就像延迟但没有运气。

我的命令有问题吗？

钯。我没有详细说明其余的配置（服务、pvc、pv），因为一切正常。

-- 更新

添加日志：

2019-11-19T15:10:05.257+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=8 port=27017 dbpath=/data/db 64-bit host=mongo-test-0
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] db version v4.2.1
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] git version: edf6d45851c0b9ee15548f0f847df141764a317e
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] modules: none
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] build environment:
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten]     distmod: ubuntu1804
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2019-11-19T15:10:05.259+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: "*" }, replication: { oplogSizeMB: 128, replSet: "rs0" } }
2019-11-19T15:10:05.259+0000 I  STORAGE  [initandlisten] 
2019-11-19T15:10:05.259+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-11-19T15:10:05.259+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-11-19T15:10:05.259+0000 I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=5442M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2019-11-19T15:10:06.307+0000 I  STORAGE  [initandlisten] WiredTiger message [1574176206:307928][8:0x7fa43d4aeb00], txn-recover: Set global recovery timestamp: (0,0)
2019-11-19T15:10:06.721+0000 I  RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-11-19T15:10:07.142+0000 I  STORAGE  [initandlisten] Timestamp monitor starting
2019-11-19T15:10:07.272+0000 I  CONTROL  [initandlisten] 
2019-11-19T15:10:07.272+0000 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-11-19T15:10:07.272+0000 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-11-19T15:10:07.272+0000 I  CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-11-19T15:10:07.272+0000 I  CONTROL  [initandlisten] 
2019-11-19T15:10:07.273+0000 I  SHARDING [initandlisten] Marking collection local.system.replset as collection version: <unsharded>
2019-11-19T15:10:07.273+0000 I  STORAGE  [initandlisten] Flow Control is enabled on this deployment.
2019-11-19T15:10:07.273+0000 I  SHARDING [initandlisten] Marking collection admin.system.roles as collection version: <unsharded>
2019-11-19T15:10:07.273+0000 I  SHARDING [initandlisten] Marking collection admin.system.version as collection version: <unsharded>
2019-11-19T15:10:07.273+0000 I  STORAGE  [initandlisten] createCollection: local.startup_log with generated UUID: acb2882f-a531-456a-8150-9d4eb0273ef1 and options: { capped: true, size: 10485760 }
2019-11-19T15:10:07.532+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.startup_log
2019-11-19T15:10:07.532+0000 I  SHARDING [initandlisten] Marking collection local.startup_log as collection version: <unsharded>
2019-11-19T15:10:07.532+0000 I  FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2019-11-19T15:10:07.533+0000 I  STORAGE  [initandlisten] createCollection: local.replset.oplogTruncateAfterPoint with generated UUID: 6c30e1dc-7da5-4698-a96e-d6736e450999 and options: {}
2019-11-19T15:10:07.773+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.replset.oplogTruncateAfterPoint
2019-11-19T15:10:07.774+0000 I  STORAGE  [initandlisten] createCollection: local.replset.minvalid with generated UUID: 703fef2a-479a-4ae9-9052-6c04cf942af4 and options: {}
2019-11-19T15:10:07.997+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.replset.minvalid
2019-11-19T15:10:07.997+0000 I  SHARDING [initandlisten] Marking collection local.replset.minvalid as collection version: <unsharded>
2019-11-19T15:10:07.998+0000 I  STORAGE  [initandlisten] createCollection: local.replset.election with generated UUID: 950b0919-e330-4983-a1ff-537fb0e28d72 and options: {}
2019-11-19T15:10:08.153+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.replset.election
2019-11-19T15:10:08.153+0000 I  SHARDING [ftdc] Marking collection local.oplog.rs as collection version: <unsharded>
2019-11-19T15:10:08.153+0000 W  REPL     [ftdc] Rollback ID is not initialized yet.
2019-11-19T15:10:08.154+0000 I  SHARDING [initandlisten] Marking collection local.replset.election as collection version: <unsharded>
2019-11-19T15:10:08.155+0000 I  REPL     [initandlisten] Did not find local initialized voted for document at startup.
2019-11-19T15:10:08.155+0000 I  REPL     [initandlisten] Did not find local Rollback ID document at startup. Creating one.
2019-11-19T15:10:08.156+0000 I  STORAGE  [initandlisten] createCollection: local.system.rollback.id with generated UUID: 9e2939db-087f-40a9-9710-585b4b26c57b and options: {}
2019-11-19T15:10:08.330+0000 I  INDEX    [initandlisten] index build: done building index _id_ on ns local.system.rollback.id
2019-11-19T15:10:08.330+0000 I  SHARDING [initandlisten] Marking collection local.system.rollback.id as collection version: <unsharded>
2019-11-19T15:10:08.331+0000 I  REPL     [initandlisten] Initialized the rollback ID to 1
2019-11-19T15:10:08.331+0000 I  REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2019-11-19T15:10:08.332+0000 I  CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2019-11-19T15:10:08.333+0000 I  NETWORK  [initandlisten] Listening on /tmp/mongodb-27017.sock
2019-11-19T15:10:08.333+0000 I  NETWORK  [initandlisten] Listening on 0.0.0.0
2019-11-19T15:10:08.333+0000 I  NETWORK  [initandlisten] waiting for connections on port 27017
2019-11-19T15:10:08.339+0000 I  SHARDING [LogicalSessionCacheReap] Marking collection config.system.sessions as collection version: <unsharded>
2019-11-19T15:10:08.339+0000 I  CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist

【问题讨论】：

什么是 rs.initiate()？
@suren 这是一个启动mongo副本集的命令。
您可以试试：args: ["mongod --oplogSize 128 --replSet rs0 --bind_ip_all && mongo localhost:27017 --eval \"rs.initiate()\"" ]
@Hanx 没用

标签： mongodb kubernetes minikube replicaset

【解决方案1】：

你可以试试这个

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
  app: mongodb
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-volume
              mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

【讨论】：

谢谢，但是启动 mongo 副本集的命令呢？我不想要 mongo 的版本 3，我想要版本 4.2.1。
已更新答案，请立即查看，如果不起作用，请告诉我。
它不起作用，因为无法读取 --smallfiles 选项。 Kubernetes 将每一行作为不同的命令。

【解决方案2】：

我不是这方面的专家，但根据您的日志，您的选项在启动期间已通过：

options: { net: { bindIp: "*" }, replication: { oplogSizeMB: 128, replSet: "rs0" } }

但是mongodb在启动过程中找不到repl设置：

Did not find local replica set configuration document at startup;
  NoMatchingDocument: Did not find replica set configuration document in local.system.replset

附加参数的第二个问题：

从 4.2 版开始，MongoDB 删除了已弃用的 MMAPv1 存储引擎和特定于 MMAPv1 的配置选项：

删除命令行选项：

mongod --smallfiles

mongod --noprealloc

我做了什么：我已经申请了以下参数：

args:
  - mongod
  - "--replSet"
  - rs0
  - "--bind_ip_all"

对于主机名、稳定网络标识符，请参考 statefulset 基础知识：

对于有 N 个副本的 StatefulSet，StatefulSet 中的每个 Pod 都将被分配一个从 0 到 N-1 的整数序号，该序号在 Set 中是唯一的 StatefulSet 可以使用 Headless Service 来控制其 Pod 的域。此 Service 管理的域采用以下形式：$(service name).$(namespace).svc.cluster.local，其中“cluster.local”是集群域。在创建每个 Pod 时，它会获得一个匹配的 DNS 子域，格式为：$(podname).$(governing service domain)，其中管理服务由 StatefulSet 上的 serviceName 字段定义。

如限制部分所述，您负责创建负责 Pod 网络身份的 Headless Service。

根据我示例中的这些规则，我还创建了无头服务并在 mongodb 初始化期间提供了这些信息：

host: "mongo-test-0.mongo-test"
host: "mongo-test-1.mongo-test"

从正在运行的 pod 中，我使用以下命令初始化了集群：

rs.initiate( {_id: "rs0", version:1, members: [
      { _id: 0, host: "mongo-test-0.mongo-test:27017" },
      { _id: 1, host: "mongo-test-1.mongo-test:27017" },
   ]
})

{
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1574614871, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1574614871, 1)
}

验证：

rs0:PRIMARY> rs.config()
{
    "_id" : "rs0",
    "version" : 1,
    "protocolVersion" : NumberLong(1),
    "writeConcernMajorityJournalDefault" : true,
    "members" : [
        {
            "_id" : 0,
            "host" : "mongo-test-0.mongo-test:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        },
        {
            "_id" : 1,
            "host" : "mongo-test-1.mongo-test:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },

rs0:PRIMARY> rs.isMaster()
{
    "hosts" : [
        "mongo-test-0.mongo-test:27017",
        "mongo-test-1.mongo-test:27017"
    ],
    "setName" : "rs0",
    "setVersion" : 1,
    "ismaster" : true,

更多关于mongodb Replication的信息请参考官方mongodb docs。

在我看来，您应该准备 env 变量和附加脚本/入口点应用到您的部署中，以便管理 f.e.正确的主机名、稳定的网络标识符、复制和初始化。

举个例子，我建议使用helm charts for mongodb。
希望对您有所帮助。

【讨论】：

谢谢@Hanx。正如您在我的问题中看到的那样，我没有使用 smallfiles 或 prealloc。因此，这不是问题。是的，正如你所说，我正在创建一个包含所有初始化脚本的图像。
如上所述，您可能应该在脚本中以环境变量的形式提供信息作为示例。每个成员都应根据运行配置验证这些参数。此外，请使用稳定的 mongo 图表。