qiuhom-1874

  前文我们了解了k8s节点污点和pod的对节点污点容忍度相关话题,回顾请参考:https://www.cnblogs.com/qiuhom-1874/p/14255486.html;今天我们来聊一下扩展k8s相关话题;

  k8s上创建资源对象的过程

  我们知道在k8s上,资源是有类型的,不同类型的资源,其定义方式和使用的字段各不相同;用户创建一个资源,实际上就是把k8s抽象的资源做实例化,即把k8s抽象出来的资源,通过资源清单赋值,创建出来的对象就是我们实例化对应类型资源的结果;用户创建一个资源,首先会把请求发送给apiserver,通过apiserver的认证、授权、准入控制以后,对应创建资源的定义就存放在etcd中,控制器通过watch机制监视apiserver上的资源变动,通过对应资源变动事件触发对应类型资源的控制器将对应资源创建出来,并通过控制器内部的和解循环监控着对应资源状态是否和用户定义的期望状态一样,如果发现不一样,内部和解循环就会被触发,对应控制器会向apiserver发起创建资源的请求,将对应资源重建,让对应资源的状态始终满足用户期望的状态;从上述的过程来看,用户创建一个资源分两个步骤,第一步是将对应请求发送给apiserver,通过apiserver把对应资源定义的信息存放在etcd中;第二个步骤是对应资源类型的控制器通过apiserver从etcd中读取对应资源的定义,将其创建出来;对于etcd来说,它本是就一个kv数据库,可以存储任意类型的kv数据,但在在k8s上,apiserver将不同类型的资源定义抽象成不同的资源,使得用户创建对应资源,必须是满足对应类型资源定义的规范,然后将规范的定义存放在etcd中;简单讲apiserver就是把用户存入etcd中的数据做了一层抽象,使得用户不能够随意将任意数据存储到etcd中,存入etcd中的数据必须是满足对应apiserver接口定义的规范;这就好比我们在使用mysql数据库时,必须遵守对应库中的对应表的定义;

  在k8s上创建自定义资源类型

  在k8s上,资源的类型有很多,比如pod,service,PersistentVolume,PersistentVolumeClaim等等,这些都是一些基础的资源类型;我们要创建某种资源,直接使用对应的资源类型,实例化一个对象即可;假如我们要在k8s上创建一个集群,我们是不是可以直接使用某种类型的资源,实例化一个集群对象呢?理论上是可以的,但是前提是对应k8s上有对应类型的资源;有对应类型的资源,用户就可以把对应创建资源的定义存放在etcd中;除了有对应类型的资源,我们还需要有对应的控制器将对应资源创建出来;这样一来对于不同集群或应用来说,其组织方式和逻辑都不同,使用的资源类型和控制器也有所不同;用户要想实例化更高级的资源来,就必须自己手动实现定义其资源类型,将对应资源类型实例化为对象;除此之外必要时还需要自己实现对应资源的控制器;简单讲用户想要实现更高级的资源类型,就必须扩展现有k8s的资源类型和控制器;

  在k8s上扩展资源类型的方式有三种,第一种是crd,crd是k8s内建的资源类型,该类型资源主要用来创建用户自定义资源类型的资源;即通过crd资源,可以将用户自定义资源类型转换为k8s上资源类型;第二种是自定义apiserver;这种方式要比第一种方式要复杂一点,需要用户手动开发程序实现对应功能的apiserver,让其用户创建自定义类型资源能够通过自定义apiserver实现;第三种方式就是修改现有k8sapiserver,让其支持对应用户自定义资源类型;

  自定义控制器

  自定义资源类型我们可以使用crd资源实现,也可以使用自定义apiserver或修改原有apiserver代码实现,但是只有资源类型是不能够让对应自定义类型资源实例化为一个自定义资源对象,只有自定义资源类型,用户创建对应资源类型的资源对象时,只能把对应资源类型的定义信息写入到etcd中,它不能真正的跑起来,要想真正的跑起来,我们还需要一个自定义控制器,专门负责监听对应的资源类型的资源变化,将对应资源实例化为对应k8s上的资源对象;当然不是所有的自定义类型的资源都需要自定义控制器,如果对应自定义类型资源调用了底层基础控制器来管控对应自定义资源,那么对应自定义类型资源就不需要使用自定义控制器;我们知道控制器是k8s上的一个重要组件,它的工作逻辑是注册监听在apiserver上对应类型的资源变动,如果对应资源状态不满足用户期望状态,它就会根据内部的和解循环来请求apiserver将对应类型资源的定义发送给它,然后根据资源定义来重建对应的资源,让其状态始终和用户期望的状态保持一致;自定义控制器也是同样的逻辑,使用自定义控制器的目的也是让对应自定义类型资源能够被自定义控制器监听,一旦对应资源发生变动,它能够将其在k8s上创建出来,并一直保持和用户期望的状态吻合;自定义控制器和自定义资源类型可以分开实现,也可以合并在一起实现,即自定义控制器程序能够自动创建crd资源,让其对应自定义类型资源能够被k8s识别并将其创建出来;具体是分开实现还是合并在一起实现,取决开发自定义控制器程序员;

  crd资源定义帮助

[root@master01 ~]# kubectl explain crd
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

DESCRIPTION:
     CustomResourceDefinition represents a resource that should be exposed on
     the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>

   spec <Object> -required-
     spec describes how the user wants the resources to appear

   status       <Object>
     status indicates the actual state of the CustomResourceDefinition

[root@master01 ~]#

  提示:crd资源是k8s上的标准资源之一,它的定义主要有apiVersion,kind,metadata,spec和status;其中kind类型为CustomResourceDefinition,apiVersion是apiextensions.k8s.io/v1;这两个是固定格式;spec字段是用来定义对应指定资源类型资源的相关属性;

  crd.spec字段说明

[root@master01 ~]# kubectl explain crd.spec
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

RESOURCE: spec <Object>

DESCRIPTION:
     spec describes how the user wants the resources to appear

     CustomResourceDefinitionSpec describes how a user wants their resource to
     appear

FIELDS:
   conversion   <Object>
     conversion defines conversion settings for the CRD.

   group        <string> -required-
     group is the API group of the defined custom resource. The custom resources
     are served under `/apis/<group>/...`. Must match the name of the
     CustomResourceDefinition (in the form `<names.plural>.<group>`).

   names        <Object> -required-
     names specify the resource and kind names for the custom resource.

   preserveUnknownFields        <boolean>
     preserveUnknownFields indicates that object fields which are not specified
     in the OpenAPI schema should be preserved when persisting to storage.
     apiVersion, kind, metadata and known fields inside metadata are always
     preserved. This field is deprecated in favor of setting
     `x-preserve-unknown-fields` to true in
     `spec.versions[*].schema.openAPIV3Schema`. See
     https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
     for details.

   scope        <string> -required-
     scope indicates whether the defined custom resource is cluster- or
     namespace-scoped. Allowed values are `Cluster` and `Namespaced`.

   versions     <[]Object> -required-
     versions is the list of all API versions of the defined custom resource.
     Version names are used to compute the order in which served versions are
     listed in API discovery. If the version string is "kube-like", it will sort
     above non "kube-like" version strings, which are ordered lexicographically.
     "Kube-like" versions start with a "v", then are followed by a number (the
     major version), then optionally the string "alpha" or "beta" and another
     number (the minor version). These are sorted first by GA > beta > alpha
     (where GA is a version with no suffix such as beta or alpha), and then by
     comparing major version, then minor version. An example sorted list of
     versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
     foo1, foo10.

[root@master01 ~]# 

  提示:crd.spec中group字段使用来描述对应自定义类型资源的群组名称,其值为字符串;names字段是用来描述自定义类型资源对应类型,名称等等,其值为一个对象;scope字段用来定义对应自定义资源是那个级别的资源;该字段的值只能为Cluster或Namespaced;versions字段是用来指定对应自定义资源的版本,以及对应类型资源的属性字段等信息,该字段为一个列表对象;

  示例:定义一个自定义类型资源

[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需与下面的 spec 字段匹配,并且格式为 '<名称的复数形式>.<组名>'
  name: crontabs.stable.example.com
spec:
  # 组名称,用于 REST API: /apis/<组>/<版本>
  group: stable.example.com
  # 列举此 CustomResourceDefinition 所支持的版本
  versions:
    - name: v1
      # 每个版本都可以通过 served 标志来独立启用或禁止
      served: true
      # 其中一个且只有一个版本必需被标记为存储版本
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # 可以是 Namespaced 或 Cluster
  scope: Namespaced
  names:
    # 名称的复数形式,用于 URL:/apis/<组>/<版本>/<名称的复数形式>
    plural: crontabs
    # 名称的单数形式,作为命令行使用时和显示时的别名
    singular: crontab
    # kind 通常是单数形式的驼峰编码(CamelCased)形式。你的资源清单会使用这一形式。
    kind: CronTab
    # shortNames 允许你在命令行使用较短的字符串来匹配资源
    shortNames:
    - ct
[root@master01 ~]# 

  应用资源清单前使用kubectl get crontab

[root@master01 ~]# kubectl get crontab
error: the server doesn't have a resource type "crontab"
[root@master01 ~]# 

  提示:在没有应用资源清单前使用kubectl get crontab,它会提示我们所没有类型为crontab的资源

  应用资源清单

[root@master01 ~]# kubectl apply -f crontab-crd.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created
[root@master01 ~]# kubectl get crontab
No resources found in default namespace.
[root@master01 ~]# 

  提示:应用资源清单后,再次使用kubectl get crontab就没有报错了,只是提示在default名称空间没有对应类型资源;

  查看crd

[root@master01 ~]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# 

  查看详情

[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name:         crontabs.stable.example.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-01-12T12:39:00Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:conditions:
    Manager:      kube-apiserver
    Operation:    Update
    Time:         2021-01-12T12:39:00Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:scope:
        f:versions:
      f:status:
        f:storedVersions:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:39:00Z
  Resource Version:  805506
  UID:               b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
  Conversion:
    Strategy:  None
  Group:       stable.example.com
  Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Properties:
          Spec:
            Properties:
              Cron Spec:
                Type:  string
              Image:
                Type:  string
              Replicas:
                Type:  integer
            Type:      object
        Type:          object
    Served:            true
    Storage:           true
Status:
  Accepted Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Conditions:
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>
[root@master01 ~]#

  使用自定义资源类型crontab创建资源

[root@master01 ~]# cat my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
[root@master01 ~]# 

  提示:上述资源清单表示创建一个类型为Crontab的资源,该资源的群组版本为stable.example.com/v1;

  应用资源清单

[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME                 AGE
my-new-cron-object   5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name:         my-new-cron-object
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  stable.example.com/v1
Kind:         CronTab
Metadata:
  Creation Timestamp:  2021-01-12T12:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  stable.example.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:cronSpec:
        f:image:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:45:29Z
  Resource Version:  806182
  UID:               31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
  Cron Spec:  * * * * */5
  Image:      my-awesome-cron-image
Events:       <none>
[root@master01 ~]# 

  提示:可以看到对应类型资源已经创建成功;以上示例只是单纯的crd的使用示例,没有任何实质的作用;

  部署自定义控制器

  示例:部署mongodb-aperator

  1、克隆项目

[root@master01 ~]# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git
Cloning into 'mongodb-kubernetes-operator'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411
Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done.
Resolving deltas: 100% (2621/2621), done.
[root@master01 ~]# 

  2、创建名称空间mongodb,并进入到mongodb-kubernetes-operator目录应用crd资源,创建自定义资源类型

[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb
namespace/mongodb created
[root@master01 mongodb-kubernetes-operator]# kubectl get ns
NAME                   STATUS   AGE
default                Active   35d
ingress-nginx          Active   22d
kube-node-lease        Active   35d
kube-public            Active   35d
kube-system            Active   35d
kubernetes-dashboard   Active   11d
mongodb                Active   4s
[root@master01 mongodb-kubernetes-operator]# ls
agent    build  deploy     docs    go.sum      pkg        release.json      scripts  testdata  version
APACHE2  cmd    dev_notes  go.mod  LICENSE.md  README.md  requirements.txt  test     tools.go
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created
[root@master01 mongodb-kubernetes-operator]# 

  验证:查看mongodb类型资源是否已经创建成功?

[root@master01 mongodb-kubernetes-operator]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
mongodb.mongodb.com                                   2021-01-13T06:38:22Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com
NAME                  CREATED AT
mongodb.mongodb.com   2021-01-13T06:38:22Z
[root@master01 mongodb-kubernetes-operator]# 

  3、安装operator

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb
deployment.apps/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
serviceaccount/mongodb-kubernetes-operator created
[root@master01 mongodb-kubernetes-operator]# 

  提示:mongodb-kubernetes-operator这个项目是将自定义控制器和自定义资源类型分开实现的;其operator只负责创建和监听对应资源类型的变化,在资源有变化时,实例化为对应资源对象,并保持对应资源对象状态吻合用户期望状态;上述四个清单中主要是创建了一个sa账户,并对对应的sa用户授权;

  operator.yaml内容

[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-kubernetes-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  template:
    metadata:
      labels:
        name: mongodb-kubernetes-operator
    spec:
      serviceAccountName: mongodb-kubernetes-operator
      containers:
        - name: mongodb-kubernetes-operator
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
          command:
            - mongodb-kubernetes-operator
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "mongodb-kubernetes-operator"
            - name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
              value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
            - name: MONGODB_IMAGE
              value: "library/mongo"
            - name: MONGODB_REPO_URL
              value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]# 

  提示:上述资源主要用deploy控制器运行对应自定义控制器为一个pod;

  验证:查看operator是否正常运行

[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          26s
[root@master01 mongodb-kubernetes-operator]# 

  提示:能够看到operator正常运行,就表示operator已经安装成功;

  验证:使用自定义资源类型创建一个mongodb 副本集集群

[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml 
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: example-mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram

# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
  name: my-user-password
type: Opaque
stringData:
  password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# 

  应用清单

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              0/2     Pending   0          9s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          88s
[root@master01 mongodb-kubernetes-operator]# 

  提示:这里可以看到对应pod处于pending状态;

  查看pod详细信息

[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events"
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  66s (x2 over 66s)  default-scheduler  0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
[root@master01 mongodb-kubernetes-operator]# 

  提示:这里提示没有可以用的pvc;

  删除mongodb名称空间下pvc

[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
NAME                            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Pending                                                     92s
[root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb
persistentvolumeclaim "data-volume-example-mongodb-0" deleted
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 mongodb-kubernetes-operator]# 

  创建pv和pvc

[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v1
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v1
    server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v2
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v2
    server: 192.168.0.99
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v3
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v3
    server: 192.168.0.99
[root@master01 ~]# 

  应用清单创建pv

[root@master01 ~]# kubectl apply -f pv-demo.yaml
persistentvolume/nfs-pv-v1 created
persistentvolume/nfs-pv-v2 created
persistentvolume/nfs-pv-v3 created
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
[root@master01 ~]# 

  创建pvc清单

[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-0 
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-1
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-2
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
[root@master01 ~]# 

  应用清单创建pvc

[root@master01 ~]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 ~]# kubectl apply -f pvc-demo.yaml
persistentvolumeclaim/data-volume-example-mongodb-0 created
persistentvolumeclaim/data-volume-example-mongodb-1 created
persistentvolumeclaim/data-volume-example-mongodb-2 created
[root@master01 ~]# kubectl get pvc -n mongodb    
NAME                            STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Bound    nfs-pv-v1   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-1   Bound    nfs-pv-v2   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-2   Bound    nfs-pv-v3   1Gi        RWO,ROX,RWX                   6s
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-0                           102s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-1                           102s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-2                           102s
[root@master01 ~]# 

  提示:可以看到对应pvc和pv已经绑定好了;

  验证:查看mongodb副本集集群是否跑起来了?

[root@master01 ~]# kubectl get pods -n mongodb
NAME                                           READY   STATUS            RESTARTS   AGE
example-mongodb-0                              2/2     Running           0          6m19s
example-mongodb-1                              0/2     PodInitializing   0          10s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running           0          7m38s
[root@master01 ~]# kubectl get pods -n mongodb -w
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          6m35s
example-mongodb-1                              1/2     Running   0          26s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          7m54s
example-mongodb-1                              2/2     Running   0          43s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Init:0/1   0          0s
example-mongodb-2                              0/2     Init:0/1   0          1s
example-mongodb-2                              0/2     Terminating   0          4s
example-mongodb-2                              0/2     Terminating   0          6s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Init:0/1      0          0s
example-mongodb-2                              0/2     Init:0/1      0          1s
example-mongodb-2                              0/2     PodInitializing   0          7s
example-mongodb-2                              1/2     Running           0          14s
example-mongodb-2                              2/2     Running           0          36s
^C[root@master01 ~]# kubectl get pods -n mongodb 
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          8m
example-mongodb-1                              2/2     Running   0          111s
example-mongodb-2                              2/2     Running   0          48s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          9m19s
[root@master01 ~]# 

  提示:可以看到对应的pod已经正常跑起来了;

  验证:使用mongo 连接mongodbpod,看看对应副本集集群是否工作正常?

[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP             NODE             NOMINATED NODE   READINESS GATES
example-mongodb-0                              2/2     Running   0          9m12s   10.244.4.101   node04.k8s.org   <none>           <none>
example-mongodb-1                              2/2     Running   0          3m3s    10.244.2.130   node02.k8s.org   <none>           <none>
example-mongodb-2                              2/2     Running   0          2m      10.244.1.130   node01.k8s.org   <none>           <none>
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          10m     10.244.3.116   node03.k8s.org   <none>           <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
        "hosts" : [
                "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
        ],
        "setName" : "example-mongodb",
        "setVersion" : 1,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "electionId" : ObjectId("7fffffff0000000000000003"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2021-01-13T06:52:27.873Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 153,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610520741, 1),
                "signature" : {
                        "hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
                        "keyId" : NumberLong("6917119940596072451")
                }
        },
        "operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY> 

  提示:可以看到三个mongodb pod是副本集关系;其中example-mongodb-0是主节点,其他两个是从节点;

  最后还是说一下,我在做上面的实验中,虽然mongodb operator已经正常工作,但是用mongo这个客户端工具连接到主节点没法写数据,提示没有权限;但对应的用户在对应库是有读写权限的;在admin库下创建用户能提示用户添加成功,但是过几秒查询用户信息,发现用户不存在,不知道什么原因,有知道的朋友麻烦告知下(博主邮箱:linux-1874@qq.com),博主将感激不尽。。

相关文章: