Development practice of Kubernetes operator model

0. Preface

Recently, we used Kubernetes's Operator Pattern to implement the operation and maintenance system when developing a microservice platform that meets our own business needs. In this paper, we sorted out the main knowledge points and technical details accumulated in the implementation process.

After reading this article, readers will have a basic understanding of Operator Pattern and be able to apply the pattern to their own business. In addition, we will also share some relevant knowledge needed to realize this operation and maintenance system.
Note: to read this article, you need to have a basic understanding of Kubernetes and Go.

 

1. What is Operator Pattern

Before we can explain what Operator Pattern is, we need to know what happened during the period when we used a Kubernetes client - kubectl for example - to issue an instruction to the Kubernetes cluster until the instruction was executed by the Kubernetes cluster.
Here is an example of the command "kubectl create -f ns-my-workspace.yaml". The whole execution link of this command is roughly as shown in the figure below:

### ns-my-workspace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-workspace

As shown in the above figure, all component interactions in the Kubernetes cluster are completed in the form of RESTful API, including the first step of the controller listening operation and the instructions sent by kubectl in the second step. Although we execute the "kubectl create -f ns-my-workspace.yaml" instruction, in fact, kubectl will send a POST request to the "API server":

curl --request POST \
  --url http://${k8s.host}:${k8s.port}/api/v1/namespaces \
  --header 'content-type: application/json' \
  --data '{
    "apiVersion":"v1",
    "kind":"Namespace",
    "metadata":{
        "name":"my-workspace"
    }
}'

As shown in the above cURL instruction, the Kubernetes API server actually accepts JSON data type instead of YAML.
Then all the resources created will be persisted to the etcd component, and the API server is the only component interacting with the etcd in the Kubernetes cluster.
After that, the created my workspace resource will be sent to the "Namespace controller" listening to the changes of the namespaces resource. Finally, the "Namespace controller" will perform the specific operation of creating my workspace Namespace. In the same way, when creating a ReplicaSet resource, it will be executed by the "ReplicaSet controller". When creating a Pod, it will be executed by the "Pod controller". Other types of resources are similar to them. These controllers together constitute the "Kubernetes API controller set" in the figure above.

It's not hard to find that there are two factors to realize the operation logic of a certain domain type in Kubernetes, such as Namespace, ReplicaSet and Pod mentioned above, that is, Kind in Kubernetes:

  1. The model abstraction of this domain type, such as the YAML data structure described in the ns-my-workspace.yaml file above, determines the RESTful API request sent by Kubernetes client to Kubernetes API server, and also describes the domain type itself.
  2. In fact, it deals with controllers of this domain type abstraction, such as "Namespace controller", "ReplicaSet controller" and "Pod controller". These controllers implement the specific business logic described in this abstraction and provide these services through the RESTful API.

When Kubernetes developers need to extend Kubernetes capabilities, they can also follow this pattern, that is, to provide an abstraction of the capabilities they want to extend and a Controller that implements the abstract concrete logic. The former is called CRD(Custom Resource Definition), and the latter is called Controller.
Operator pattern is a pattern to realize Kubernetes extensibility in this way. Operator pattern thinks that the solution of a domain problem can be imagined as an "operator". The operator operates the cluster API between the user and the cluster through an "order" to achieve the purpose of completing various requirements in this domain. The order here is CR(Custom Resource, an instance of CRD), and the operator is the controller and the implementer of the specific logic. The reason why operator is emphasized, rather than the traditional server role in the computer field, is that operator does not create and provide new services in essence. It is just a combination of existing Kubernetes API service s.
The "operation and maintenance system" in this paper is an operator to solve the problems in the field of operation and maintenance.

 

2. Operator Pattern Practice

In this section, we will build a Kubernetes Operator by using the kubebuilder tool. After this section, we will obtain a CRD and its corresponding Kubernetes API controller in our Kubernetes cluster for simple deployment of a microservice. That is, when we create YAML as follows:

apiVersion: devops.my.domain/v1
kind: DemoMicroService
metadata:
  name: demomicroservice-sample
spec:
  image: stefanprodan/podinfo:0.0.1

An example of Kubernetes deployment can be obtained:

All sample codes in this section are provided in: https://github.com/l4wei/kubebuilder-example

2.1 Kubebuilder implementation

Kubebuilder(https://github.com/kubernetes-sigs/kubebuilder )It is a scaffolding tool to build Kubernetes APIs controller and CRD with Go language. By using kubebuilder, users can follow a simple programming framework and use Go language to implement an operator conveniently.

2.1.1 installation

Before you install kubebuilder, you need to install the Go language and kustomize, and make sure it works properly.
Kustmize is a customized tool for generating Kubernetes YAML Configuration files. You can generate the Kubernetes YAML Configuration you need in batches by following a set of kustmize configuration. kubebuilder uses kustomize to generate some YAML configurations required by the controller. mac users can use brew to install kustomize easily.
Then use the following script to install kubebuilder:

os=$(go env GOOS)
arch=$(go env GOARCH)

# download kubebuilder and extract it to tmp
curl -L https://go.kubebuilder.io/dl/2.2.0/${os}/${arch} | tar -xz -C /tmp/

# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.2.0_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

If you can see the help document with kubebuilder -h, it means that the installation of kubebuilder is successful.

2.1.2 create project

Use the following script to create a kubebuilder project:

mkdir example
cd example
go mod init my.domain/example
kubebuilder init --domain my.domain

My.domain of the above command is generally the domain name of your organization, and "example" is generally the project name of your Go language project. According to this setting, if this Go project as a module is to be relied on by other Go projects, it is generally named "my.domain/example".
If your example directory is set up under the ${GOPATH} directory, you do not need the command "go mod init my.domain/example". Go language can also find the go pkg under the example directory.
Then make sure that the following two commands have been executed on your development machine:

export GO111MODULE=on
sudo chmod -R 777 ${GOPATH}/go/pkg

The execution of the above two commands can solve the problem of cannot find package... (from $goroot) that may occur during development.
After creating the project, your example directory structure will be roughly as follows:

.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│   └── manager
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   └── role_binding.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
└── main.go

The manager in the bin directory in the above directory is the binary executable compiled by the project, that is, the executable of this controller.
The configuration under the config directory is all kustmize. For example, the file under the config/manager directory generates the kustmize configuration of the controller deployment YAML configuration file. If you execute the following instructions:

kustomize build config/manager

You can see the YAML configuration generated by kustomize:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
  name: system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: controller-manager
  namespace: system
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --enable-leader-election
        command:
        - /manager
        image: controller:latest
        name: manager
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
      terminationGracePeriodSeconds: 10

The above is to deploy bin/manager to YAML configurations of Kubernetes cluster.

2.1.3 create API

The project created above is just an empty shell, and has not provided any Kubernetes API, nor can it handle any CR. Use the following script to create a Kubernetes API:

kubebuilder create api --group devops --version v1 --kind DemoMicroService

The group of the above command and the domain entered in the previous project creation will form the first half of the apiVersion field in the Kubernetes API YAML resource you created. The above version is the second half, so the apiVersion in your customized resource YAML should be written as: devops.my.domain/v1. The above kind is the kind field in your custom resource. The resource created by this instruction looks like the config / samples / devices ˊ V1 ˊ demomicroservice.yaml file created by kubebuilder:

apiVersion: devops.my.domain/v1
kind: DemoMicroService
metadata:
  name: demomicroservice-sample
spec:
  # Add fields here  foo: bar

Enter this command and you will be prompted whether to create Resource (CRD) or Controller (Controller). Enter "y" for approval.
After you execute the command, your engineering structure will look like this:

.
├── Dockerfile
├── Makefile
├── PROJECT
├── api
│   └── v1
│       ├── demomicroservice_types.go
│       ├── groupversion_info.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── manager
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_demomicroservices.yaml
│   │       └── webhook_in_demomicroservices.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── demomicroservice_editor_role.yaml
│   │   ├── demomicroservice_viewer_role.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   └── role_binding.yaml
│   ├── samples
│   │   └── devops_v1_demomicroservice.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── controllers
│   ├── demomicroservice_controller.go
│   └── suite_test.go
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
└── main.go

Compared with before API creation, the following are added:

  • api Directory - the data structure code that defines the Kubernetes API you create.
  • controllers directory -- that is, the implementation code of the controller.
  • config/crd Directory - the kustomize configuration in this directory generates the YAML configuration of the CRD you want to define.

Enter the following command:

 make manifests

You can see the CRD you created when you created the Kubernetes API in the file config / CRD / bases / devices.my.domain \ demomicroservices.yaml:


---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.2.4
  creationTimestamp: null
  name: demomicroservices.devops.my.domain
spec:
  group: devops.my.domain
  names:
    kind: DemoMicroService
    listKind: DemoMicroServiceList
    plural: demomicroservices
    singular: demomicroservice
  scope: Namespaced
  validation:
    openAPIV3Schema:
      description: DemoMicroService is the Schema for the demomicroservices API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: DemoMicroServiceSpec defines the desired state of DemoMicroService
          properties:
            foo:
              description: Foo is an example field of DemoMicroService. Edit DemoMicroService_types.go
                to remove/update
              type: string
          type: object
        status:
          description: DemoMicroServiceStatus defines the observed state of DemoMicroService
          type: object
      type: object
  version: v1
  versions:
  - name: v1
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

 

2.1.4 API attribute definition

Kubernetes API has been created. Now we need to define the attributes of the API, which really describe the abstract features of the created CRD.
In our DemoMicroService Kind example, we simply abstract a deployment CRD of microservice, so our CRD has only one attribute, that is, the container image address of the service.
To do this, we only need to modify the api/v1/demomicroservice_types.go file:

The above git diff shows that we have changed the Foo attribute of the original example to the Image attribute we need. The definition of API attribute and CRD basically only need to modify the file.
Again:

 make manifests

You can see that the generated CRD resource has changed, which will not be covered here.
Now we also modify the config / samples / devices? V1? Demomicroservice.yaml file, which needs to be used later to test our implemented controller:

apiVersion: devops.my.domain/v1
kind: DemoMicroService
metadata:
  name: demomicroservice-sample
spec:
  image: stefanprodan/podinfo:0.0.1

 

2.1.5 logic realization of controller

CRD is defined, and now the controller is implemented.
The controller logic we want to implement in this example is very simple, which can be described as follows:

  1. When we execute kubectl create - f config / samples / devices? V1? DemoMicroService.yaml, the controller will create a Kubernetes Deployment resource in the cluster to implement the DemoMicroService deployment.
  2. When we execute kubectl delete - f config / samples / devices? V1? DemoMicroService.yaml, the controller will delete the Deployment resource created in the cluster, indicating that the DemoMicroService is offline.

2.1.5.1 implementation of deployment

Before writing the code, we need to understand the development mode of the kubebuilder program.
Because we want to implement the controller of DemoMicroService, we need to focus on the file "Controllers / DemoMicroService" controller.go ". If it is not a complex function, we usually only need to change the file. In the file, we need to focus on the Reconcile method:

func (r *DemoMicroServiceReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    _ = context.Background()
    _ = r.Log.WithValues("demomicroservice", req.NamespacedName)

    // your logic here

    return ctrl.Result{}, nil
}

In short, whenever the kubernetes cluster monitors the change of DemoMicroService CR, it will call the Reconcile method, and use the changed DemoMicroService resource name and its namespace as the parameter of Reconcile method to locate the changed resource. That is, the above req parameter. The structure of this parameter is:

type Request struct {
    // NamespacedName is the name and namespace of the object to reconcile.
    types.NamespacedName
}
type NamespacedName struct {
    Namespace string
    Name      string
}

A friend who is familiar with front-end development may associate with the development mode of React. They are really similar. They are all the changes of monitoring objects, and then execute some logic according to the changes of monitoring objects. However, kubebuilder is more extreme. Instead of abstracting the concept of life cycle, it only provides a Reconcile method. Developers need to determine the life cycle of CRD in this method and execute different logic in different life cycles.
The following code implements the deployment function:

func (r *DemoMicroServiceReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("demomicroservice", req.NamespacedName)

    dms := &devopsv1.DemoMicroService{}
    if err := r.Get(ctx, req.NamespacedName, dms); err != nil {
        if err := client.IgnoreNotFound(err); err == nil {
            log.Info("No corresponding DemoMicroService resource, That's where you go resource Life cycle after deletion")
            return ctrl.Result{}, nil
        } else {
            log.Error(err, "It's not an undetected error, it's an unexpected error, so it's directly returned here")
            return ctrl.Result{}, err
        }
    }

    log.Info("Coming here means DemoMicroService resource Found, i.e resource It has been successfully created. When entering the resource To execute the main process of logic")
    podLabels := map[string]string{
        "app": req.Name,
    }
    deployment := appv1.Deployment{
        TypeMeta: metav1.TypeMeta{
            Kind:       "Deployment",
            APIVersion: "apps/v1",
        },
        ObjectMeta: metav1.ObjectMeta{
            Name:      req.Name,
            Namespace: req.Namespace,
        },
        Spec: appv1.DeploymentSpec{
            Selector: &metav1.LabelSelector{
                MatchLabels: podLabels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: podLabels,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name:            req.Name,
                            Image:           dms.Spec.Image,
                            ImagePullPolicy: "Always",
                            Ports: []corev1.ContainerPort{
                                {
                                    ContainerPort: 9898,
                                },
                            },
                        },
                    },
                },
            },
        },
    }
    if err := r.Create(ctx, &deployment); err != nil {
        log.Error(err, "Establish Deployment resource error")
        return ctrl.Result{}, err
    }
    return ctrl.Result{}, nil
}

The above code shows two stages in the life cycle: Line 8 represents the stage after DemoMicroService resource has been successfully deleted, and we have done nothing at this time. And in line 16, after the DemoMicroService resource is created / updated, we build a Deployment resource and create it into the Kubernetes cluster.
There is a problem in this code, that is, it is unable to update DemoMicroService resource. If the spec.image of the same DemoMicroService resource is changed, the same Deployment resource will be create d again in the above code, which will lead to an error of "already exists". In order to explain the development logic conveniently, we did not deal with this problem. Please note.

2.1.5.1 realization of offline

In fact, when we explained the deployment logic in the previous section, we can implement the offline logic: we only need to delete the created deployment in the "after deletion" life cycle stage. But there is a problem in this way. We delete the deployment after DemoMicroService resource is deleted successfully. If the logic of deleting the deployment is wrong and the deployment is not deleted successfully, then the situation that the deployment is still there and DemoMicroService is no longer there. If we need to use DemoMicroService to manage Deployment, then this is not the result we want.
So we'd better delete the Deployment before the DemoMicroService really disappears (that is, the period from "delete DemoMicroService" to "completely disappear DemoMicroService")? See the following code example:

const (
    demoMicroServiceFinalizer string = "demomicroservice.finalizers.devops.my.domain"
)

func (r *DemoMicroServiceReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("demomicroservice", req.NamespacedName)

    dms := &devopsv1.DemoMicroService{}
    if err := r.Get(ctx, req.NamespacedName, dms); err != nil {
        if err := client.IgnoreNotFound(err); err == nil {
            log.Info("No corresponding DemoMicroService resource, That's where you go resource Life cycle after deletion")
            return ctrl.Result{}, nil
        } else {
            log.Error(err, "It's not an undetected error, it's an unexpected error, so it's directly returned here")
            return ctrl.Result{}, err
        }
    }

    if dms.ObjectMeta.DeletionTimestamp.IsZero() {
        log.Info("Enter into apply this DemoMicroService CR Logic")
        log.Info("At this point, you must ensure that resource Of finalizers There's a controller in it finalizer")
        if !util.ContainsString(dms.ObjectMeta.Finalizers, demoMicroServiceFinalizer) {
            dms.ObjectMeta.Finalizers = append(dms.ObjectMeta.Finalizers, demoMicroServiceFinalizer)
            if err := r.Update(ctx, dms); err != nil {
                return ctrl.Result{}, err
            }
        }

        if _, err := r.applyDeployment(ctx, req, dms); err != nil {
            return ctrl.Result{}, nil
        }
    } else {
        log.Info("Go to delete this DemoMicroService CR Logic")
        if util.ContainsString(dms.ObjectMeta.Finalizers, demoMicroServiceFinalizer) {
            log.Info("If finalizers Is cleared, the DemoMicroService CR It no longer exists, so it must be deleted before Deployment")
            if err := r.cleanDeployment(ctx, req); err != nil {
                return ctrl.Result{}, nil
            }
        }
        log.Info("empty finalizers,After that DemoMicroService CR It's going to disappear")
        dms.ObjectMeta.Finalizers = util.RemoveString(dms.ObjectMeta.Finalizers, demoMicroServiceFinalizer)
        if err := r.Update(ctx, dms); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

To facilitate the presentation of the main logic, I encapsulate the code for creating and deleting a Deployment into two methods, applyDeployment and cleanDeployment.
As shown in the above code, to judge the stage from "delete DemoMicroService" to "completely disappear DemoMicroService", the key point is to judge the two meta information: DemoMicroService.ObjectMeta.DeletionTimestam and DemoMicroService.ObjectMeta.Finalizers. The former indicates the specific occurrence time of "delete DemoMicroService". If it is not 0, it indicates that the "delete DemoMicroService" command has been issued. The latter indicates what other logic has not been executed before the DemoMicroService is actually deleted, that is, before "DemoMicroService disappears completely". If Finalizers is not empty, then the DemoMicroService It won't really disappear.
ObjectMeta.Finalizers of any resource is a list of strings, each of which indicates that a pre delete logic has not been executed. As shown in "DemoMicroService. Finalizers. Devices. My. Domain" above, it means the pre delete logic of DemoMicroService controller to DemoMicroService resource. After the DemoMicroService is created, the Finalizer will be quickly planted by DemoMicroService controller, and after the command of "delete DemoMicroService" is issued, and pre delete After the logic (delete Deployment in this case) is executed correctly, the Finalizer is erased by the DemoMicroService controller. At this point, finalizers on DemoMicroService is empty, and kubernetes will make the DemoMicroService disappear completely. Kubernetes created the stage of the resource's life cycle from "issue delete instruction to the resource" to "the resource disappears completely".
What happens if Finalizers cannot be empty for various reasons? The answer is that this resource can never be deleted. If you use kubectl delete to delete, this instruction will never return. This is also a problem often encountered when using Finalizers mechanism. If you find that a resource cannot be deleted all the time, please check whether you have planted a Finalizer that will not be deleted.

2.1.6 commissioning and release

 

2.1.5.1 commissioning

Before we start to run & debug the controller we wrote, we'd better set an abbreviated name for our DemoMicroService CRD, because we don't need to enter the name of "demomicroservice" for later operations.

As shown in the figure above, add a line of comments to the file "API / V1 / demomicroservice" types. Go ". We set the abbreviation of" demomicroservice "to" dms ".
By the way, in kubebuilder, all comments with "+ kubebuilder" are useful. Don't delete them easily. They are some configurations of the kubebuilder project.
After the modification, execute the "make manifests" command. It will be found that the CRD file generated by kubebuiler has been modified and the abbreviated configuration has been added:

After setting the abbreviation again, we execute the following command to install CRD to the Kubernetes cluster currently connected to your development machine:

make install

After that, you can see that your cluster has its own CRD installed:

Now we can start our controller program. Because the author uses the GoLand IDE, I directly click to start the main function in main.go:

Readers can choose to use their own startup mode according to their own development tools, in a word, just run the Go program. You can also start directly with the following command:

make run

After the controller is started locally, your development machine is the controller of DemoMicroService. The Kubernetes cluster you connect to will send changes about DemoMicroService resource to your development machine for execution, so the break point will also be broken.
Next, we check whether the code we wrote before works normally and execute the command:

kubectl apply -f ./config/samples/devops_v1_demomicroservice.yaml

We will see that the resource appears in the Kubernetes cluster:

The above "dms" is the abbreviation of "demomicroservice" we just set.
And the Deployment created with DemoMicroService and its created Pod:

When we execute the command to delete dms, these deployment s and pod s will also be deleted.

2.1.5.1 release

Use the following command to publish the controller to the Kubernetes cluster currently connected to your development machine:

make deploy

After that, kubebuilder will create a namespace dedicated to the controller in the cluster. In our example, the namespace is example system. After that, you can see that your controller has been published to the cluster you are currently connected to through the following command:

kubectl get po -n example-system

If the pod fails to be published, it is mostly caused by the lack of domestic connection to the image warehouse of gcr.io. Search the image warehouse of "GCR. IO" in the project and replace it with the image warehouse that you can easily access.

2.2 summary

In this section, we have a general understanding of the use of kubebuilder scaffolding, and the development method of kubebuilder program. And a controller to realize the deployment and offline of microservice is implemented.
Perhaps the reader will ask, why not create a Deployment directly, but implement it in such a troublesome way. That's because the customized CRD can better abstract the developer's business scenarios. For example, in our example, our microservice only cares about the image address, and other Deployment attributes can all be defaulted, so our DemoMicroService looks much cleaner than Deployment.
In addition, it has the following advantages:

  1. Kubernetes' resource ensures the consistency of execution results: kubernetes naturally conforms to idempotence for the execution of resources, and the resourceVersion mechanism provided in kubernetes also solves the problem of inconsistent results during concurrent execution. If the developers solve these problems themselves, they are often laborious and hard-working.
  2. kubebuilder's development model helps developers save a lot of work, including monitoring resource changes, retry of resource errors, and generating necessary YAML configurations.

    1. What's worth mentioning here is the retry mechanism of kubebuilder. If the execution of your own resource fails, kubebuilder will help you retry until the resource is successfully executed, which saves the workload of your own implementation of retry logic.
  3. The call link is safe and controllable. By depositing your business logic into CRD and controller, you can fully enjoy the rbac authority management and control system of Kubernetes. It is more secure, convenient and precise to control the controller interface you developed.

Because our example is too simple, these advantages may sound pale. In the next section, after we go to more complex operation and maintenance scenarios, we can have a deeper understanding of these advantages described above.

 

3. Implementation of operation and maintenance system

If you are only interested in the operator pattern and its practice and don't care how the operation and maintenance system is implemented, you can skip this section.
The following figure shows what the microservice O & M controller does in the microservice platform we developed. Readers can see the position of such an operator in the whole microservice platform

dmsp in the figure above represents our microservice platform, and dmsp OPS operator is the controller of the operation and maintenance system. It can be seen that because of the existence of dmsp OPS operator, the instructions to be issued by the user operation control console are very simple, and the actual operation and maintenance operations can be executed by dmsp OPS operator. And dmsp OPS operator, as the capability in Kubernetes cluster, also precipitated to the bottom of the technology stack, completely separated from the business logic of the upper layer.

3.1 complete abstraction of microservices

In Section 2, we implemented a demo microservice. In fact, that demo microservice only cares about the image address, which is obviously not enough. So we implemented MicroServiceDeploy CRD and its controller, which can abstract and realize more o & M functions. A MicroServiceDeploy CR looks like the following:

apiVersion: custom.ops/v1
kind: MicroServiceDeploy
metadata:
  name: ms-sample-v1s0
spec:
  msName: "ms-sample"                     # Microservice name
  fullName: "ms-sample-v1s0"              # Microservice instance name
  version: "1.0"                          # Microservice instance version
  path: "v1"                              # The large version of the microservice instance. The string will appear in the domain name of the microservice instance
  image: "just a image url"               # Image address of microservice instance
  replicas: 3                             # Number of replica s for microservice instances
  autoscaling: true                       # Does the microservice enable automatic capacity expansion and contraction
  needAuth: true                          # Whether tenant base authentication is required when accessing the microservice instance
  config: "password=88888888"             # Runtime configuration item for this microservice instance
  creationTimestamp: "1535546718115"      # The creation timestamp of the microservice instance
  resourceRequirements:                   # Machine resources required by the microservice instance
    limits:                                 # The maximum resource configuration that this microservice instance will use
      cpu: "2"
      memory: 4Gi
    requests:                               # The minimum resource configuration needed by the microservice instance
      cpu: "2"
      memory: 4Gi
  idle: false                             # Whether to enter no-load state

In fact, the above resource creates many other kubernetes resources, which really constitute the actual capabilities of the microservice. The way these kubernetes resources are created is basically the way they are explained in Section 2.
Next, I will introduce these kubernetes resources separately, and explain the significance and role of these kubernetes resources respectively.

3.2 Service&ServiceAccount&Deployment

The first is a necessary service for a microservice, serviceaccount and Deployment. You should be familiar with these three kinds of resource s. Here is just a few instructions. Post the YAML configuration created by MicroServiceDeploy controller directly.

3.2.1 Service&ServiceAccount

apiVersion: v1
kind: Service
metadata:
  labels:
    app: ms-sample
    my-domain-ops-controller-make: "true"
  name: ms-sample
spec:
  ports:
  - name: http
    port: 9898
    protocol: TCP
    targetPort: 9898
  selector:
    app: ms-sample
status:
  loadBalancer: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    my-domain-ops-controller-make: "true"
  name: ms-sample

The above my domain OPS controller make is a label printed by the custom controller itself to distinguish that the resource is created by our custom controller.

3.2.2 Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app.ops.my.domain/last-rollout-at: "1234"
  labels:
    app: ms-sample
    my-domain-ops-controller-make: "true"
  name: ms-sample-v1s0
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ms-sample
    type: RollingUpdate
  template:
    metadata:
      annotations:
        app.ops.my.domain/create-at: "1234"
        prometheus.io/scrape: "true"
      labels:
        app: ms-sample
    spec:
      containers:
      - env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: SERVICE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app']
        image: "just a image url"
        imagePullPolicy: Always
        name: ms-sample
        ports:
        - containerPort: 9898
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 400Mi
          requests:
            cpu: 100m
            memory: 400Mi
        volumeMounts:
        - mountPath: /home/admin/logs
          name: log
        - mountPath: /home/admin/conf
          name: config
      initContainers:
      - command:
        - sh
        - -c
        - chmod -R 777 /home/admin/logs || exit 0
        image: busybox
        imagePullPolicy: Always
        name: log-volume-mount-hack
        volumeMounts:
        - mountPath: /home/admin/logs
          name: log
      volumes:
      - hostPath:
          path: /data0/logs/ms-sample/1.0
          type: DirectoryOrCreate
        name: log
      - configMap:
          defaultMode: 420
          name: ms-sample-v1s0
        name: config

The above Deployment has three points to explain:

  1. The annotation of app.ops.my.domain/create-at in Pod is the annotation that the controller gives to Pod, which is used to force the Pods under the Deployment to restart. Even if there is no other change when the Deployment is applied, these Pods will be restarted, which is very useful when the Pods are forced to restart.
  2. The volume with the above name as log indicates the volume mount of the log. It indicates that the logs collected in the / home/admin/logs directory of the container will be synchronized to the / data0/logs/ms-sample/1.0 directory of the host. To make this mechanism work, you need to ensure that the service in your container prints the logs to the directory / home/admin/logs.
  3. The above volume mount with the name of config indicates that the ConfigMap configuration with the name of ms-sample-v1s0 is mounted to the / home/admin/conf directory in the container, so that you can read the runtime configuration in the container by reading the configuration file in the / home/admin/conf directory. Details will be provided in Section 3.3.

3.3 ConfigMap for runtime configuration items

The ms-sample-v1s0 ConfigMap mentioned in section 3.2.2 is as follows:

apiVersion: v1
data:
  ms.properties: name=foo
kind: ConfigMap
metadata:
  labels:
    app: ms-sample
    my-domain-ops-controller-make: "true"
  name: ms-sample-v1s0

When the above ConfigMap is mount ed under / home/admin/conf in the container, an ms.properties file will be created under / home/admin/conf. the content of the file is "name=foo". The runtime configuration can be obtained by reading the file inside the container. And the configuration is dynamically updated in real time. That is to say, if ConfigMap changes, the contents of the files in the container will also change, so that the latest configuration will take effect even if the container does not restart.

3.3 resource management

In Kubernetes, the term resource generally refers to the default machine resource of the system, namely cpu and memory.
Resource management here refers to the management and control of the total resources of the microservice deployed namespace, as well as the resource limitation of each microservice deployed container. YAML for this purpose is:

apiVersion: v1
kind: ResourceQuota
metadata:
  labels:
    my-domain-ops-controller-make: "true"
  name: default-resource-quota
  namespace: default
spec:
  hard:
    requests.cpu: "88"
    limits.cpu: "352"
    requests.memory: 112Gi
    limits.memory: 448Gi
---
apiVersion: v1
kind: LimitRange
metadata:
  labels:
    my-domain-ops-controller-make: "true"
  name: default-limit-range
  namespace: default
spec:
  limits:
  - default:
      cpu: 400m
      memory: 2Gi
    defaultRequest:
      cpu: 100m
      memory: 500Mi
    max:
      cpu: "2"
      memory: 8Gi
    type: Container

Unlike other kubernetes resources, the above two resources are not created when MicroServiceDeploy CR is deployed, but when the controller is deployed, as a configuration for a namespace in the cluster. So you need to create the above two resources in the init method of the controller.
The ResourceQuota above limits the total amount of resources that default namespace can occupy.
The LimitRange limits the amount of resources that can be occupied by all containers in the default namespace that do not limit the amount of resources. The reason why there is a default resource quota for each container is that kubernetes will grade the pod according to the resource configuration: if a pod is not configured with the resource quantity, the pod has the lowest importance; secondly, the pod with the resource configuration, but the limits! = requests; finally, the pod with the resource configuration, and the limits == requests Pod, the most important. Kubernetes will release the less important pod when the total resources are insufficient, and use it to schedule the more important pod.
The three levels described above are: best effort (lowest priority), Burstable, Guaranteed (highest priority). This level is called QoS(Quality of Service) level. You can view the QoS level of the Pod in the status.qosClass field of Kubernetes Pod resource.

3.4 automatic expansion and contraction of HPA

The HPA(HorizontalPodAutoscaler) resource for auto scaling is also created when MicroServiceDeploy CR is deployed. It aims at the MicroServiceDeploy CR's representative microservices:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app: ms-sample
    my-domain-ops-controller-make: "true"
  name: ms-sample-v1s0
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ms-sample-v1s0
  targetCPUUtilizationPercentage: 81

The HPA resource above indicates that the Pod under ms-sample-v1s0 Deployment will be expanded and shrunk according to the cpu utilization, and the range of Pod number is: [1, 10].
There is a lot to talk about automatic expansion and contraction. I will write a single article to explain this in detail.

Tags: Mobile Kubernetes Attribute Go curl

Posted on Tue, 17 Mar 2020 02:36:39 -0700 by stilgar