Three publishing strategies to solve the problem of rapid application delivery in K8s

By Hao Shuwei, senior R & D Engineer of alicloud

Preface

Software technology updates quickly, but our goal is always the same. That is to increase the deployment frequency of applications and shorten the iteration cycle of product functions on the premise of security and stability. This benefit is that enterprises can obtain the price value of products in a shorter time, get customer feedback and respond to customer needs faster, so as to further improve product competition In addition, enterprises can also release more resources to invest in R & D of innovative business and create more value, which is a virtuous circle process.

The rapid iteration of application products can bring us all kinds of benefits, but the challenges also coexist with it. The higher frequency of application release means that the risk of unexpected failure of online business is greater. In addition to fully testing and verifying the iterative function in the pre release test environment before the product goes online, formulating the most appropriate application release strategy is another very important topic, because it can minimize the risk of business failure and the resulting loss.

Key points of cloud native application delivery

We say that frequent product iterations mean greater failure risk, especially for traditional applications and cloud native applications. Because cloud native applications are usually based on cloud distributed deployment mode, and each application may be used by multiple functional components to provide complete services together, each component has its own independent iterative process and plan. In this case, the more functional components there are, the greater the probability of error. So how to improve these pain points in the application delivery layer, we summarize the following key points of cloud native application delivery.

  • How to make full use of the advantages of cloud native architecture infrastructure. This advantage can be summarized as two points: flexibility and high availability;
  • How to have the ability of cross platform migration and delivery. There are great differences in computing, storage and network resources at the bottom of the infrastructure. In the past, the difference in infrastructure was determined by the application at the top, while the delivery of cloud native application needs to have the ability of cross platform migration and delivery;
  • How to realize the autonomy of application operation and maintenance. Autonomy is not equal to automation. Automation refers to triggering a process, which can automatically achieve a desired result after the end of the process. Autonomy refers to that when an application is in a highly available running state, if a copy of a functional component fails, the application can automatically remove the failed copy and supplement the new application copy;
  • How to make applications more predictable. The end state of application delivery is predictable when we write the application layout template. If the application delivery becomes more predictable, the risk will be minimized;
  • How to improve the average recovery time of application. If the application has faults beyond the scope of application autonomy that need human intervention, the faster average recovery time means lower business loss.

Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services that facilitates declarative configuration and automation. Its own platform capabilities have met most of the requirements we mentioned earlier. Kubernetes uses container technology to deploy applications, including but not limited to:

  • More agile application creation and deployment
  • Portability
  • Environmental consistency
  • Loose coupling and distribution
  • Resource isolation
  • Efficient and high-density resource utilization

Kubernetes also provides powerful capabilities for application management, scheduling, monitoring, and O & M:

  • Service discovery and load balancing capabilities
  • Automatic deployment and rollback of applications
  • Autonomous repair capability of application
  • Storage orchestration capability
  • Key and configuration management capabilities

But Kubernetes also has many functions that are not provided but allowed to be extended, such as log collection, monitoring and alarm capabilities. The following figure shows Alibaba cloud container service, which is based on the support of standard Kubernetes, has enhanced and upgraded the capabilities closely related to users, including providing the most flexible and low-cost global access capability, strong security architecture support capability, and the ability to deeply integrate Alibaba cloud's basic resource services. After the double 11 Massive user experience has been verified and precipitated, and various product forms such as proprietary, hosting, no service, edge and dragon bare metal are supported. All the demonstrations we have done later today are on this platform.

Boundary of application delivery

What are the boundaries of application delivery in Kubernetes?

From a simple point of view, we can think that the delivery of application is its network service mode, the back-end resources of service and the persistent storage of business data. These resources are abstracted into service, deployment/pod, volume resources, etc.

Taking a wordpress application as an example, it includes two functional components: the front-end component handles user requests and the back-end component stores data. The front-end component includes a frontend service and three pods, and the back-end component includes a backend service and a pod component. Therefore, the resources delivered by this wordpress application are two services and a total of four back-end pods. This backend pod resource is managed by deployment in Kubernetes. The service resource is equivalent to a load balancer, which routes requests to the backend pod. It involves calls between components in the cluster and external users accessing services in the cluster, so there are different types of division.

According to the different ways of service exposure, it can be divided into the following categories:

ClusterIP

By assigning a fixed virtual IP (Cluster IP) that can be accessed within the cluster to Kubernetes' Service, the access within the cluster can be realized. Is the most common way.

apiVersion: v1
kind: Service
metadata:
  name: wordpress
spec:
  type: ClusterIP      # The default service type. The service is only accessible within the cluster
  ports:
    - port: 80         # Exposed to the service port inside the cluster
      targetPort: 80   # Service port on which the container listens
      protocol: TCP
  selector:
    app: wordpress     # Forward request to backend pod with same label

NodePort

NodePort maps the service port to a port of the cluster node. If you do not specify this port, the system will select a random port. Most of the time we should let Kubernetes choose the port. It's too expensive for users to choose the available port themselves.

apiVersion: v1
kind: Service
metadata:
  name: wordpress
spec:
  type: NodePort       # NodePort service type. The service exposes a fixed static port for external access of the cluster
  ports:
    - port: 80         # Exposed to the service port inside the cluster
      targetPort: 80   # Service port on which the container listens
      protocol: TCP
      nodePort: 31570  # Services can be accessed through this port outside the cluster
  selector:
    app: wordpress     # Forward request to backend pod with same label

Although the NodePort method can expose services to access outside the cluster, it also has many disadvantages:

  • Each port can only be one service
  • Port range is limited, generally 30000-32767
  • If the IP address of the node changes, you need to make some changes to adapt

So this method is not recommended in production, but it can be used if your application is more cost sensitive and can tolerate service unavailability window.

LoadBalancer

LoadBalancer is a standard way for services to be exposed outside the cluster or on the public network, but it depends on the capability of a load balancer provided by cloud provider. The load balancer will allocate an ip address separately and listen to the specified port of the back-end service. The requested traffic will be forwarded to the corresponding service of the back-end through the specified port.

apiVersion: v1
kind: Service
metadata:
  name: wordpress
spec:
  type: LoadBalancer       # The LoadBalancer service type generally depends on the load balancing capability provided by the public cloud manufacturer
  ports:
    - port: 80         # Exposed to the service port inside the cluster
      targetPort: 80   # Service port on which the container listens
      protocol: TCP
  selector:
    app: wordpress     # Forward request to backend pod with same label

Ingress

The type of ClusterIP service is limited to intra cluster communication. NodePort can expose the service access portal, but each node will occupy a port, which will increase the complexity of port management. LoadBalancer usually needs the support of a third-party cloud provider, which has certain constraints. The Ingress service type is different from the three service types in front of us. In fact, it is not a service type, but similar to the existence of a cluster service portal. It can route traffic to the corresponding back-end services based on different paths or subdomains you have configured, more like an "intelligent routing" service.

The resource types involved in application publishing and several modes of service resource types are introduced. How can service find the corresponding back-end pod? This is the function of the tag. We can tag each application's pod and service with the same tag. The mechanism of the tag is the key point of several application publishing strategies we will talk about later.

Publishing strategy applied

In the Kubernetes cluster, in addition to selecting the service exposure mode according to the business needs, it is very important to choose a correct publishing strategy in order to make the application provide services smoothly during the upgrade.

Rolling publication

The first application publishing strategy is rolling publishing, which is also a common strategy. It is to gradually deploy the new version of the application by replacing each instance one by one until all instances are replaced.

As shown in the figure below, the current service version provided by my application is v1. There are three copies of the back end of this service, but when I update version V2, it is a copy and a sub local start to replace until all the back ends of the final service are replaced with v2.

The arrangement file of an application example is as follows:

  • go-demo-v1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: go-demo
  template:
    metadata:
      labels:
        app: go-demo
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: go-demo
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
  type: ClusterIP
  • Deployment version v1
$ kubectl apply -f go-demo-v1.yaml
  • View pod running status
$ kubectl get po
NAME                       READY   STATUS    RESTARTS   AGE
go-demo-78bc65c564-2rhxp   1/1     Running   0          19s
go-demo-78bc65c564-574z6   1/1     Running   0          19s
go-demo-78bc65c564-sgl2s   1/1     Running   0          19s
  • Access to application services
$ while sleep 0.1; do curl http://172.19.15.25; done
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
  • Update go-demo-v1.yaml to go-demo-v2.yaml and update the image tag
...
registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v2
...
  • Deployment version v2
$ kubectl apply -f go-demo-v2.yaml
  • You can see that the pod will be replaced by the new version of pod one by one
$kubectl get po -w
NAME                                READY   STATUS              RESTARTS   AGE
application-demo-8594ff4967-85jsg   1/1     Running             0          3m24s
application-demo-8594ff4967-d4sv8   1/1     Terminating         0          3m22s
application-demo-8594ff4967-w6lpz   0/1     Terminating         0          3m20s
application-demo-b98d94554-4mwqd    1/1     Running             0          3s
application-demo-b98d94554-ng9wx    0/1     ContainerCreating   0          1s
application-demo-b98d94554-pmc5g    1/1     Running             0          4s
  • Access service will find that in the process of application rolling upgrade, version v1 and v2 will be accessed. The time depends on the startup speed of the application
$ while sleep 0.1; do curl http://172.19.15.25; done
Version: v1
Version: v2
Version: v1
Version: v1
Version: v2
Version: v1
Version: v1
Version: v2

The advantage of rolling publishing is that it is relatively simple and does not take up too much computing resources. The disadvantages are:

  • Versions are slowly replaced between instances
  • This rolling release may take a while
  • Unable to control flow

In terms of the end state of the application in the cluster, there is only the application backend of version 1 or the backend of version 2 in the cluster. If version 2 is defective, the online service is applied to the overall user. Although we have a mechanism to roll back quickly, the cost of involving the overall user failure is too high.

Blue green release

The second is the blue-green release. The blue / Green release is that the back-end pods of application version 1 and version 2 are deployed in the environment, and the release version is determined by controlling the traffic switch. Compared with the rolling release, the application end state under the blue-green release strategy can have version 1 and version 2 pods at the same time. We can determine which version of the backend the current service uses by switching the service traffic.

The orchestration file for an application example is shown below.

  • go-demo-v1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v1
spec:
  replicas: 4
  selector:
    matchLabels:
      app: go-demo
      version: v1
  template:
    metadata:
      labels:
        app: go-demo
        version: v1
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • go-demo-v2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v2
spec:
  replicas: 4
  selector:
    matchLabels:
      app: go-demo
      version: v2
  template:
    metadata:
      labels:
        app: go-demo
        version: v2
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • service.yaml
apiVersion: v1
kind: Service
metadata:
  name: go-demo
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
    version: v1
  type: ClusterIP
  • Deploy the above three resources
$ kubectl apply -f go-demo-v1.yaml -f go-demo-v2.yaml -f service.yaml
  • Access service can see that currently only version 1 services are accessed
$ while sleep 0.1; do curl http://172.19.8.137; done
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
  • Modify spec.selector lower version=v2 of service.yaml
apiVersion: v1
kind: Service
metadata:
  name: go-demo
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
    version: v2
  type: ClusterIP
  • Redeployment
$ kubectl apply -f service.yaml
  • Revisit the service and you can see that you've quickly switched to version 2
$ [root@iZbp13u3z7d2tqx0cs6ovqZ blue-green]# while sleep 0.1; do curl http://172.19.8.137; done
Version: v2
Version: v2
Version: v2

We just mentioned that the rolling upgrade process takes time. Even if it is rolled back, it will take a certain time to complete the roll back. In the case of application defects in the new version, the blue-green release strategy can be quickly implemented in v1 and v2 Before the two versions, the traffic is cut, so the time of switching traffic is much shorter than that of rolling upgrade. But the disadvantage of blue-green publishing is the same as that of rolling publishing. This defect will affect the whole user. The service will either switch to version 2 or to version 1 100%, which is a non-zero or 100 Even though the blue-green release strategy can greatly shorten the recovery time, it is not acceptable in some scenarios. In addition, there are two versions of pod replicas in the cluster environment at the same time, and the resource consumption is twice of that of rolling publishing.

Canary release (grayscale release)

The third publishing strategy to be introduced is Canary publishing. Canary deployment is that application version 1 and version 2 are deployed in the environment at the same time, and user requests may be routed to the back end of version 1 or version 2, so that some new users can access version 2 applications. In this release strategy, we can gradually control the application switch to the new version by adjusting the traffic percentage. Compared with the blue-green deployment, it not only inherits the advantages of the blue-green deployment, but also takes up more than twice the resources required by the blue-green deployment. In the case of the new version with defects, it only affects a small number of users and minimizes the loss.

For the concept of gray-scale publishing, some people think that it is a different thing from Canary publishing. It is released in the same way as canary, but for different purposes:

  • Canary release is more likely to get some feedback from users quickly. For example, I may not be sure whether the user experience of this new version of my function can be well accepted by the public. I hope to get some timely feedback from online users, and then iterate v3 version after the functional experience adjustment on the product side;
  • The gray-scale release is that my product function has been designed and developed very well. Now it is necessary to gradually replace the old version online, but to control the risk that the release may bring, so the gray-scale release is required.

Example application 1 is as follows. In this example, we control the flow proportion by the number of pod s.

  • go-demo-v1.yaml set the number of copies to 9
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v1
spec:
  replicas: 9
  selector:
    matchLabels:
      app: go-demo
      version: v1
  template:
    metadata:
      labels:
        app: go-demo
        version: v1
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • go-demo-v2.yaml set the number of copies to 1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-demo
      version: v2
  template:
    metadata:
      labels:
        app: go-demo
        version: v2
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • service.yaml
apiVersion: v1
kind: Service
metadata:
  name: go-demo
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
  type: ClusterIP
  • Deploy the above three resources
$ kubectl apply -f go-demo-v1.yaml -f go-demo-v2.yaml -f service.yaml
  • Access to the service can see that basically 10% of the traffic is switched to version 2
$ while sleep 0.1; do curl http://172.19.8.248; done
Version: v1
Version: v2
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1

In addition, we can use nginx address controller to control traffic switching, which is more accurate.

  • go-demo-v1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: go-demo
      version: v1
  template:
    metadata:
      labels:
        app: go-demo
        version: v1
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • go-demo-v2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-demo-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-demo
      version: v2
  template:
    metadata:
      labels:
        app: go-demo
        version: v2
    spec:
      containers:
      - name: go-demo
        image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/go-demo:v2
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  • service-v1.yaml
apiVersion: v1
kind: Service
metadata:
  name: go-demo-v1
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
    version: v1
  type: ClusterIP
  • service-v2.yaml
apiVersion: v1
kind: Service
metadata:
  name: go-demo-v2
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: go-demo
  selector:
    app: go-demo
    version: v2
  type: ClusterIP
  • ingress.yaml, set nginx.ingress.kubernetes.io/service-weight: | go demo-v1: 100, go demo-v2: 0, version 1 - 100% traffic, version 2 - 0% traffic
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-weight: |
        go-demo-v1: 100, go-demo-v2: 0
  name: go-demo
  labels:
    app: go-demo
spec:
  rules:
    - host: go-demo.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: go-demo-v1
              servicePort: 80
          - path: /
            backend:
              serviceName: go-demo-v2
              servicePort: 80
  • Deploy the above 4 Resources
$ kubectl apply -f go-demo-v1.yaml -f go-demo-v2.yaml -f service-v1.yaml -f service-v2.yaml -f nginx.yaml
  • Access to the service can see 100% of the traffic to version 1
$ while sleep 0.1; do curl http://go-demo.example.com; done
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
Version: v1
  • Update ingress.yaml and set the flow ratio to 50:50
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-weight: |
        go-demo-v1: 50, go-demo-v2: 50
  name: go-demo
  labels:
    app: go-demo
spec:
  rules:
    - host: go-demo.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: go-demo-v1
              servicePort: 80
          - path: /
            backend:
              serviceName: go-demo-v2
              servicePort: 80
  • Visit the service and you can see that the traffic is 50% to version 1 and 50% to version 2
$ while sleep 0.1; do curl http://go-demo.example.com; done
Version: v2
Version: v1
Version: v1
Version: v1
Version: v2
Version: v2
Version: v1
Version: v1
Version: v2
Version: v2
  • Update ingress.yaml and set the traffic ratio to 0:100
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/service-weight: |
        go-demo-v1: 0, go-demo-v2: 100
  name: go-demo
  labels:
    app: go-demo
spec:
  rules:
    - host: go-demo.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: go-demo-v1
              servicePort: 80
          - path: /
            backend:
              serviceName: go-demo-v2
              servicePort: 80
  • Access to the service can see 100% of the traffic to version 2
$ while sleep 0.1; do curl http://go-demo.example.com; done
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2
Version: v2

Whether it's Canary release or grayscale release, the disadvantage is that the release cycle is relatively slow.

Among these publishing strategies,

  • When you update and release the application in the development and test environment, use rolling release;
  • In the production environment, rolling update or blue-green release can be used when the new version has been fully tested in advance;
  • If the update of the new version of the application needs to control the risk to the maximum extent and reduce the impact of the fault on users, then use Canary release or gray release.

The above is the introduction of several publishing strategies we often use in Kubernetes.

"Alibaba cloud native Pay attention to microservices, Serverless, containers, Service Mesh and other technology fields, focus on cloud native popular technology trends, cloud native large-scale landing practice, and become the technology circle that most understands cloud native developers. "

Tags: Kubernetes curl Nginx network

Posted on Wed, 11 Mar 2020 21:45:38 -0700 by Plasma