K8s Deployment--Introduction to k8s and etcd Cluster Creation

Introduction to Kubernetes

  • Kubernetes is an open source container cluster management system for Google in 2014. Kubernetes is referred to as K8S.Kubernetes v1.0 was officially released in July 2015, and the latest stable version so far is v1.9
  • K8S is used to deploy, extend, and manage containerized applications.
  • K8S provides a range of functions such as container marshalling, resource scheduling, resilient scaling, deployment management, service discovery, and so on.
  • Grouping applications into logical units by using the concepts of "labels" and "pods."
  • The goal of Kubernetes is to make deploying containerized applications simple and efficient.
  • Official website: http://www.kubernetes.io

The role of Kubernetes

  • Automatically deploy, extend, and manage container applications
  • Resource Scheduling
  • Deployment Management
  • Service Discovery
  • Expansion and contraction
  • Monitor

Kubernetes feature

  • Self-repair
    • Restart failed containers in case of node failure, replace and redeploy them to ensure the expected number of copies; kill containers that fail health checks, and do not process client requests until ready to ensure online services are not interrupted.
  • Elastic Stretch
    • Use commands, UI, or CPU-based usage to automatically and quickly expand and scale application instances to ensure high availability during peak and concurrent business hours; recycle resources during peak business hours to run services at a minimum cost.
  • Automatic Deployment and Rollback
    • K8S uses a roll-over strategy to update applications - one Pod at a time, instead of deleting all Pods at the same time. If problems occur during the update process, the changes will be rolled back to ensure that the upgrade is not affected by the business.
  • Service Discovery and Load Balancing
    *K8S provides a unified access entry (internal IP address and DNS name) for multiple containers, and all containers associated with load balancing eliminates the need for users to consider container IP issues.
  • Confidentiality and Configuration Management
    • Manage confidential data and application configurations without exposing sensitive data in a mirror to improve the security of sensitive data.Some commonly used configurations can be stored in K8S for easy application use.
  • Storage Layout
    • Mounting external storage systems, whether from local storage, public clouds (such as AWS), or network storage (such as NFS, GlusterFS, Ceph) is used as part of cluster resources, greatly increasing storage usage flexibility.
  • Batch Processing
    • Provides one-time tasks, scheduled tasks, and scenarios for batch data processing and analysis.

Kubernetes Cluster Architecture and Components

Master Component

  • kube-apiserver
    • The Kubernetes API, the unified entry to the cluster, the coordinator of each component, provides interface services with the RESTful API. All object resource addition, deletion, monitoring and monitoring operations are handed over to APIServer before being submitted to Etcd storage.
  • kube-controller-manager
    Handles the common background tasks in a cluster - one resource for each controller, and Controller Manager is responsible for managing these controllers.
  • kube-scheduler
    • Depending on the scheduling algorithm, a Node node is selected for the newly created od, which can be deployed arbitrarily, either on the same node or on different nodes.
  • etcd
    • Distributed Key Value Storage System.Used to hold cluster state data, such as object information such as Pod, Service, etc.

Node Component

  • kubelet
    • kubelet is Master's Agent on the Node node, which manages the lifecycle of native running containers, such as creating containers, mounting data volumes on Pods, downloading secret s, getting containers and node status, and so on.kubelet converts each Pod into a set of containers.
  • kube-proxy
    • Implement Pod network proxy on Node node, maintain network rules and four-tier load balancing.
  • docker or rocket
    • Container engine, run container.

kubernetes core concepts

  • Pod
    • Minimum Deployment Unit
    • A collection of containers
    • A container in a Pod shares a network namespace
    • Pod is transient
  • Controllers
    • ReplicaSet: Ensure the expected number of Pod copies
    • Deployment: Stateless application deployment
    • StatefulSet: Stateful application deployment
    • DaemonSet: Make sure all Node s run the same Pod
    • Job: One-Time Task
    • Cronjob: Timed task
    • Higher Level Objects, Deploy and Manage Pod s
  • Service
    • Prevent Pod from becoming disconnected
    • Define access policies for a set of Pod s
  • Label: Label, attached to a resource, used to associate objects, query, and filter
  • Namespaces: Namespaces, which logically isolates objects
  • Annotations: Notes

Environment planning for kubernertes platform

  • Single node deployment plan
  • Deployment Steps
    • 1: Self-signed ETCD certificate
    • 2:ETCD Deployment
    • 3:Node Install docker
    • 4:Flannel deployment (write subnet to etcd first)
  • master
    • 5: Self-signed APIServer certificate
    • 6: Deploy APIServer components (token, csv)
    • 7: Deploy controller-manager (specify apiserver certificate) and scheduler components
  • node
    • 8: Generate kubeconfig (bootstrap, kubeconfig, and kube-proxy.kubeconfig)
    • 9: Deploy kubelet components
    • 10: Deploy kube-proxy components
  • Join Cluster
    • 11:kubectl get CSR & & kubectl certificate approve Allow method certificates to join the cluster
    • 12: Add a node
    • 13: View the kubectl get node node

Deployment experiment

Experimental environment

  • Master01:192.168.80.12
  • Node01:192.168.80.13
  • Node02:192.168.80.14

etcd component deployment

Introduction to etcd

  • Etcd is an open source project initiated by the CoreOS team in June 2013 with the goal of building a highly available distributed key-value database.Within etcd, raft protocol is used as a consistency algorithm, and etcd is based on Go language.
  • As a service discovery system, etcd has the following characteristics:
    • Simple: Easy to install, configure, interact with, and use the HTTP API
    • Security: Supports SSL certificate validation
    • Fast: According to officially provided benchmark data, a single instance supports 2k+read operations per second
    • Reliable: Use raft algorithm for distributed system data availability and consistency

Master 01 server operation

  • Self-signed etcd component certificate
    [root@master01 ~]# systemctl stop firewalld.service //close firewall
    [root@master01 ~]# setenforce 0 //close selinux
    [root@master01 ~]# mkdir k8s //Create k8s directory
    [root@master01 ~]# ls
    anaconda-ks.cfg  k8s
    [root@master01 ~]# Mount.cifs //192.168.80.2/shares/K8S/k8s01/mnt/ //Mount packages prepared in the host machine
    Password for root@//192.168.80.2/shares/K8S/k8s01:
    [root@master01 ~]# cd /mnt/
    [root@master01 mnt]# ls
    etcd-cert     etcd-v3.3.10-linux-amd64.tar.gz     k8s-cert.sh                           master.zip
    etcd-cert.sh  flannel.sh                          kubeconfig.sh                         node.zip
    etcd.sh       flannel-v0.10.0-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz
    [root@master01 mnt]# Cd/root/k8s/ //back to k8s directory
    [root@master01 k8s]# vim cfssl.sh //Edit script to download cfssl official package as a ca-certified package
    curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
    curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
    curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
    chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
    :wq
    [root@master01 k8s]# bash cfssl.sh //Execute script to download official cfssl package
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
    100  9.8M  100  9.8M    0     0   457k      0  0:00:22  0:00:22 --:--:--  581k
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
    100 2224k  100 2224k    0     0   300k      0  0:00:07  0:00:07 --:--:--  517k
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
    100 6440k  100 6440k    0     0   276k      0  0:00:23  0:00:23 --:--:--  221k
    [root@master01 k8s]# Ls/usr/local/bin/ //Check to see if the certificate was successfully downloaded
    cfssl  cfssl-certinfo  cfssljson
    [root@master01 k8s]# mkdir etcd-cert //Create Certificate Storage Directory
    [root@master01 k8s]# ls
    etcd-cert
    [root@master01 k8s]# cd etcd-cert/ //Enter certificate store directory
    [root@master01 etcd-cert]# Cat > ca-config.json < < EOF //Define a CA certificate
    > {
    >   "signing": {
    >     "default": {
    >       "expiry": "87600h"          //Certificate invalidation
    >     },
    >     "profiles": {
    >       "www": {
    >          "expiry": "87600h",
    >          "usages": [
    >             "signing",
    >             "key encipherment",
    >             "server auth",        //Server-side authentication
    >             "client auth"         //Client Validation
    >         ]
    >       }
    >     }
    >   }
    > }
    > EOF
    [root@master01 etcd-cert]# Cat > ca-csr.json < < EOF // Implement Certificate Signature
    > {
    >     "CN": "etcd CA",
    >     "key": {
    >         "algo": "rsa",         //Use asymmetric keys
    >         "size": 2048           //Key Length
    >     },
    >     "names": [
    >         {
    >             "C": "CN",             //Identity information, customizable
    >             "L": "Beijing",
    >             "ST": "Beijing"
    >         }
    >     ]
    > }
    > EOF
    [root@master01 etcd-cert]# Cfssl gencert-initca-csr.json | cfssl json-bare ca - //Use command to generate ca certificate
    2020/02/09 16:53:08 [INFO] generating a new CA key and certificate from CSR
    2020/02/09 16:53:08 [INFO] generate received request
    2020/02/09 16:53:08 [INFO] received CSR
    2020/02/09 16:53:08 [INFO] generating key: rsa-2048
    2020/02/09 16:53:08 [INFO] encoded CSR
    2020/02/09 16:53:08 [INFO] signed certificate with serial number 400787333165311350366024741004548366561538833100
    [root@master01 etcd-cert]# ls
    ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem    //ca certificate generated successfully
    [root@master01 etcd-cert]# Cat > server-csr.json < < EOF //Specify communication validation between three etcd nodes
    > {
    >     "CN": "etcd",
    >     "hosts": [
    >     "192.168.80.12",         //Cluster IP address settings, master address
    >     "192.168.80.13",         //node01IP address
    >     "192.168.80.14"          //Noe02IP address
    >     ],
    >     "key": {
    >         "algo": "rsa",
    >         "size": 2048
    >     },
    >     "names": [
    >         {
    >             "C": "CN",
    >             "L": "BeiJing",
    >             "ST": "BeiJing"
    >         }
    >     ]
    > }
    > EOF
    [root@master01 etcd-cert]# Cfssl gencert-ca=ca.pem-ca-key=ca-key.pem-config=ca-config.json-profile=www server-csr.json | cfssl json-bare server//Generate ETCD certificate server-key.pem server.pem
    2020/02/09 16:59:12 [INFO] generate received request
    2020/02/09 16:59:12 [INFO] received CSR
    2020/02/09 16:59:12 [INFO] generating key: rsa-2048
    2020/02/09 16:59:12 [INFO] encoded CSR
    2020/02/09 16:59:12 [INFO] signed certificate with serial number 155295832576786241095177900248601469934260652049
    2020/02/09 16:59:12 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
    websites. For more information see the Baseline Requirements for the Issuance and Management
    of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
    specifically, section 10.2.3 ("Information Requirements").
    [root@master01 etcd-cert]# ls
    ca-config.json  ca-csr.json  ca.pem      server-csr.json  server.pem
    ca.csr          ca-key.pem   server.csr  server-key.pem                   //Build Successful
  • Deploying etcd services
    [root@master01 etcd-cert]# Cd/mnt/ //Enter the directory the host mounted to
    [root@master01 mnt]# ls
    etcd-cert     etcd-v3.3.10-linux-amd64.tar.gz     k8s-cert.sh                           master.zip
    etcd-cert.sh  flannel.sh                          kubeconfig.sh                         node.zip
    etcd.sh       flannel-v0.10.0-linux-amd64.tar.gz  kubernetes-server-linux-amd64.tar.gz
    [root@master01 mnt]# CP etcd-v3.3.10-linux-amd64.tar.gz flannel-v0.10.0-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz etcd.sh/root/k8s//Copy the package and etcd execution script to the k8s working directory
    [root@master01 mnt]# Cd/root/k8s/ //back to k8s working directory
    [root@master01 k8s]# tar zvxf etcd-v3.3.10-linux-amd64.tar.gz //decompression etcd package
    etcd-v3.3.10-linux-amd64/
    etcd-v3.3.10-linux-amd64/Documentation/
    etcd-v3.3.10-linux-amd64/Documentation/platforms/
    etcd-v3.3.10-linux-amd64/Documentation/platforms/container-linux-systemd.md
    etcd-v3.3.10-linux-amd64/Documentation/platforms/aws.md
    etcd-v3.3.10-linux-amd64/Documentation/platforms/freebsd.md
    etcd-v3.3.10-linux-amd64/Documentation/rfc/
    ...
    [root@master01 k8s]# Mkdir/opt/etcd/{cfg, bin, ssl}-p //Recursively create etcd working directory
    [root@master01 k8s]# MV etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl/opt/etcd/bin//Copy the etcd command file to the bin directory in the working directory
    [root@master01 k8s]# Ls/opt/etcd/bin/ //view
    etcd  etcdctl
    [root@master01 k8s]# CP etcd-cert/*.pem/opt/etcd/ssl///Copy the certificate file to the etcd working directory SSL directory
    [root@master01 k8s]# Ls/opt/etcd/ssl/ //view
    ca-key.pem  ca.pem  server-key.pem  server.pem
    [root@master01 k8s]# bash etcd.sh etcd01 192.168.80.12 etcd02=https://192.168.80.13:2380,etcd03=https://192.168.80.14:2380//execute startup script etcd01 for master 01 server address etcd02, etcd03 for node01, and node02IP address. Later we will deploy etcd in node01 and node02 respectively to form etcd clusters, and script execution will generate etcd configuration files at the same time
    Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
    //After executing the startup script, it will enter a stuck state, waiting for other nodes to join, it also has a certain timeout time, beyond which errors will occur, do not care
  • Reopen a new session box

    [root@master01 ~]# Ps-ef | grep etcd //Check to see if the process is open
    root      16146      1  0 17:14 ?        00:00:00 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.80.12:2380 --listen-client-urls=https://192.168.80.12:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.80.12:2379 --initial-advertise-peer-urls=https://192.168.80.12:2380 --initial-cluster=etcd01=https://192.168.80.12:2380,etcd02=https://192.168.80.13:2380,etcd03=https://192.168.80.14:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
    root      16191  16160  0 17:15 pts/1    00:00:00 grep --color=auto etcd    //Successfully opened
    [root@master01 ~]# Scp-r/opt/etcd/ root@192.168.80.13:/opt/ //copy etcd working directory to node01 node
    The authenticity of host '192.168.80.13 (192.168.80.13)' can't be established.
    ECDSA key fingerprint is SHA256:Ih0NpZxfLb+MOEFW8B+ZsQ5R8Il2Sx8dlNov632cFlo.
    ECDSA key fingerprint is MD5:a9:ee:e5:cc:40:c7:9e:24:5b:c1:cd:c1:7b:31:42:0f.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '192.168.80.13' (ECDSA) to the list of known hosts.
    root@192.168.80.13's password:
    etcd                                                                       100%  509   495.7KB/s   00:00
    etcd                                                                       100%   18MB  98.7MB/s   00:00
    etcdctl                                                                    100%   15MB  95.0MB/s   00:00
    ca-key.pem                                                                 100% 1675     1.6MB/s   00:00
    ca.pem                                                                     100% 1265   416.6KB/s   00:00
    server-key.pem                                                             100% 1675     2.3MB/s   00:00
    server.pem                                                                 100% 1338     2.0MB/s   00:00
    [root@master01 ~]# Scp-r/opt/etcd/ root@192.168.80.14:/opt/ //copy etcd working directory to node02 node
    The authenticity of host '192.168.80.14 (192.168.80.14)' can't be established.
    ECDSA key fingerprint is SHA256:Ih0NpZxfLb+MOEFW8B+ZsQ5R8Il2Sx8dlNov632cFlo.
    ECDSA key fingerprint is MD5:a9:ee:e5:cc:40:c7:9e:24:5b:c1:cd:c1:7b:31:42:0f.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '192.168.80.14' (ECDSA) to the list of known hosts.
    root@192.168.80.14's password:
    etcd                                                                       100%  509   523.8KB/s   00:00
    etcd                                                                       100%   18MB  79.6MB/s   00:00
    etcdctl                                                                    100%   15MB 140.4MB/s   00:00
    ca-key.pem                                                                 100% 1675     1.9MB/s   00:00
    ca.pem                                                                     100% 1265   296.4KB/s   00:00
    server-key.pem                                                             100% 1675     2.4MB/s   00:00
    server.pem                                                                 100% 1338   423.3KB/s   00:00
    [root@master01 ~]# Scp/usr/lib/systemd/system/etcd.service root@192.168.80.13:/usr/lib/systemd/system//startup script copied to node01 node
    root@192.168.80.13's password:
    etcd.service                                                               100%  923   628.8KB/s   00:00
    [root@master01 ~]# Scp/usr/lib/systemd/system/etcd.service root@192.168.80.14:/usr/lib/systemd/system//startup script copied to node02 node
    root@192.168.80.14's password:
    etcd.service                                                               100%  923   684.8KB/s   00:00

    node01 server operation

  • Change the copied etcd profile

    [root@node01 ~]# systemctl stop firewalld.service //close firewall
    [root@node01 ~]# setenforce 0 //close selinux
    [root@node01 ~]# vim /opt/etcd/cfg/etcd
    #[Member] 
    ETCD_NAME="etcd02"                 //Change name to etcd02
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://192.168.80.13:2380 "//Change IP Address 192.168.80.13
    ETCD_LISTEN_CLIENT_URLS="https://192.168.80.13:2379 "//Change IP Address 192.168.80.13
    
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.80.13:2380 "//Change IP Address 192.168.80.13
    ETCD_ADVERTISE_CLIENT_URLS="https://192.168.80.13:2379 "//Change IP Address 192.168.80.13
    ETCD_INITIAL_CLUSTER="etcd01=https://192.168.80.12:2380,etcd02=https://192.168.80.13:2380,etcd03=https://192.168.80.14:2380 "//Note: No change here
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"
    :wq 
    [root@node01 ~]# Start etcd service directly after systemctl start etcd //edit is complete
    [root@node01 ~]# systemctl status etcd //View service status
    ● etcd.service - Etcd Server
    Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
    Active: active (running) since Day 2020-02-09 17:25:38 CST; 50s ago    //normal operation
    Main PID: 15905 (etcd)
    ...

    node02 server operation

  • Change the copied etcd profile

    [root@node02 ~]# systemctl stop firewalld.service //close firewall
    [root@node02 ~]# setenforce 0 //close selinux
    [root@node02 ~]# vim /opt/etcd/cfg/etcd
    #[Member]
    ETCD_NAME="etcd03"                                     //Change name to etcd03
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://192.168.80.14:2380 "//Change IP Address 192.168.80.14
    ETCD_LISTEN_CLIENT_URLS="https://192.168.80.14:2379 "//Change IP Address 192.168.80.14
    
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.80.14:2380 "//Change IP Address 192.168.80.14
    ETCD_ADVERTISE_CLIENT_URLS="https://192.168.80.14:2379 "//Change IP Address 192.168.80.14
    ETCD_INITIAL_CLUSTER="etcd01=https://192.168.80.12:2380,etcd02=https://192.168.80.13:2380,etcd03=https://192.168.80.14:2380 "//Note: No change here
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"
    :wq
    [root@node02 ~]# systemctl start etcd //start service
    [root@node02 ~]# systemctl status etcd //View status
    ● etcd.service - Etcd Server
    Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
    Active: active (running) since Day 2020-02-09 17:32:29 CST; 4s ago   //Run Successfully
    Main PID: 15926 (etcd)
    ...

    Back to master 01 server operation

  • Check Cluster Status
    [root@master01 k8s]# cd etcd-cert/ //Enter the certificate directory All to be viewed in the certificate store directory because you want to verify the view with a ca certificate
    [root@master01 etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem--key-file=server-key.pem --endpoints="https://192.168.80.12:2379, https://192.168.80.13:2379, https://192.168.80.14:2379" cluster-health //use directory to view cluster status
    member accc4008f61328 is healthy: got healthy result from https://192.168.80.13:2379
    member 88ef2b8e883800a0 is healthy: got healthy result from https://192.168.80.12:2379
    member fafd8a15257570ee is healthy: got healthy result from https://192.168.80.14:2379
    cluster is healthy        //Cluster created successfully

    etcd component deployed successfully

Tags: Linux Kubernetes JSON SSL

Posted on Sun, 09 Feb 2020 12:36:28 -0800 by realestninja