Detailed explanation and implementation of LVS + preserved to realize high available load balancing cluster

Cluster technology overview:
LB (load balancing cluster): LVS, Haproxy, Nginx, F5 BigIP
HA (high availability cluster): maintained, RFCS, Pacemaker, Heartbeat
HP (high performance cluster): Hadoop,spark
Keepalived:
Keepalived is the next lightweight and highly available solution for linux. Similar to the functions implemented by heartbeat, it can achieve high availability of services or networks. Heartbeat is powerful, but deployment and use are relatively cumbersome. Compared with heartbeat, keepalived achieves high availability mainly through virtual route redundancy. Its configuration is relatively simple.
Keepalived is specially used to monitor the status of each service node in the cluster system, and detect the status of each service node according to the exchange mechanism of layer 3-5. If a service node has an exception or a work failure, keepalived can detect it, remove the failed node, and automatically rejoin the service node after the failure node recovers Server cluster.
VRRP (Virtual Router Redundancy Protocol)
The purpose of vrrp is to solve the problem of light point fault in the static router. The network can run continuously and stably through vrrp.
VRRP can virtualize two or more physical server devices into a virtual router. This virtual router provides external services through virtual IP (can be multiple), but within the virtual router, multiple physical routers work together. Generally, only one physical server provides external services at the same time. This physical server is called MASTER. MASTER is generally generated by the election algorithm, and provides external services through its virtual IP. When MASTER fails, its virtual IP will be bound to BACKUP, which will provide external services. So when does BACKUP know that MASTER is out of order? This depends on the heartbeat detection between the MASTER and the standby. MASTER tells BACKUP every once in a while that its information indicates that it is alive. If BACKUP hasn't received MASTER's alive message for a long time, it will know that MASTER has hung up and needs to switch it for service.
How does the above constitute a group of virtual routers? It is because every virtual router has a unique ID, called VRID. A VRID and a group of IP addresses constitute a virtual router. In VRRP, all messages are sent in the form of IP multicast. So in a virtual router, MASTER will always send VRRP packets, and the router in BACKUP only accepts the packets sent by MASTER. At the same time, monitor the operation status of MASTER.
When there is no labeled router as MASTER, many physical routers generally select MASTER by priority. If the priority is the same, the router with large IP will be used as MASTER.
How keepalived works:
keepalived achieves high availability through vrrp, and can monitor the running state of servers in the cluster and isolate faults.
keepalived works in the network layer, transport layer and application layer of TCP/IP reference model.
Keepalived sends an ICMP packet to each node of the server cluster through ICMP at the network layer. If a node does not respond to the packet, it is considered that the node has failed, and keepalived will remove the node.
Keepalived uses TCP port connection and scanning technology in the transport layer to determine whether the cluster nodes are normal. For example, the default web port is 80, and the ssh port is 22. Once keepalived detects that these ports do not respond to the packet return in the transport layer, it is considered that these ports are abnormal, and then forces the corresponding nodes of this port to be removed from the server cluster group.
The application layer is used to run various application services. Users can write programs to run keepalived. Keepalived can check whether various programs or services are allowed to be normal according to the user settings. If the monitoring results of keepalived are inconsistent with the user settings, keepalived will remove the corresponding services from the server.
Architecture of keepalived

keepalived architecture is divided into two layers: user layer and kernel layer.
The kernel layer is at the bottom, including IPVS and NETLINK. IPVS module is a third-party module introduced by keepalived to implement IP based load balancing cluster. The Director in LVS usually needs to install the software, which is used to select a physical server from the back-end server to respond to customer requests.
NETLINK is mainly used to implement some advanced routing framework and some related network functions, and complete various network requests sent by Netlink Reflector module in user control layer.
The user layer has four parts: Scheduler I/O Multiplexer, Memory Management (Mngt), Control Plane and Core components. Among them, the Scheduler I/O Multiplexer is an I/O multiplexing scheduler, and the load arranges all internal requests of keepalived. Memory Management (Mngt) provides some common ways to access memory. Control Plane can compile and parse configuration files.
Core components is the core component of keepalived, which contains some functional modules, such as WatchDog, checkers, VRRP Stacks, IPVS wrapper and Netlink Reflector. Among them, WatchDog sets a counter and a threshold value for the detected target. WatchDog will increase the count value by itself, and then wait for the monitored target to reset the count value periodically. Once an error occurs to the monitored target and the counter value cannot be reset, WatchDog will detect it and take corresponding recovery measures. Checkers can detect the running state of the server and isolate the faults. VRRP STack can realize failover in HA cluster. IPVS wrapper is the implementation of IPVS function. Netlink Reflector is used to set and switch virtual IP for failover in high availability cluster.
Preparation before construction:
1. Prepare four virtual machines first. Two as the master-slave structure (or the master-slave structure) of preserved, and two as RS
2. keepalived is ready to be in DR mode, because DIP and VIP in NAT mode need failover, which is relatively troublesome, so we will not demonstrate it here first.
     3,DIP: 192.168.0.121 192.168.0.127
     4,VIP: 192.168.0.100
     5,RIP: 192.168.0.122 192.168.0.124
6. Ensure that the time of all four machines is synchronized, and all firewalld and selinux are closed ()
Set up:
1. First, configure two RS, install httpd, and add index.html

yum install httpd
echo "This is 122 rs1 web page" > /var/www/html/index.html
echo "This is 124 rs2 web page" > /var/www/html/index.html

2. Configure ARP ignore and ARP announce (both RS1 and RS2 require operation)

echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/ens33/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/ens33/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

3. Bind VIP in lo interface and add route (both RS1 and RS2 operate)

ifconfig lo:10 192.168.0.100 netmask 255.255.255.255 broadcast 192.168.0.100 up
route add -host 192.168.0.100 dev lo:10

4. Start httpd service, test two RS first

[root@dr ~]# elinks -dump http://192.168.0.122
                            This is 122 rs1 web page
[root@dr ~]# elinks -dump http://192.168.0.124
                         This is 124 rs2 web test page

5. Two schedulers installed keepalived

yum install keepalived

6. Configure scheduler

! Configuration File for keepalived

global_defs {                    #Global configuration
   notification_email {        #Set alarm email address
     gaofan@localhost        #This address can have multiple
   }
   notification_email_from root@localhost    #Mailing address
   smtp_server 127.0.0.1                        #Address of mail server
   smtp_connect_timeout 30        #Set the timeout for connecting to the SMTP server
   router_id LVS_DEVEL1                #An ID of the server running keepalived, information in the subject of the message
}
vrrp_script chk_mt {            #It can realize the monitoring of cluster functions, the monitoring of cluster resources and the change of priority, and then realize the switch between the active and standby
  script "[[ -f /etc/keepalived/down  ]] && exit 1 || exit 0"
  interval 1
  weight -2
}
vrrp_instance VI_1 {        #Example 1
    state MASTER            #Table name this machine is a MASTER server
    interface ens33        #Detected network interface
    virtual_router_id 61        #Virtual routing ID. under the same VRRP? Instance, MASTER and BACKUP must be consistent
    priority 100                    #Define priorities
    advert_int 1                    #Set the health detection time between MASTER and BACKUP, 1s by default
    authentication {        #Type and password of authentication
        auth_type PASS    #type
        auth_pass 1111       #Password. The primary and secondary passwords must be the same. Generally, a string of numbers is randomly generated through openssl
    }
    virtual_ipaddress {        #Set virtual IP address
        192.168.0.100/32 dev ens33 label ens33:0
    }
    track_script{
       chk_mt
    }
}

virtual_server 192.168.0.100 80 {        #Virtual server
    delay_loop 6        #Set operation inspection time in s
    lb_algo rr                    #Set the load scheduling algorithm. Here, polling is used
    lb_kind DR                    #Load balancing mechanism
    nat_mask 255.255.255.255    #Subnet mask for virtual IP
#   persistence_timeout 50        #Session hold time
    protocol TCP            #Forwarding protocol, divided into TCP and UDP
    sorry_server 127.0.0.1 80        #If RS all breaks down, I will be a sorry server, better than no access, no embarrassment!
    real_server 192.168.0.122 80 {    #RS1 definition
        weight 1            #Weight, the larger, the higher the load of the server
        HTTP_GET {        #In the state detection setting part of RS, the available methods are listed below
            url {                    #Use here to detect whether the status code of the default Index.html under RS is 200
      path /
                status_code 200
            }
            connect_timeout 3        #Timeout time
            nb_get_retry 3            #retry count
            delay_before_retry 3    #retry interval
        }
    }
         
    real_server 192.168.0.124 80 {
        weight 2
        HTTP_GET {
            url {
              path /
                status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

Copy the configuration file to BACKUP scheduler, and modify the following two places:
state BACKUP
priority 99 (lower than MASTER)
Then install httpd on two schedulers and configure the sorry page.
The detection of RS in the above can be as follows:
1. HTTP get: using status code
2. Use default TCP "check
3. MD5 encryption
You need to use the genhash command to get the MD5 code of a page of an RS
For example:

genhash -s 192.168.0.122 -p 80 -u /test.html
MD5SUM = f5ac8127b3b6b85cdc13f237c6005d80

Then the configuration is changed to:

HTTP_GET{
    url{
        path /test.html        digest f5ac8127b3b6b85cdc13f237c6005d80
    }
}

7. Start the keepalived of both dispatchers.
8. View the VIP and ipvs rules, where both the ipvs rules and the scheduler rules will be generated.

[root@dr keepalived]# ip addr list
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:40:8e:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.121/24 brd 192.168.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.0.100/32 scope global ens33:0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe40:8e97/64 scope link
       valid_lft forever preferred_lft forever
[root@dr keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.100:80 rr
  -> 192.168.0.122:80             Route   1      0          0         
  -> 192.168.0.124:80             Route   2      0          0         

9. Test with another scheduler

[root@ga keepalived]# for ((i=1;i<=10;i++))
> do
> elinks -dump http://192.168.0.100
> done
                         This is 124 rs2 web test page
                            This is 122 rs1 web page
                         This is 124 rs2 web test page
                            This is 122 rs1 web page
                         This is 124 rs2 web test page
                            This is 122 rs1 web page
                         This is 124 rs2 web test page
                            This is 122 rs1 web page
                         This is 124 rs2 web test page
                            This is 122 rs1 web page

10. Reduce the priority of the MASTER, because there is detection in my configuration file, so when there is a down file in the keepalived directory, the priority of the MASTER will be reduced by 2, so that the VIP will float to the dispatcher 2
. Check again. You can see the same effect as above, and you can also grab packets for testing.
11. Turn off the two RS's and test whether they will get the sorry page.

[root@dr keepalived]# elinks -dump http://192.168.0.100
   Sorry,I am 127 sorry page

12. Add:
VRRP script and track script are used together in the above configuration.
There are four common types:
1. Use the kill command. You can use the kill-0 httpd command. A signal of 0 indicates the monitoring of the running state of the program or process. If the process is found to be closed or other exceptions are found, the status code 1 will be returned. Otherwise, if the process is normal, the status code 0 will be returned
2. Check the operation status of port "< / dev / TCP / 127.0.0.1/80" indicates the status detection of port 80 of this machine
3. Through shell detection
4. Monitor through script

Published 29 original articles, won praise 2, visited 7648
Private letter follow

Tags: network yum Nginx Hadoop

Posted on Wed, 12 Feb 2020 03:48:01 -0800 by Schism