keepalived high availability cluster

Catalog

Cluster Cluster

Cluster type:

  1. LB lvs/nginx(http/upstream, stream/upstream)
  2. HA High Availability
    SPoF: Single Point of Failure
  3. HPC

Formulas for system availability:

A=MTBF/(MTBF+MTTR)A=MTBF/(MTBF+MTTR)A=MTBF/(MTBF+MTTR)

index: 99%, ..., 99.999%,99.9999%

Solutions to Improve System Usability and Reduce MTTR

Means: Redundancy redundant

  1. active/passive Main standby
  2. active/active Double main
  3. active --> HEARTBEAT --> passive
  4. active <–> HEARTBEAT <–> active

High availability is "service"

HA nginx service:
  vip/nginx process[/shared storage]
Resources: Components that make up a highly available service
(1) Number of passive nodes
(2) Resource switching

File sharing:

NAS: File Sharing Server;
SAN: Storage Area Network, Block Level Sharing

Network partition:

quorum: Quorum
with quorum: > total/2
without quorum: <= total/2

Cluster fissures:

In a high availability (HA) system, when two connected nodes are disconnected, the original system is divided into two independent nodes. At this time, the two nodes begin to compete for sharing resources, resulting in system chaos and data damage.

Some concepts:

Auxiliary devices: ping node, quorum disk
Failover: Failover: The operation of transferring resources to other nodes when the primary node of a resource fails
Failback: Failback: The process of retrieving resources that have previously been transferred to other nodes after the primary node of a resource has failed to modify its online status.

HA Cluster implementation scheme:

  1. AIS: Complete and complex application interface specification HA colony
    RHCS: Red Hat Cluster Suite Red Hat Cluster Suite
    heartbeat
    corosync
  2. VRRPProtocol Implementation: Virtual Routing Redundancy Protocol
    keepalived

keepalived:

Keepalived is a routing software written in C. The main goal of the project is to provide simple and powerful load balancing and high availability facilities for Linux systems and Linux-based infrastructure. The load balancing framework relies on the well-known and widely used Linux Virtual Server (IPVS) kernel module to provide Layer 4 load balancing.

Keepalived implements a set of checking programs to dynamically and adaptively maintain and manage load-balanced server pools according to their health. On the other hand, VRRP implements a high availability protocol. VRRP is the basis of router failover.

In addition, Keepalived implements a set of hooks for VRRP finite state machines, providing low-level and high-speed protocol interaction. In order to provide the fastest network fault detection, Keepalived implements the BFD protocol. VRRP state transition can consider BFD prompts to drive fast state transition. Keepalived frameworks can be used alone or together to provide a flexible infrastructure.

VRRP Protocol: Virtual Router Redundancy Protocol

Some terms:

Virtual Router: Virtual Router
Virtual Router Identity: VRID(0-255), Unique Identity Virtual Router
Physical router:
master: Main Equipment
backup: Standby Equipment
_priority: priority
VIP: Virtual IP
VMAC: Virutal MAC (00-00-5e-00-01-VRID)

Announcement:

Heart beat, priority, etc.
Periodicity

Operation mode:

Preemptive
non-preemptive

Authentication:

No certification
Simple Character Authentication: Pre-shared Key
MD5

Working mode:

Master/Standby: Single Virtual Router
Master/Master: Master/Reserve (Virtual Router 1), Reserve/Master (Virtual Router 2)

Functions:

vrrp protocol completes address flow
Generate ipvs rules for the node where the vip address resides (predefined in the configuration file)
Health status detection for each RS in ipvs cluster
Script-based call interface supports nginx, haproxy and other services by executing scripts to complete the functions defined in scripts, and then affect cluster transactions.

Components:

  1. Core components:

vrrp stack
ipvs wrapper
checkers

  1. Control components:

Configuration file analyzer

  1. IO multiplexer
  2. Memory Management Component

KeepAlived Composition

KeepAlived Implementation

  1. HA Cluster configuration preparation:
    (1) The time of each node must be synchronized
    (2) Ensure that iptables and selinux are not blocked
    (3) Nodes can communicate with each other by hostname
    (4) root users among nodes can communicate with each other based on ssh service of key authentication
  2. Keepalived installation:
yum -y install keepalived
  1. Procedural environment:
    Main configuration file: / etc/keepalived/keepalived.conf
    Environment Profile: / etc/sysconfig/keepalived

  2. KeepAlived configuration
    Configuration grammar:

[ka1]$ vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {   #Global configuration
    notification_email {   # Mail notification, keep alived malfunction, send mail
       root@localhost    #E-mail address
    }
    notification_email_from keepalived@localhost  #Where does the mail go from?
    smtp_server 127.0.0.1    #Mail server stmp address
    smtp_connect_timeout 30  #timeout
    router_id ka1  #host name
    #vrrp_skip_check_adv_addr   #Skip checking data messages, default checking
    #vrrp_strict    #Strict compliance with vrrp protocol, without vip, unicast address, ipv6 will not be able to start
    #vrrp_iptables  #Do not generate iptables rules
    vrrp_mcast_group4 224.100.100.100   #Multicast, default 224.0.0.18
    #vrrp_garp_interval 0   #Delay in sending arp messages
    #vrrp_gna_interval 0   #Message Delay
}


#Configure virtual routers:
vrrp_instance VI_1 {   #Name of configuration instance
    nopreempt  #Define working mode as non-preemptive mode
    preempt_delay 300  #Preemptive mode, the delay time of triggering a new election operation after the node is online, default mode
    state MASTER    #Master MASTER, from BACKUP
    interface eth0   #Binding to the physical interface used by the current virtual router
    virtual_router_id 66  #Current virtual router unique identifier, range 0-255
    priority 100   #Priority, Range 1-254
    advert_int 1   #vrrp notification interval, default 1s
    authentication {   #Authentication mechanism
        auth_type PASS   # {AH|PASS}, set the password type,
        auth_pass 123   #Password
    }
    virtual_ipaddress {
        192.168.99.99/24 dev eth0 label eth0:1  #Setting up Virtual Network Card
    }
    track_interface {  #Configuration of monitoring network interface, once a fault occurs, it will be converted to FAULT status to achieve address transfer
        eth0
    }
}

Define notification scripts:

notify_master <STRING>|<QUOTED-STRING>: 
#The script triggered when the current node becomes the primary node

notify_backup <STRING>|<QUOTED-STRING>: 
#A script triggered when the current node is turned into a standby node

notify_fault <STRING>|<QUOTED-STRING>: 
#A script triggered when the current node changes to a "failed" state

notify <STRING>|<QUOTED-STRING>: 
#Common format notification trigger mechanism, a script can complete the above three state transitions when notification

Calling method of script:

[ka1]$ vim /etc/keepalived/keepalived.conf
#Add the following line at the end of the vrrp_instance VI_1 statement block
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"

Sample notification script

#!/bin/bash
#
contact='root@localhost'
notify() {
    mailsubject="$(hostname) to be $1, vip floating"
    mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
    master)
        notify master
        ;;
    backup)
        notify backup
        ;;
    fault)
        notify fault
        ;;
    *)
        echo "Usage: $(basename $0) {master|backup|fault}"
        exit 1
        ;;
esac

Log configuration

[ka1]$ vim /etc/sysconfig/keepalived
    KEEPALIVED_OPTIONS="-D -S 6"

[ka1]$ vim /etc/rsyslog.conf
    local6.*       /data/keepalived.log

Mail Settings

vim ~/.mailrc or /etc/mail.rc

set from=12345678@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=12345678@qq.com
set smtp-auth-password=lzhdjmtznbftbiai
set smtp-auth=login
set ssl-verify=ignore

Test: echo test mail | mail - s test 12345678@qq.com

KeepAlived dual master configuration example:

! Configuration File for keepalived
global_defs {
    notification_email {
        root@localhost
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id node1
    vrrp_mcast_group4 224.0.100.100
}

#Provide a service
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 6  #Notice that it can't be the same.
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 571f97b2
    }
    virtual_ipaddress {
        192.168.0.100/24 dev eth0   #It's different from the VIP below.
    }
}

#Providing another service
vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 8   #Notice that it can't be the same.
    priority 98
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 578f07b2
    }
    virtual_ipaddress {
        192.168.0.200/24 dev eth0  #It's different from the VIP above.
    }
}

Tags: network vim Nginx Linux

Posted on Wed, 07 Aug 2019 02:16:01 -0700 by agent47