redis master slave + sentinel+VIP high availability structure

preface

sentinel is a highly available program of redis. It can discover and automatically switch the configuration of redis service in the master-slave state. sentinel can also manage multiple sets of redis masters and slaves

The application can be directly connected to the sentinel through a driver similar to jedis to achieve high availability. Jedis will find the real main library address in sentinel, and then let the program connect to the real main library address operation

However, there are three problems in this architecture. First, to realize this function, the configuration of the application needs to change from redis to connection sentry. Second, if the sentry hangs up, the application will report an error and cannot switch( Three point two Third, if one set of sentry management has more than one set of redis master-slave, it is not very good management

There are two solutions. One is to add nginx like load balancing in front of the sentry to control the access of jedis to the sentry address. The other is to add highly available vip operations to the redis master-slave to replace the jedis direct sentinel, because sentinel supports access to the foot book operations when handoff occurs

This article is about joining the highly available vip mode


sentinel communication principle

Before I talk about the subject, I want to talk about the principle of sentinel

Communication principle:

When a sentinel is not connected to sentinel at all, sentinel, the first sentinel, takes the initiative to communicate with the main database of redis, asking if there is any other connection information between the secondary database of redis and sentinel. If not, sentinel creates a configuration and waits for other sentinel information to be synchronized

Then, the second sentinel company enters the redis main database and asks if there are any other slave databases and sentinel addresses. At this time, the redis main database tells it that there are other sentinels. At this time, the second sentinel will take the initiative to synchronize the sentinel information with the first sentinel, thus forming a sentinel cluster of two nodes

Then, the third and fourth sentinels are so synchronized that they form a larger cluster

Switching principle:

When a sentinel finds that the main database is down, he will send a subjective suspend message sdown to the sentinels of other nodes. When most of the other sentinels (you can set the number) think that the main database is indeed down, then the sentinel cluster will get an objective suspend message odown state, Sentinel, the first sentinel who initiates the subjective suspend information, will start the formal redis master-slave switch. If not enough sentinel returns the objective suspend down information, the switch will not be triggered, which avoids the flash switch caused by some network reasons

In the process of switching, the slave database will be transferred to the master database, and the master database will be transferred to the slave database. If the fault master database fails to get up, the switch will be unsuccessful, but the slave database will continue to switch to the master database. When the fault master database is renewed, sentinel will take the initiative to connect and change its status to the slave database, and then the new slave database will automatically resynchronize the master-slave data and structure without human intervention

Finally, sentinel will communicate with each other, synchronize the latest redis master-slave information from the latest, and wait for the next switch. This interval can also be set, generally 3 seconds


Setting up the environment

How to build redis master and slave is not the focus of this paper, so we will not explain it, just list the relevant important parameters

Open redis profile redis.conf

#Local redis password
requirepass "123123"
#When the local machine is a slave library, the password of the host machine, usually to avoid trouble, will be consistent with the above, but it can be inconsistent with the above
masterauth "123123"
#Address and port of the host when the native is a slave
slaveof 172.25.111.187 6379

Then, we will talk about sentinel's configuration later. Open the configuration file sentinel.conf

#Binding accessible addresses, 0.0.0.0 It's all
bind 0.0.0.0
#In the network protection mode, no refers to the external network that can be accessed directly. yes refers to the need to configure bind ip or set access password. The sentry does not involve data, and usually does not need to configure password
protected-mode no
#Access port
port 26379
#Data directory, pid and log
dir "/data/redis/data"
pidfile "/data/redis/data/config/sentinel_26379.pid"
logfile "/data/redis/data/logs/sentinel_26379.log"
#Allow background operation
daemonize yes
#redis main database information to be monitored,
#jdzxyd_189: the name of the project can be changed as long as you can understand it
#172.25.111.187 : IP address of the main database. Remember that it is the IP address of the main database
#6379: redis port of the main database
#3: The number of sdown confirmations of subjective suspend information, that is to say, if three sentinel s think that the primary redis is hung, it is really hung and returns to the state of odown
sentinel monitor jdzxyd_189 172.25.111.187 6379 3
#Confirm the hang time, that is, if jdzxyd is within 5 seconds_ 189 if the main database of the project does not respond, SDOWN is considered, and the unit is milliseconds
sentinel down-after-milliseconds jdzxyd_189 5000
#Failover expiration time. When failover starts, jdzxyd_189 project still hasn't triggered any failover operation. Currently sentinel will consider this failover failure. The unit is Ms
sentinel failover-timeout jdzxyd_189 15000
#This jdzxyd_ Authentication password of 189 project
sentinel auth-pass jdzxyd_189 123123
#Reconfigure the client after failover, jdzxyd_189 when executing a script, a large number of parameters will be passed. Please refer to the relevant documents. This script is used to configure VIP
sentinel client-reconfig-script jdzxyd_189 /root/yw/change_vip_6379.sh

Some people may have doubts about how many sentinels are reasonable. Considering that the parameters of our monitor can be set, we can refer to the concept of zookeeper, that is, most principles. If the total number is 3 sentinels, then the monitor will set 2 sentinels, if the total number is 5, then the monitor will set 3 sentinels, and so on. However, it should be noted that too many sentinels will affect the speed of switching effectiveness, So we still need to take a comprehensive measure. Also, don't forget that a sentinel can manage multiple redis masters and slaves. You can consider the combination of public sentinel cluster + private sentinel cluster, which is flexible and changeable

Then, let's look at the script for switching vip

#!/usr/bin/env bash
#Among the parameters passed from sentinel client reconfig script, $6 is the IP of the new main library, so we need to get it
MASTER_IP=$6
#vip address of highly available HA
VIP='172.25.111.189'
#Subnet
NETMASK='24'
#Network card name
INTERFACE='ens160'
#Virtual network card number
key='1'
ifctrl=/usr/sbin/ifconfig
arpingd=/usr/sbin/arping
sshh=/usr/bin/ssh
timeoutd=/usr/bin/timeout
#LOCAL_IP="`${ifctrl} ${INTERFACE}|grep 172.25|awk '{print $2}'|awk -F':' '{print $2}'`"
LOCAL_IP="`${ifctrl} ${INTERFACE}|grep 172.25|awk '{print $2}'`"
if [ ${MASTER_IP} = ${LOCAL_IP} ];then
    ping -c 1 ${VIP} > /dev/null 2>&1
    if [ $? -eq 0 ];then
        ${timeoutd} 5 ${sshh} -o stricthostkeychecking=no ${VIP} "${ifctrl} ${INTERFACE}:${key} down" >/dev/null 2>&1
    fi
    ${ifctrl} ${INTERFACE}:${key} ${VIP}/${NETMASK}
    ${arpingd} -q -c 1 -A ${VIP} -I ${INTERFACE}
    exit
else
    ${ifctrl} ${INTERFACE}:${key} down
    exit
fi

I don't want to talk about the script function in detail, but I want to stress that when I write a script, I must pay attention to the environment, because linux doesn't recognize the character encoding of windows, which is easy to cause wonderful things

When everything is ready, hang vip in the main database, start redis and sentinel, and the cluster will be finished

ifconfig ens160:1 172.25.111.189/24
/usr/local/bin/redis-server /data/redis/data/config/redis_6379.conf >/var/log/redis_6379.log 2>&1
/usr/local/bin/redis-sentinel /data/redis/data/config/sentinel_26379.conf >/var/log/sentinel_26379.log 2>&1


Switch resolution

We can use the sentinel log / data/redis/data/logs/sentinel_26379.log When you see the process of switching, you can see that sdown,odown and script enabled logs are all in it


















Tags: Database Redis network Jedis

Posted on Tue, 12 May 2020 02:40:18 -0700 by teanza