Redis sentinel mechanism

  • Through the sentinel mechanism of redis, the switch between the master and slave servers of redis can be realized, and the election operation between multiple slave servers can be completed. The disadvantage of Redis master-slave replication: there is no way to conduct dynamic election on the master, and Sentinel mechanism is needed to complete the dynamic election.

brief introduction

  • Sentinel process is used to monitor the working status of Master Master in Redis cluster

  • When the Master master server fails, you can switch between Master and Slave servers to ensure high availability (HA) of the system

  • It has been integrated in the version of Redis 2.6 +, and the sentinel mode of Redis has been stabilized after the version of 2.8.

The role of the sentinel process

  • Monitoring: sentinel constantly checks that your Master and Slave are working properly.
  • Notification: when there is a problem with a monitored Redis node, sentinel can send notifications to administrators or other applications through the API.
  • Automatic failover: when a Master fails to work normally, sentinel will start an automatic failover operation.
    • It will upgrade one Slave of the failed Master to a new Master, and change the other Slave of the failed Master to copy the new Master;
    • When the client tries to connect to the failed Master, the cluster will also return the address of the new Master to the client, so that the cluster can replace the failed Master with the current Master.
    • After the Master and Slave servers are switched, the contents of the Master's redis.conf, Slave's redis.conf and sentinel.conf configuration files will change accordingly. That is to say, the Master's redis.conf configuration file will have another row of slavof configuration, and the sentinel.conf monitoring target will be changed accordingly.

Analysis of fault determination principle

  1. Each Sentinel process sends a PING command to the Master master Master server, Slave server and other Sentinel processes in the cluster once a second.

  2. If an instance takes longer than the value specified by the down after milliseconds option from the last valid reply to the PING command, it will be marked as SDOWN by the Sentinel process.

  3. If a Master master Master server is marked as SDOWN, all Sentinel processes monitoring the Master master Master server should confirm once per second that the Master master Master server has indeed entered the SDOWN state.

  4. When a sufficient number of Sentinel processes (greater than or equal to the value specified in the configuration file) confirm that the Master master server has entered the SDOWN status within the specified time range, the Master master server will be marked as the ODOWN.

  5. In general, each Sentinel process sends INFO commands to all Master master Master servers and Slave slave servers in the cluster every 10 seconds.

  6. When the Master master server is marked as * * ODOWN * * by Sentinel process, the frequency of Sentinel process sending INFO command to all Slave Slave Slave servers of the offline Master master server will be changed from once every 10 seconds to once every second.

  7. If there are not enough Sentinel processes to allow the Master master to log off, the objective logoff status of the Master master will be removed. If the Master master server sends the PING command back to the Sentinel process to return a valid reply, the Master master Master server's subjective offline status will be removed.

Case demonstration

  • Modify * * sentinel.conf * * of slave machine:

    # The ip port of the redis master node monitored by sentinel 
    # The name of the master node that can be named by itself can only be composed of the letters A-z, the numbers 0-9, and the three characters "." - ".
    # When the sentinel sentinel thinks that the master master node is out of contact, it is objectively considered that the master node is out of contact
    # sentinel monitor <master-name> <master ip> <master port> <quorum>
    sentinel monitor mymaster 192.168.10.133 6379 1
    
  • Description of other configuration items

sentinel.conf

# Port of sentinel instance running is 26379 by default
port 26379
 
# sentinel's working directory
dir /tmp
 
# The ip port of the redis master node monitored by sentinel 
# The name of the master node that can be named by itself can only be composed of the letters A-z, the numbers 0-9, and the three characters "." - ".
# When the sentinel sentinel thinks that the master master node is out of contact, it is objectively considered that the master node is out of contact
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
sentinel monitor mymaster 127.0.0.1 6379 2
 
# When the required pass foobared authorization password is enabled in the Redis instance, all clients connecting to the Redis instance must provide the password
# Set the password of sentinel connection master-slave note that the same authentication password must be set for master-slave
# sentinel auth-pass <master-name> <password>
sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
 
 
# After the specified number of milliseconds, the master node does not respond to sentry sentinel. In this case, sentry subjectively believes that the master node is offline for 30 seconds by default
# sentinel down-after-milliseconds <master-name> <milliseconds>
sentinel down-after-milliseconds mymaster 30000
 
# This configuration item specifies the maximum number of slave s that can synchronize the new master in the event of failover,
//The smaller the number, the longer it takes to complete failover,
//But if the number is larger, it means that more slave s are unavailable due to replication.
//You can set this value to 1 to ensure that only one slave at a time is in a state where command requests cannot be processed.
# sentinel parallel-syncs <master-name> <numslaves>
sentinel parallel-syncs mymaster 1
 
 
 
# Failover timeout can be used in the following aspects: 
#1. The interval between two failures of the same sentinel to the same master.
#2. When a slave synchronizes data from a wrong master, the time is calculated. Until the slave is corrected to synchronize data with the correct master.
#3. The time required to cancel an ongoing failover.  
#4. When performing failover, configure the maximum time required for all slaves to point to the new master. However, even after this timeout, slaves will still be correctly configured to point to the master, but they will not follow the rules configured by parallel syncs
# Default three minutes
# sentinel failover-timeout <master-name> <milliseconds>
sentinel failover-timeout mymaster 180000
 
# SCRIPTS EXECUTION
 
#Configure the script that needs to be executed when an event occurs. You can notify the administrator through the script, for example, send an email to notify the relevant personnel when the system is not running normally.
#There are the following rules for the running results of scripts:
#If the script returns 1 after execution, the script will be executed again later. The number of repetitions is currently 10 by default
#If the script returns 2, or a higher return value than 2, the script will not execute repeatedly.
#If the script is terminated during execution due to receiving the system interrupt signal, it behaves the same as when the return value is 1.
#The maximum execution time of a script is 60s. If this time is exceeded, the script will be terminated by a SIGKILL signal and then executed again.
 
#Notification script: when sentinel has any warning level event (such as the subjective failure and objective failure of redis instance), it will call the script. At this time, the script should notify the system administrator about the abnormal operation of the system through email, SMS and other ways. When the script is called, two parameters are passed to the script, one is the type of the event and the other is the description of the event.
#If the script path is configured in the sentinel.conf configuration file, the script must exist in the path and be executable. Otherwise, sentinel cannot start successfully.
#Notification script
# sentinel notification-script <master-name> <script-path>
sentinel notification-script mymaster /var/redis/notify.sh
 
# Client reconfiguration master node parameter script
# When a master changes due to failover, the script will be called to inform the relevant client about the change of the master address.
# The following parameters will be passed to the script when the script is called:
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
# At present, < state > is always "failover",
# < role > is one of "leader" or "observer". 
# The parameters from IP, from port, to IP, to port are used to communicate with the old master and the new master (i.e. the old slave)
# This script should be universal and can be called multiple times, not targeted.
# sentinel client-reconfig-script <master-name> <script-path>
 sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
 
  • Start sentinel service through redis Sentinel
./redis-sentinel sentinel.conf

Tags: Programming Redis

Posted on Thu, 12 Mar 2020 23:17:19 -0700 by rgriffin3838