Redis Clustering on a Single Linux Server

1. Introduction to Redis

Redis is currently a large-scale cache middleware, which is widely used because of its powerful, efficient and convenient functions.

Redis released 3.0.0 in 2015, officially supporting redis clusters.The redis cluster was designed with decentralization and Middleware in mind.That is, each node in the cluster has an equal relationship, and each node holds its own data and the state of the entire cluster.Each node connects to all other nodes, and these connections remain active, which ensures that we can get data from other nodes simply by connecting to any one of the nodes in the cluster.

2. Introduction to Redis Cluster

A Redis cluster is a facility for data sharing between multiple Redis nodes.Redis clusters do not support redis commands that require multiple keys to be processed at the same time, because executing these commands requires moving data between multiple nodes, and under high load, these commands will degrade the performance of the redis cluster and result in unpredictable behavior.

Redis clusters provide a degree of availability through partitioning, allowing the cluster to continue processing command requests even if some of the nodes in the cluster fail or cannot communicate.

3. Redis Cluster Principles

How does Redis make sense of these nodes and data?

Redis does not use traditional consistent hashes to allocate data, but uses another method called hash slots.Redis cluster allocates 16384 slots by default. When we set a key, we use the CRC16 algorithm (cyclic redundancy check code) to get the slot we belong to, and then assign the key to the node hill of the hash slot interval, which is CRC16 (key)%16384.

Note: There must be more than three primary nodes to create a cluster, otherwise the cluster will fail.

The Redis cluster has other master nodes to vote and elect when one of its primary nodes is down, and more than half of the voting results determine that the primary node is down. If more than half of the primary node is down, the state of the entire cluster is fail.This is why at least three primary nodes are required to set up a Redis cluster environment.

Benefits of Redis Clustering:

  • 1) Ability to automatically split data into multiple nodes;
  • 2) The ability to continue processing command requests when a portion of the nodes in the cluster fails or cannot communicate;
  • 3) All rredis nodes are interconnected, and binary protocols are used internally to optimize transmission speed and bandwidth; binary protocols: message headers and message entities are given during transmission.If the message header length is fixed.The message entity can be obtained by parsing the message header;
  • 4) Fail of nodes takes effect only when more than half of the nodes in the cluster fail detection.
  • 5) The client connects directly to the redis node, does not need the intermediate proxy layer, does not need to connect all nodes in the cluster, and can connect any available node in the cluster.

The hash slot distribution Redis uses is good or bad, and the benefit is clear, for example: I want to add a node D, which redis cluster does by taking a portion of slot from the front of each node to D.Roughly this will happen:
Node A covers 1365-5460 Nodes B covers 6827-10922 Nodes C covers 12288-16383 Nodes D covers 01364,5461-6826,10923-12287 Similar to deleting a node, it can be deleted once the move is complete.

So the redis cluster is such a shape, as shown in the following figure:

4. Redis-cluster master-slave mode

To ensure high availability of data, Redis-cluster adds a master-slave mode where one master node corresponds to one or more slave nodes.The primary node provides data access, while the secondary node retrieves data backups from the primary node; when the primary node is down, one of the secondary nodes is selected to act as the primary node, thereby ensuring that the cluster does not hang.

5. Case Implementation

Reference for redis clustering on multiple Linux implementations Redis database cluster

(1) Case environment

Since you use a single server to implement a clustered environment, you can distinguish between them based on their ports!

(2) Case implementation

download Redis Cluster

1) Create six nodes

[root@docker ~]# tar zxf redis-4.0.14.tar.gz -C /usr/local
[root@docker ~]# cd /usr/local/redis-4.0.14/
[root@docker redis-4.0.14]# Make & & make install //compile install redis
[root@redis ~]# mkdir -p /usr/local/cluster/700{0..5}
//Since multiple nodes are created on a single machine, this directory is created beforehand to hold the configuration information for each node.
[root@docker ~]# cp /usr/local/redis-4.0.14/redis.conf /usr/local/cluster/7000
//Copy the original redis profile
[root@docker ~]# Vim/usr/local/cluster/7000/redis.conf//compile the configuration file for 7000 nodes
  92 port 7000                             //Modify Listening Port
 158 pidfile /var/run/              //Modify pid file name
 672 appendonly yes                 //Turn on aof persistence
 676 appendfilename "appendonly-7000.aof"           //Modify persisted profile name
 814 cluster-enabled yes                   //Turn on cluster mode for redis
 822 cluster-config-file nodes-7000.conf     //Modify Cluster Profile Name
 828 cluster-node-timeout 5000             //Modify the waiting time of the cluster
[root@docker ~]# cd /usr/local/cluster/7000
[root@docker 7000]# redis-server redis.conf //You must enter the appropriate directory to start the redis service

However, this error occurs at startup, as shown in Fig.

[root@redis 7000]# Vim/usr/local/cluster/7000/redis.conf//Write a configuration file for 7000 nodes
 136 daemonize yes                      //Open the daemon to run in the background
[root@redis 7000]# echo 512 > /proc/sys/net/core/somaxconn 
[root@redis 7000]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
[root@redis 7000]# sysctl -p
vm.overcommit_memory = 1
[root@redis 7000]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
//Simple optimization of the redis service based on the warning message just prompted

2) Start each node

[root@redis 7000]# redis-server redis.conf //Enter its directory to run the redis service
[root@redis 7000]# cp redis.conf ../7001
[root@redis 7000]# cp redis.conf ../7002
[root@redis 7000]# cp redis.conf ../7003
[root@redis 7000]# cp redis.conf ../7004
[root@redis 7000]# cp redis.conf ../7005
//Since six nodes need to be opened, their profiles need to be divided into six
[root@redis 7000]# sed -i s/7000/7001/g ../7001/redis.conf 
[root@redis 7000]# sed -i s/7000/7002/g ../7002/redis.conf 
[root@redis 7000]# sed -i s/7000/7003/g ../7003/redis.conf 
[root@redis 7000]# sed -i s/7000/7004/g ../7004/redis.conf 
[root@redis 7000]# sed -i s/7000/7005/g ../7005/redis.conf 
//Since all six nodes are port-dependent, you only need to modify the content in the configuration file named after the port number.
[root@redis 7000]# cd ../7001 && redis-server redis.conf
[root@redis 7001]# cd ../7002 && redis-server redis.conf
[root@redis 7002]# cd ../7003 && redis-server redis.conf
[root@redis 7003]# cd ../7004 && redis-server redis.conf
[root@redis 7004]# cd ../7005 && redis-server redis.conf
//All start services by entering their respective directories
[root@redis 7005]# netstat -anpt | grep redis     
tcp        0      0*               LISTEN      5615/redis-server 1 
tcp        0      0*               LISTEN      5620/redis-server 1 
tcp        0      0*               LISTEN      5524/redis-server 1 
tcp        0      0*               LISTEN      5592/redis-server 1 
tcp        0      0*               LISTEN      5605/redis-server 1 
tcp        0      0*               LISTEN      5610/redis-server 1 
tcp        0      0*               LISTEN      5615/redis-server 1 
tcp        0      0*               LISTEN      5620/redis-server 1 
tcp        0      0*               LISTEN      5524/redis-server 1 
tcp        0      0*               LISTEN      5592/redis-server 1 
tcp        0      0*               LISTEN      5605/redis-server 1 
tcp        0      0*               LISTEN      5610/redis-server 1 
//View ports are already listening, and the remaining 17,000 such ports are used for communication between clusters

3) Allocate hash slots for each node and join the cluster environment

[root@redis 7005]# Redis-cli-p 7000 //Specify any port to enter the redis terminal> CLUSTER INFO                  //View Cluster Details

cluster_state:fail                          //Discovery is now fail
........................                       //Omit some content> CLUSTER NODES           //View Cluster Nodes
d554512885b2679d432d0d6b011c9ea56ea1ebeb :7000@17000 myself,master - 0 0 0 connected
//Found that only 7000 nodes are participating in the cluster> exit
[root@redis 7005]# redis-cli -h -p 7000 cluster addslots {0..5461}
[root@redis 7005]# redis-cli -h -p 7001 cluster addslots {5462..10922}
[root@redis 7005]# redis-cli -h -p 7002 cluster addslots {10923..16383}
//Assign hash slots for each cluster stage> CLUSTER MEET 7000
//Used to connect different Redis nodes supported by the open cluster to join the working cluster
[root@redis 7005]# redis-cli -p 7000> CLUSTER INFO
cluster_state:ok                     //Looking at the cluster state again, we found that it was already "ok"
........................                   //Omit some content> CLUSTER NODES                   //View cluster node information again


The preset environment is 7000 as the primary node of 7003, 7001 as the primary node of 7004, and 7002 as the primary node of 7005, as follows:

[root@redis 7005]# Redis-cli-p 7003 //node connected to 7003> CLUSTER REPLICATE d554512885b2679d432d0d6b011c9ea56ea1ebeb
OK           //This is the ID number of 7000> exit
[root@redis 7005]# Redis-cli-p 7004 //node connected 7004> CLUSTER REPLICATE 182a55ee8b38afbf5b48209979f220dc4da7c14a
OK                    //This is ID number 7001> exit
[root@redis 7005]# Redis-cli-p 7005 //node connected to 7005> CLUSTER REPLICATE 82b62bda5b31d8ee27aee8d7320663ee86276e9f
OK                   //This is ID number 7002> CLUSTER NODES           //View the details of the cluster node again


4) Access data testing

[root@redis 7005]# Redis-cli-p 7000 //Enter Node Insert Data Test> set name lzj
(error) MOVED 5798           
//Error found, prompt should be 7001 node to store this data> exit
[root@redis 7005]# Redis-cli-p 7000-c //Use the'-c'option to indicate entry into the cluster> set name lzj                  //Reinsert data
-> Redirected to slot [5798] located at
OK               //Prompt to store data on 7001 node> get name               //Get the value for the key

5) Simulate single node failure

[root@redis 7001]# cd ../7001
[root@redis 7001]# redis-cli -p 7001 shutdown
//Simulate 7001 Node Failure
[root@redis 7001]# redis-cli -p 7000> CLUSTER NODES            //Enter the node to view the state of the cluster node> get name


[root@redis 7001]# redis-server redis.conf 
//Enter the directory corresponding to 7001 and start the redis node
[root@redis 7001]# redis-cli -p 7000> CLUSTER NODES               //Enter Cluster to View Cluster Status


6) Install ruby software

download Packages required by ruby

[root@redis ~]# yum -y install rpm-build openssl openssl-devel
//Dependencies required to install ruby
[root@redis ~]# tar zxf ruby-2.3.1.tar.gz -C /usr/src
cd [root@redis ~]# cd /usr/src/ruby-2.3.1/
[root@redis ruby-2.3.1]# ./configure --prefix=/usr/local/ruby && make && make install
//Compile and install ruby, possibly for a long time
[root@redis ~]# ln -s /usr/local/ruby/bin/* /usr/local/bin
//Create symbolic links for gem commands
[root@redis ~]# ln -s /usr/local/redis-4.0.14/src/* /usr/local/bin
//Create symbolic links for common redis commands
[root@redis ~]# gem install redis-3.3.0.gem 
//Install Redis Cluster Prerequisites using the gem command

7) Add Primary Node 7006

[root@redis ~]# redis-trib.rb check
//Check Cluster Node Status


[root@redis ~]# mkdir /usr/local/cluster/7006
[root@redis ~]# cd /usr/local/cluster/7006
//Create the appropriate directory and enter
[root@redis 7006]# cp ../7000/redis.conf .
[root@redis 7006]# sed -i s/7000/7006/g redis.conf 
//Copy and modify the configuration file
[root@redis 7006]# redis-server redis.conf 
//Modify redis service for 7006 nodes
[root@redis ~]# redis-trib.rb add-node
//Add 7006 nodes to 7000 node cluster environment (master state by default)
[root@redis ~]# redis-trib.rb check
//View Cluster Status Again


When screenshots were taken, we accidentally blocked it. We can see that 7006 node has no slots and can not store data!

[root@redis ~]# redis-trib.rb reshard //allocate 7000 slots for this cluster
How many slots do you want to move (from 1 to 16384)? 4096    //How many nodes to operate on
What is the receiving node ID?7add42108b3fc3e9c0af2846fb06ceaf8b172c87  
//To 7006 nodes, enter the ID number of 7006 nodes here
Source node #1:all //Assign from all nodes
[root@redis ~]# redis-trib.rb check
//View Cluster Status Again


8) Add from Node 7007

[root@redis ~]# mkdir /usr/local/cluster/7007
[root@redis ~]# cd /usr/local/cluster/7007
//Create the appropriate directory and enter
[root@redis 7007]# cp ../7000/redis.conf .
[root@redis 7007]# sed -i s/7000/7007/g redis.conf 
//Copy and modify the configuration file
[root@redis 7007]# redis-server redis.conf
//Start 7007 Node redis Service
[root@redis ~]# redis-trib.rb add-node --slave
//Add 7007 nodes to the 7000 cluster and specify it as slave
[root@redis ~]# redis-trib.rb check
//View Cluster Status


9) Add from Node 7008

[root@redis ~]# mkdir /usr/local/cluster/7008
[root@redis ~]# cd /usr/local/cluster/7008
[root@redis 7008]# cp ../7000/redis.conf .
[root@redis 7008]# sed -i s/7000/7008/g redis.conf 
[root@redis 7008]# redis-server redis.conf 
//Create the appropriate directory, modify the configuration file, and start the service
[root@redis ~]# redis-trib.rb add-node --slave --master-id 7add42108b3fc3e9c0af2846fb06ceaf8b172c87
//Add a node to the cluster, specify it as salve, and specify the primary node ID number (the ID of 7006 nodes)
[root@redis ~]# redis-trib.rb check
//View Cluster Status


10) Delete 7000 nodes

[root@redis ~]# redis-trib.rb reshard //Operate on cluster nodes
How many slots do you want to move (from 1 to 16384)? 4096                     //Operations for multiple nodes
What is the receiving node ID? 
7add42108b3fc3e9c0af2846fb06ceaf8b172c87             //Enter here the slot node that will be deleted to that node (7006 node)
Source node d554512885b2679d432d0d6b011c9ea56ea1ebeb    //Enter Deleted Nodes
Source node  done
Do you want to proceed with the proposed reshard plan (yes/no)? yes
[root@redis ~]# redis-trib.rb del-node d554512885b2679d432d0d6b011c9ea56ea1ebeb
//Delete 7000 nodes in 7000 cluster
[root@redis ~]# redis-trib.rb check //View cluster status


To the end of this article, thank you for reading ----------------

Tags: Linux Redis Ruby Docker vim

Posted on Sat, 21 Dec 2019 13:14:06 -0800 by Rabioza123