Redis Cluster Data Migration Single Instance Scheme

Preface

There are many methods for synchronous migration of Redis cluster, such as redis-port,redis-migrate-tool and other tools. This paper mainly talks about the migration of CentOS from cluster to single instance through redis-migrate-tool.

redis-migrate-tool is an open source redis data migration tool of Weipinghui. It is based on redis replication, fast and stable. The github address is: https://github.com/vipshop/redis-migrate-tool .

The source of migration tools can be: individual redis instances, twemproxy clusters, redis cluster s, rdb files, aof files.
The target of the migration tool can be: a separate redis instance, a twemproxy cluster, a redis cluster, an rdb file.

The Redis version currently tested is 3.0.0, and if it is 4.0.x, it may be wrong.

[2019-07-31 16:57:50.393] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 4860 bytes from master[172.16.255.34:7001@17001]
[2019-07-31 16:57:50.393] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[172.16.255.34:7001@17001] is received, used: 0 s
[2019-07-31 16:57:50.393] rmt_redis.c:1643 rdb file node172.16.255.34:7001@17001-1564563470312707-6709.rdb write complete
[2019-07-31 16:57:50.393] rmt_redis.c:6446 ERROR: Can't handle RDB format version -1549717496
[2019-07-31 16:57:50.393] rmt_redis.c:6715 ERROR: Rdb file for node[172.16.255.34:7001@17001] parsed failed

As Can't handle RDB format version -1549717496 is a version compatibility problem, all versions of. rdb are incompatible, and rmt tools do not seem to support 4.0.X.

Consider downloading. If appendonly yes is turned on, it can be migrated through. aof files.

Current Cluster Data Existence

172.16.255.34:7000 172.16.255.34:7001
172.16.255.35:7002 172.16.255.35:7003
172.16.255.36:7004 172.16.255.36:7005

Cluster data needs to be migrated to 172.16.255.34:6379 for a single instance

install

Dependence:

$ yum -y install automake libtool autoconf bzip2 git

Install redis-migrate-tool:

$ git clone https://github.com/vipshop/redis-migrate-tool
$ cd redis-migrate-tool
$ autoreconf -fvi
$ ./configure
$ make

View command help to check whether the installation was successful

$ src/redis-migrate-tool -h

If it appears

$ src/redis-migrate-tool -h
This is redis-migrate-tool-0.1.0

Usage: redis-migrate-tool [-?hVdIn] [-v verbosity level] [-o output file]
                  [-c conf file] [-C command]
                  [-f source address] [-t target address]
                  [-p pid file] [-m mbuf size] [-r target role]
                  [-T thread number] [-b buffer size]
....

The installation was successful.

More information Check out redis-migrate-tool usage details

Configuration file changes

The default configuration file for redis-migrate-tool is rmt.conf, which is changed as follows

$ vi rmt.conf

[source]
type: redis cluster
servers :
-172.16.255.34:7000

[target]
type: single
servers:
-172.16.255.34:6379

[common]
listen: 0.0.0.0:8888

Esc: WQ save exit.

Sorce is a cluster, and you only need to configure a server.

Start migrating

$ src/redis-migrate-tool -o log -d

Start running the migration in the background and view the run log tail-200 log.

....
[2019-08-01 11:59:19.919] rmt_redis.c:6601 Rdb file for node[172.16.255.34:7000] parsed finished, use: 0 s.
[2019-08-01 11:59:19.919] rmt_redis.c:6709 All nodes' rdb file parsed finished for this write thread(0).

You can see that all nodes'RDB file parsed finished indicates that the run is complete.

Running Data Check

$ src/redis-migrate-tool -o log -d -C redis_check

Default Sampling 1000 Data Checks

Check job is running...

Checked keys: 1000
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 1000

All keys checked OK!
Check job finished, used 1.041s

Successful migration~

If midway migration fails, use redis-cli-h 172.16.255.34 to clear keys

172.16.255.34:6379> flushall

keys * checks to see if it's empty, so that data migration like lists does not increase. redis_check checks for inconsistencies in part of the data, and you'll find that lists are all displayed.

[2019-07-31 17:00:27.185] rmt_check.c:848 ERROR: key checked failed: check key's value error, value is inconsistent. key(len:15, type:list): 509A4C6A4WAY_46

Java uses JedisCluster to connect clusters to JedisPool

@Override
public void afterPropertiesSet() throws Exception {
    if (mJedis == null) {
        mJedisPool = new JedisPool(genericObjectPoolConfig, hostName, Integer.valueOf(port), timeout);
        mJedis = mJedisPool.getResource();
    }
}

@Autowired
private Jedis jedis;

Links to the original text

Recommended reading

Tags: Redis github git Jedis

Posted on Thu, 01 Aug 2019 23:13:15 -0700 by raghavan20