Analysis of Redis Cache Penetration, Cache Avalanche and redis Concurrency

It is common to use redis as a cache, but there may be a series of problems after using redis, especially when the amount of data is large. Some classical problems are as follows:

(1) Data consistency between caches and databases

Distributed environments (not to mention stand-alone) are very prone to data consistency problems between caches and databases. To this end, it can only be said that if your project requires strong consistency in caching, then do not use caches. We can only adopt appropriate strategies to reduce the probability of data inconsistency between cache and database, but can not guarantee strong consistency between the two. Appropriate strategies include appropriate cache update strategy, timely update of the cache after updating the database, and adding retry mechanism when the cache fails, such as MQ mode message queue.

(2) Cache Breakdown

Cache breakdown means that malicious users simulate requests for data that do not exist in many caches. Because there is no data in the cache, these requests fall directly on the database in a short time, resulting in database anomalies. This is what we encountered in the actual project, some snap-up activities, secondkill activities of the interface API was brushed by a large number of malicious users, resulting in a short period of time database c timeout, good in the database is read and write separation, but also for interface current limitation, hold.

Solutions:

Scheme 1. Queuing with Mutex Locks

It is a common practice in the industry to compare prices, that is, according to the key to obtain the value of space-time, lock, load data from the database and release the lock. If other threads fail to acquire locks, wait for a period of time and try again. It is important to note that if distributed locks are to be used in distributed environments, it is enough for a single machine to use a common lock (synchronized, Lock).

public String getWithLock(String key, Jedis jedis, String lockKey, String uniqueId, long expireTime) {
    // Get value by key
    String value = redisService.get(key);
    if (StringUtil.isEmpty(value)) {
        // Distributed locks, refer in detail to https://blog.csdn.net/fanrenxiang/article/details/79803037
        //The encapsulated tryDistributed Lock includes setnx and expire functions, which are not supported in lower versions of redis
        try {
            boolean locked = redisService.tryDistributedLock(jedis, lockKey, uniqueId, expireTime);
            if (locked) {
                value = userService.getById(key);
                redisService.set(key, value);
                redisService.del(lockKey);
                return value;
            } else {
                // Other threads come in and wait for 50ms to retry without getting the lock
                Thread.sleep(50);
                getWithLock(key, jedis, lockKey, uniqueId, expireTime);
            }
        } catch (Exception e) {
            log.error("getWithLock exception=" + e);
            return value;
        } finally {
            redisService.releaseDistributedLock(jedis, lockKey, uniqueId);
        }
    }
    return value;
}

The idea of doing this is clear and can relieve the pressure of database to a certain extent, but the lock mechanism increases the complexity of logic and reduces the throughput, which is a bit of a cure-all.

Scheme 2. Current Limitation and Fusion of Interface and Degradation

Important interfaces must have a current limiting strategy to prevent users from brushing the interface maliciously. At the same time, they should be ready for degradation. When some services in the interface are unavailable, they should fuse and fail to return quickly.

Scheme 3. Bloom filter

bloomfilter is similar to a hash set, which is used to quickly determine whether an element exists in a set or not. Its typical application scenario is to quickly determine whether a key exists in a container or not, and return directly if it does not exist. The key of Bloom filter lies in hash algorithm and container size. Let's see the effect of simple implementation. Here I use guava to implement Bloom filter:

<dependencies>  
     <dependency>  
         <groupId>com.google.guava</groupId>  
         <artifactId>guava</artifactId>  
         <version>23.0</version>  
     </dependency>  
</dependencies>  
public class BloomFilterTest {
 
    private static final int capacity = 1000000;
    private static final int key = 999998;
 
    private static BloomFilter<Integer> bloomFilter = BloomFilter.create(Funnels.integerFunnel(), capacity);
 
    static {
        for (int i = 0; i < capacity; i++) {
            bloomFilter.put(i);
        }
    }
 
    public static void main(String[] args) {
        /*Return to the computer's most precise time, unit subtlety*/
        long start = System.nanoTime();
 
        if (bloomFilter.mightContain(key)) {
            System.out.println("Successfully filtered to" + key);
        }
        long end = System.nanoTime();
        System.out.println("Bloom filter consumption time:" + (end - start));
        int sum = 0;
        for (int i = capacity + 20000; i < capacity + 30000; i++) {
            if (bloomFilter.mightContain(i)) {
                sum = sum + 1;
            }
        }
        System.out.println("Misjudgement rate:" + sum);
    }
}
Successful filtering to 999998
 Bloom filter consumption time: 215518
 Misjudgement rate: 318

As you can see, it takes only about 0.2 milliseconds to match key in 100w data, which is fast enough. Then we simulated the key that does not exist in BloomFilter for 1w, and the matching error rate is 318/10000. That is to say, the error rate is about 3%. The default error-tolerant rate of tracking BloomFilter source code is 0.03:

public static <T> BloomFilter<T> create(Funnel<T> funnel, int expectedInsertions /* n */) {
  return create(funnel, expectedInsertions, 0.03); // FYI, for 3%, we always get 5 hash functions
}

We can call this method of BloomFilter to explicitly specify the error rate:

private static BloomFilter<Integer> bloomFilter = BloomFilter.create(Funnels.integerFunnel(), capacity,0.01);

Under our breakpoint tracking, the difference between the error rate of 0.02 and the default time of 0.03 is as follows:

Comparing the two error rates, it can be found that the array size is 8142363 at 0.02, 7298440 at 0.03, the error rate is reduced by 0.01, and the array size maintained by BloomFilter is reduced by 843923. It can be seen that the default error rate of BloomFilter is 0.03 after the designer weighs the performance of the system. Note that Bloom filters do not support deletion operations. To solve the cache penetration problem here is:

public String getByKey(String key) {
    // Get value by key
    String value = redisService.get(key);
    if (StringUtil.isEmpty(value)) {
        if (bloomFilter.mightContain(key)) {
            value = userService.getById(key);
            redisService.set(key, value);
            return value;
        } else {
            return null;
        }
    }
    return value;
}

(3) Cache avalanche problem

A large number of Keys expire (fail) at the same time, and then a large wave of requests are instantaneously dropped in the database, resulting in connection anomalies.

Solution:

Solution 1: Lock queuing as well as buffer penetration to achieve the same goal.

Solution 2: Establish backup cache, cache A and cache B, A set timeout time, B set no value timeout time, read the cache from A first, A did not read B, and update A cache and B cache;

Option 3: Add a random time length when setting the cache timeout time, for example, the timeout time of the cache key is a fixed 5 minutes plus a random 2 minutes, which can avoid the avalanche problem to some extent.

public String getByKey(String keyA,String keyB) {
    String value = redisService.get(keyA);
    if (StringUtil.isEmpty(value)) {
        value = redisService.get(keyB);
        String newValue = getFromDbById();
        redisService.set(keyA,newValue,31, TimeUnit.DAYS);
        redisService.set(keyB,newValue);
    }
    return value;
}

(4) Cache concurrency

Concurrency here refers to concurrency problems caused by multiple redis clients setting keys at the same time. In fact, redis itself is a single-threaded operation, multiple clients operate concurrently, according to the principle of first-come first-execute, first-come first-execute, the rest of the blocking. Of course, another solution is to put redis.set operation in the queue to make it serialized, one must be executed, the specific code will not be up, of course, locking is also possible, as for why not use redis transactions, leave it to the officers themselves to think and explore.
 

 

 

 

 

 

Tags: Database Redis Jedis Google

Posted on Wed, 14 Aug 2019 02:24:21 -0700 by Rodis