Use scan instead of keys in RedisTemplate

Keys * This command should never be misused in production environments.Especially in the case of large data.Keys can cause Redis locks and increase Redis CPU usage.Many companies prohibit this order

You can use the scan command when you need to scan keys to match your own keys

Helper implementation of scan operation

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.function.Consumer;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.core.Cursor;
import org.springframework.data.redis.core.ScanOptions;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;

@Component
public class RedisHelper {
    
    @Autowired
    private StringRedisTemplate stringRedisTemplate;
    
    /**
     * scan Realization
     * @param pattern    Expression
     * @param consumer    Operations on iterated key s
     */
    public void scan(String pattern, Consumer<byte[]> consumer) {
        this.stringRedisTemplate.execute((RedisConnection connection) -> {
            try (Cursor<byte[]> cursor = connection.scan(ScanOptions.scanOptions().count(Long.MAX_VALUE).match(pattern).build())) {
                cursor.forEachRemaining(consumer);
                return null;
            } catch (IOException e) {
                e.printStackTrace();
                throw new RuntimeException(e);
            }
        });
    }

    /**
     * Get qualified key s
     * @param pattern    Expression
     * @return
     */
    public List<String> keys(String pattern) {
        List<String> keys = new ArrayList<>();
        this.scan(pattern, item -> {
            //Eligible key
            String key = new String(item,StandardCharsets.UTF_8);
            keys.add(key);
        });
        return keys;
    }
}

However, there is a problem: cursor s cannot be moved, scans can only be scanned once, and redis link errors can easily occur

Learn about scan, hscan, sscan, zscan first

http://doc.redisfans.com/key/scan.html

Why are keys unsafe?

  • The keys operation causes the database to be temporarily locked and other requests to be blocked; problems occur when there is a lot of business

Spring RedisTemplate implements scan

1. hscan sscan zscan

  • The "field" in the example is the key of the value redis, which is found from the hash in which the key is "field"
  • The opsForHash, opsForSet and opsForZSet of redisTemplate can correspond to sscan, hscan, zscan respectively
  • Of course, this online example is also wrong, because instead of traversing with cursor, only scan checked it once
  • You can lazy with.count(Integer.MAX_VALUE) and check it all back in a flash; but what's the difference between this and keys?Funny Face & Questioning Face
  • You can use the hscan, sscan, zscan methods of (JedisCommands) connection.getNativeConnection() to implement cursor traversal, referring to chapter 2.2 below
try {
    Cursor<Map.Entry<Object,Object>> cursor = redisTemplate.opsForHash().scan("field",
    ScanOptions.scanOptions().match("*").count(1000).build());
    while (cursor.hasNext()) {
        Object key = cursor.next().getKey();
        Object valueSet = cursor.next().getValue();
    }
    //Close cursor
    cursor.close();
} catch (IOException e) {
    e.printStackTrace();
}
  • cursor.close(); the cursor must be closed or the connection will continue to grow; you can use the client lists`info clients`info stats command to view the client connection status, and you will find that the scan operation persists
  • The redisTemplate.execute we normally use will actively release the connection, you can check the source confirmation
client list
......
id=1531156 addr=xxx:55845 fd=8 name= age=80 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=scan
......
org.springframework.data.redis.core.RedisTemplate#execute(org.springframework.data.redis.core.RedisCallback<T>, boolean, boolean)

finally {
    RedisConnectionUtils.releaseConnection(conn, factory);
}

2. scan

Most of the examples given online at 2.1 are this

  • This connection.scan cannot move cursor, it can only scan once
public Set<String> scan(String matchKey) {
    Set<String> keys = redisTemplate.execute((RedisCallback<Set<String>>) connection -> {
        Set<String> keysTmp = new HashSet<>();
        Cursor<byte[]> cursor = connection.scan(new ScanOptions.ScanOptionsBuilder().match("*" + matchKey + "*").count(1000).build());
        while (cursor.hasNext()) {
            keysTmp.add(new String(cursor.next()));
        }
        return keysTmp;
    });

    return keys;
}

2.2 Using MultiKeyCommands

  • Get connection.getNativeConnection; connection.getNativeConnection() The actual object is Jedis (debug can see), which implements many interfaces
public class Jedis extends BinaryJedis implements JedisCommands, MultiKeyCommands, AdvancedJedisCommands, ScriptingCommands, BasicCommands, ClusterCommands, SentinelCommands 
  • Move the cursor to get when scan.getStringCursor() exists and is not 0
public Set<String> scan(String key) {
    return redisTemplate.execute((RedisCallback<Set<String>>) connection -> {
        Set<String> keys = Sets.newHashSet();

        JedisCommands commands = (JedisCommands) connection.getNativeConnection();
        MultiKeyCommands multiKeyCommands = (MultiKeyCommands) commands;

        ScanParams scanParams = new ScanParams();
        scanParams.match("*" + key + "*");
        scanParams.count(1000);
        ScanResult<String> scan = multiKeyCommands.scan("0", scanParams);
        while (null != scan.getStringCursor()) {
            keys.addAll(scan.getResult());
            if (!StringUtils.equals("0", scan.getStringCursor())) {
                scan = multiKeyCommands.scan(scan.getStringCursor(), scanParams);
                continue;
            } else {
                break;
            }
        }

        return keys;
    });
}

divergent thinking

cursor has no close. Who is blocking it, Redis?

  • During the test, I basically just started more than ten scan s without closing cursor, and the next requests got stuck

redis side analysis

  • client lists`info clients`info stats view
    Only a dozen connections were found, and no blocked or rejected connections were found
  • The maximum number of connections allowed for config get maxclients query redis is 10000
1) "maxclients"
2) "10000"`
  • redis-cli can also log in directly on other machines

To sum up, redis itself is not stuck

Applying side analysis

  • netstat looks at the connection to redis, 6333 is the redis port; the connection persists
➜  ~ netstat -an | grep 6333
netstat -an | grep 6333
tcp4       0      0  xx.xx.xx.aa.52981      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52979      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52976      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52971      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52969      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52967      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52964      xx.xx.xx.bb.6333     ESTABLISHED
tcp4       0      0  xx.xx.xx.aa.52961      xx.xx.xx.bb.6333     ESTABLISHED
  • jstack View stack information for the application
    Many WAITING threads were found, all getting redis connections
    So it's almost certain that the redis thread pool for the application is full
"http-nio-7007-exec-2" #139 daemon prio=5 os_prio=31 tid=0x00007fda36c1c000 nid=0xdd03 waiting on condition [0x00007000171ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006c26ef560> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:590)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:441)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:362)
        at redis.clients.util.Pool.getResource(Pool.java:49)
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
        at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:276)
        at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:469)
        at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:132)
        at org.springframework.data.redis.core.RedisTemplate.executeWithStickyConnection(RedisTemplate.java:371)
        at org.springframework.data.redis.core.DefaultHashOperations.scan(DefaultHashOperations.java:244)

To sum up, it's application side stuck

Follow-up

  • After noon, redis client lists showed that the scan connection was still alive and not released; the application thread was still stuck
  • Check config get timeout, redis does not set timeout, config set timeout xxx can be set, unit seconds; but when redis timeout is set, redis releases the connection and the application is still stuck
1) "timeout"
2) "0"
  • netstat looks at the connection to redis, 6333 is the redis port; the connection changed from ESTABLISHED to CLOSE_WAIT;
  • The jstack is stuck in the JedisConnectionFactory.getConnection as it was
➜  ~ netstat -an | grep 6333
netstat -an | grep 6333
tcp4       0      0  xx.xx.xx.aa.52981      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52979      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52976      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52971      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52969      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52967      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52964      xx.xx.xx.bb.6333     CLOSE_WAIT
tcp4       0      0  xx.xx.xx.aa.52961      xx.xx.xx.bb.6333     CLOSE_WAIT
  • Retrospective TCP Four Waves
    ESTABLISHED indicates that the connection has been established
    CLOSE_WAIT indicates that the remote calculator is closing the connection and waiting for the socket connection to close
    Conform to phenomenon
  • redis connection pool configuration
    Based on the netstat-an above, you can basically determine that the redis connection pool size is 8; combined with the code configuration, if not specified, it does default to 8
redis.clients.jedis.JedisPoolConfig
private int maxTotal = 8;
private int maxIdle = 8;
private int minIdle = 0;
  • How do I configure a larger connection pool?
    A. Original Configuration
@Bean
public RedisConnectionFactory redisConnectionFactory() {
    RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration();
    redisStandaloneConfiguration.setHostName(redisHost);
    redisStandaloneConfiguration.setPort(redisPort);
    redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd));
    JedisConnectionFactory cf = new JedisConnectionFactory(redisStandaloneConfiguration);
    cf.afterPropertiesSet();
    return cf;
}

readTimeout, not specified by connectTimeout, has a default value of 2000 ms

org.springframework.data.redis.connection.jedis.JedisConnectionFactory.MutableJedisClientConfiguration
private Duration readTimeout = Duration.ofMillis(Protocol.DEFAULT_TIMEOUT);
private Duration connectTimeout = Duration.ofMillis(Protocol.DEFAULT_TIMEOUT); 

B. Modified Configuration

    1. Configuration 1: Some interfaces are already Deprecated
@Bean
public RedisConnectionFactory redisConnectionFactory() {
    JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
    jedisPoolConfig.setMaxTotal(16); // --up to 16 connections can be established
    jedisPoolConfig.setMaxWaitMillis(10000); // --10s could not get a connection to the connection pool,
                                             // --Direct error can not get a resource from the pool

    jedisPoolConfig.setMaxIdle(16);
    jedisPoolConfig.setMinIdle(0);

    JedisConnectionFactory cf = new JedisConnectionFactory(jedisPoolConfig);
    cf.setHostName(redisHost); // -- @Deprecated 
    cf.setPort(redisPort); // -- @Deprecated 
    cf.setPassword(redisPasswd); // -- @Deprecated 
    cf.setTimeout(30000); // -- @Deprecated does not appear to be working, timed out for 30s and did not close the connection to the connection pool;
                          // --redis does not set a timeout, ESTABLISHED is always set; redis sets a timeout and CLOSE_WAIT is always set after it

    cf.afterPropertiesSet();
    return cf;
}
    1. Configuration 2: This is a new configuration for friends in the group, the effect is the same
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration();
redisStandaloneConfiguration.setHostName(redisHost);
redisStandaloneConfiguration.setPort(redisPort);
redisStandaloneConfiguration.setPassword(RedisPassword.of(redisPasswd));

JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(16);
jedisPoolConfig.setMaxWaitMillis(10000);
jedisPoolConfig.setMaxIdle(16);
jedisPoolConfig.setMinIdle(0);

cf = new JedisConnectionFactory(redisStandaloneConfiguration, JedisClientConfiguration.builder()
        .readTimeout(Duration.ofSeconds(30))
        .connectTimeout(Duration.ofSeconds(30))
        .usePooling().poolConfig(jedisPoolConfig).build());

Reference resources

Notes on using redistemplate-cursor scan

How to access Redis data structures using RedisTemplate

Use of Keys and Scan in Redis

Deep understanding of Redis's scan command

Detailed configuration of spring-boot-starter-redis

Reason investigation for large number of CLOSE_WAIT on-line

How redis configures the standAlone version of jedisPool

An irregular use of jedis caused a significant increase in the close_wait bug for the redis client

Tags: Java Redis Jedis Apache

Posted on Thu, 29 Aug 2019 19:13:24 -0700 by sONOCOOLO