edit-icon download-icon

Troubleshoot Jedis connection pool errors

Last Updated: Jul 20, 2017

The problem of connection pool access failure occurs frequently in Jedis connection pool mode. The error message is as follows:

  1. redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

Identify the cause as follows and perform troubleshooting accordingly:

Check the network

Run telnet host 6379 to check the network. After a connection is established, input the auth password and press Enter to check whether +OK\r\n is returned. If yes, send a ping request or read/write request and then check whether a response is returned properly. Repeat the preceding process multiple times to locate network problems.

Check the JedisPool connections setting

You must complete connection pool setting when using JedisPool. Connection pool access failure occurs when the number of connections indicated by MaxTotal is exceeded. In this case, you can run netstat -an | grep 6379 | grep EST | wc -l on the accessing client to view the number of connections and compare it with the value of MaxTotal. If the two values are close to each other or the number of excessive connections is small, the connection pool setting is proper.

Check the JedisPool code

JedisPool requires that resources be returned by calling returnResource or close after getResource is executed. Check whether the JedisPool code is used correctly. The sample code is as follows:

  1. JedisPoolConfig config = new JedisPoolConfig();
  2. //Maximum idle connections, which are evaluated by the application. Do not set it to a value greater than the maximum connections of an ApsaraDB for Redis instance.
  3. config.setMaxIdle(200);
  4. //Maximum connections, which are evaluated by the application. Do not set it to a value greater than the maximum connections of an ApsaraDB for Redis instance.
  5. config.setMaxTotal(300);
  6. config.setTestOnBorrow(false);
  7. config.setTestOnReturn(false);
  8. String host = "*.aliyuncs.com";
  9. String password = "password";
  10. JedisPool pool = new JedisPool(config, host, 6379, 3000, password);
  11. Jedis jedis = null;
  12. try {
  13. jedis = pool.getResource();
  14. /// ... do stuff here ... for example
  15. jedis.set("foo", "bar");
  16. String foobar = jedis.get("foo");
  17. jedis.zadd("sose", 0, "car");
  18. jedis.zadd("sose", 0, "bike");
  19. Set<String> sose = jedis.zrange("sose", 0, -1);
  20. } finally {
  21. if (jedis != null) {
  22. jedis.close();
  23. }
  24. }
  25. /// ... when closing your application:
  26. pool.destroy();

Check for nf_conntrack packet loss

Run dmesg to check whether the client is abnormal.

  1. nf_conntrack: table full, dropping packet

If nf_conntract packet loss occurs, modify sysctl -w net.netfilter.nf_conntrack_max=120000.

Check TIME_WAIT

Run ss -s to check whether excessive TIME_WAIT connections exist.

If yes, modify the following parameters:

  1. sysctl -w net.ipv4.tcp_max_tw_buckets=180000
  2. sysctl -w net.ipv4.tcp_tw_recycle=1

Check DNS name resolution

Bind the host address in the /etc/hosts file and then check whether the problem persists. If yes, DNS name resolution is normal.

  1. 192.168.1.1 *.redis.rds.aliyuncs.com

Help

If the problem persists, capture packets, record the error occurrence time and error message, and send the record together with the packet capture file to the Alibaba Cloud after-sales personnel. The packet capture command is sudo tcpdump -i eth0 tcp and port 6379 -n -nn -s 74 -w redis.cap.

Thank you! We've received your feedback.