Redis performance diagnosis series of articles

Introduction: Based on years of experience in using and operating Redis, this article details the causes that may cause Redis performance problems and analyzes the corresponding solutions. I also hope that this series of articles can help you use Redis more reasonably and locate quickly. and solve the problem.

(This article was first published on the "Database Architect" account, subscribe to the "Database Architect" account, and learn database technology together)

As an in-memory database with the highest utilization rate in the industry, Redis has very high performance, and the QPS pressure test of a single node can reach more than 180,000. But because of this, when an application accesses Redis, if it finds that the response delay increases, it will have a great impact on the business.

For example, when using Redis on a daily basis, you must have encountered the following problems more or less:
Why did the Redis service, which used to be stable in the past, suddenly slow down from a certain point in time today?

Why is accessing the same command from Redis, sometimes it responds very quickly, and sometimes it is very slow?

Why does accessing Redis suddenly get stuck and automatically recover after a while, which also causes many glitches in business requests?
. . .

If you don't understand the architecture of Redis, the implementation principles of core functions, and even the usage restrictions of some commands, etc., then the troubleshooting of this slow access problem will be confusing, and you don't know where to start.

Based on years of experience in using and operating Redis, this article combs in detail the reasons that may cause Redis performance problems and analyzes the corresponding solutions. I also hope that this series of articles can help you use Redis more reasonably, and locate and solve them quickly. question.

1. Link Analysis of Redis Access Architecture

First of all, before deeply analyzing the Redis service, it is necessary to find out whether the access to Redis is really slow.

If we find that the response delay of our application service becomes longer, we first need to check the inside of the application to confirm whether the access path to Redis is slow, which slows down the response throughput of the entire service.
Here are two more critical self-checks:

For the request of the application service to access Redis, record the response delay of each request (such as using a distributed link tracking system, etc.) to see if the response time of accessing Redis is longer;
Troubleshoot multiple nodes of the application service to see if every node has a problem, or if only one has a problem.

These two are the key self-examinations before in-depth analysis of Redis service problems, which can be done with half the effort!

For the first point, there may be two reasons why the link from the application to Redis is slow:

There is a problem with the link between the application and the Redis service, such as the network load of the server where Redis is located is too high, packet loss, switch problems, slow proxy, etc.;

Redis itself is indeed slow for a few reasons.

Generally, there will be relevant monitoring at the server level, and network problems can be easily discovered, such as network card full, network card frequency reduction [10 Gigabit to Gigabit] and so on.

For the response time of the Redis access link, a simulation monitoring can be done. The following Redis access architecture, the application goes through the domain name system and the VIP system, and finally arrives at the server where Redis is located. In this case, it can simulate the request domain name and the VIP request respectively , Request three paths to connect directly to the Redis Server to evaluate whether the response time has indeed become longer.


The following is another Redis architecture, and the access paths are different, so the direction of investigation will not be different.


2. Redis Performance Benchmark Evaluation

If the verification finds that the response time of the service requesting Redis has become longer, then the focus of the problem analysis can be placed on Redis at this moment.

Next, we focus on analyzing the performance of Redis.

First, we need to perform a benchmark performance test on Redis to understand the benchmark performance of our Redis service on the current environment server.

What is benchmark performance?

Simply put, benchmark performance refers to what is the maximum response delay and average response delay for accessing Redis on a server with normal load?

Why test benchmark performance? Refer to the official response delay test to judge whether your Redis service is slow?

The answer is no.

Because Redis has different performance in different software and hardware environments, CPUs of different frequency models and different SSD hard drives will greatly affect the performance of Redis. When the server configuration is relatively low, the delay is 10ms, and the response of Redis is considered to be slow, but if the configuration is relatively high, it may be considered that Redis is slow when the delay is 1ms.

Therefore, only by knowing the benchmark performance of our Redis on the production environment server can we further evaluate, when the latency is reached, it is considered that Redis is indeed slow.

The tools that come with Redis can help us complete this kind of evaluation. The following two benchmark performance tests are introduced.

Method 1: redis-cli --intrinsic-latency
In order to avoid the network delay between the business test server and the Redis server, you need to test the response delay of the instance directly on the Redis server. Execute the following command to test the maximum response delay of this instance within 120 seconds:

shell> redis-cli -h 127.0.0.1 -p 6379 --intrinsic-latency 120
Max latency so far: 4 microseconds.
Max latency so far: 5 microseconds.
Max latency so far: 15 microseconds.
Max latency so far: 23 microseconds.
Max latency so far: 64 microseconds.
Max latency so far: 196 microseconds.
Max latency so far: 245 microseconds.
Max latency so far: 246 microseconds.
Max latency so far: 254 microseconds.
Max latency so far: 259 microseconds.
29298480 total runs (avg latency: 4.0958 microseconds / 40957.76 nanoseconds per run).

Worst run took 63x longer than the average latency.

As you can see from the output, the maximum response delay in these 120 seconds is 259 microseconds (0.259 milliseconds).

Method 2: redis-benchmark

Redis-benchmark is the official Redis performance test tool that comes with Redis, which can effectively test the performance of Redis services.

shell> redis-benchmark -h 127.0.0.1 -p 6379 -t set,get -c 500 -n 100000
====== SET ======
100000 requests completed in 1.02 seconds
500 parallel clients
3 bytes payload
keep alive: 1

0.00% <= 1 milliseconds
0.05% <= 2 milliseconds
99.09% <= 3 milliseconds
99.88% <= 4 milliseconds
100.00% <= 4 milliseconds
97847.36 requests per second

====== GET ======
100000 requests completed in 1.02 seconds
500 parallel clients
3 bytes payload
keep alive: 1

0.00% <= 1 milliseconds
0.05% <= 2 milliseconds
99.29% <= 3 milliseconds
99.92% <= 4 milliseconds
100.00% <= 4 milliseconds
97656.24 requests per second

This command evaluates the operation response time of the set and get commands, and executes 10w operations with 500 concurrent operations. It can be seen from the output results that the QPS of set reaches 97847, and the response time is within 4ms; the QPS of get reaches 97656, The maximum response time is also within 4ms;
After understanding the benchmark performance test method, we can follow the following steps to determine whether Redis is really slow:

Test the benchmark performance of a normal Redis instance on a server with the same configuration
Find a potentially slow Redis instance and test the benchmark performance of this instance
Comparing the running delay of this instance with the normal Redis benchmark performance, if the performance gap is more than twice, it can be considered that the response of the Redis service has indeed slowed down.

If it is confirmed that the Redis service is slow, how to troubleshoot the problem? The following series of articles will analyze in detail step by step from multiple dimensions, welcome to pay attention.