Connection timeouts caused by slow requests are the common issues that affect the service performance. The slow log feature of ApsaraDB for Redis allows you to find the IP address of the client that sends these requests and troubleshoot issues based on the details of slow logs.

Prerequisites

  • The database version of the ApsaraDB for Redis instance is Redis 4.0 or later, and the latest minor version is used.
    Note If the instance version does not meet the requirements, you can upgrade the instance version after assessing the compatibility with your business. For more information, see Upgrade the minor version and Upgrade the major version.
  • ApsaraDB for Redis instances of Community Edition instances and performance-enhanced instances of Enhanced Edition instances are used.

Background information

Slow logs are used to record requests whose execution time exceeds a specified threshold. The slow log is classified into the data node slow log and proxy slow log.
Note For standard instances, only slow logs of data nodes are collected.
Slow log type Description Parameter
Slow logs from data nodes
  • The command execution time collected in slow logs that are generated on a data node only includes the command execution time on the data node. The communication time between the data node and a proxy or client, and the latency of the command in the single-threading queue are not included.
  • The number of slow logs of data nodes is small due to the high performance of ApsaraDB for Redis.
  • slowlog-log-slower-than: specifies the threshold of command execution time. If the time consumed to run a command exceeds this threshold, the command is recorded in a slow log. Default value: 10000 μs (10 ms).
    Note Typically, the latency seems to be higher than the specified value of this parameter, because the specified value does not include the time consumed to transmit and process data among clients, proxies, and data nodes.
  • slowlog-max-len: specifies the maximum number of slow logs that can be stored. Default value: 1024.

For more information, see Parameter overview and configuration guide.

Slow logs from the proxy server
  • The command execution time collected in proxy slow logs starts from the time when the proxy server sends a request to a data node and ends at the time when the proxy receives the response from the data node. This time includes the command execution time on the data node, the data transmission time over the network, and the queuing latency of the command.
  • Proxy slow logs are retained for 72 hours. However, the number of proxy slow logs is not limited.
  • The latency recorded in proxy slow logs is similar to the latency that you experience on your application. We recommend that you check this type of logs when you troubleshoot timeout issues.
rt_threshold_ms: specifies the threshold of slow logs from the proxy server. Default value: 500 ms. We recommend that you set the threshold to a value that is similar to the client timeout. The value can be 200 to 500 ms.

For more information, see Parameter overview and configuration guide.

Methods used to query slow logs

Slow log type Method
Slow logs from data nodes Connect to the ApsaraDB for Redis instance from a client and run the SLOWLOG GET command. For more information, see RedisCommands.
Log on to the ApsaraDB for Redis console or call an API operation:
Slow logs from the proxy server

Procedure

In most cases, service timeouts may be caused by slow requests. We recommend that you perform the following steps to troubleshoot the timeout issues.

  1. If a service timeout issue occurs, check the slow logs generated on the proxy server first. For more information, see Query slow logs.
    Note
    • For standard instances, go to Step 3 and analyze slow logs of data nodes.
    • If no log exists, you can check the network between the client and the ApsaraDB for Redis instance.
  2. Find the command that is recorded by the earliest proxy slow log.
    Note If slow requests occur on data nodes and cause command accumulation, these requests are recorded in proxy slow logs.

    In this example, the earliest recorded slow log is caused by the KEYS command. The IP address on the right of the log entry is the IP address of the client that sends the command.

    Find the earliest slow log in proxy slow logs
  3. Check the data node slow logs to find the proxy slow logs that cause the timeout issue.
    Note
    • When you connect to a cluster instance of Community Edition by using the default endpoint (the endpoint of the proxy server), the Host Address displayed on the Data nodes tab are the IP addresses of proxy nodes rather than the client IP addresses. To avoid this restriction, you can use a private endpoint to connect to the instance. For more information, see Use a private endpoint to connect to an ApsaraDB for Redis instance. To use the endpoint of the proxy server to connect to an instance, you can use ApsaraDB for Redis Enhanced Edition (Tair). Tair provides client IP addresses in data node logs.
    • Typically, the command that generates slow logs first in the proxy slow log can also generate data node slow logs. The number of slow logs for a data node is less than that of the proxy server. This is due to the different definitions of the execution time and different thresholds of slow logs.

    In this example, after you view proxy slow logs, you can see that the slow log caused by the KEYS command also exists in data node slow logs. No other slow logs that are displayed on the Proxy tab exist on the Data nodes tab. This shows that the KEYS command causes the timeout.

    View slow logs on the Data nodes tab
  4. In proxy slow logs, you can search for the client IP address based on the command that is found in Step 2.