Diagnostic reports help you evaluate the operational conditions of an ApsaraDB for Redis instance, such as performance level, skewed requests distribution, and slow queries. Diagnostic reports also help you identify anomalies on the instance.

Prerequisites

Create a diagnostic report

Components of a diagnostic report

  • Basic instance information: displays the basic information about an instance such as the instance ID, specification, type, and the zone in which the instance is deployed.
  • Summary: displays the score of the health status and describes the reasons why points are deducted.
  • Performance level: displays the statistics and status of key performance metrics related to the instance.
  • TOP 10 nodes that receive the greatest number of slow queries: displays the top 10 data nodes that receive the greatest number of slow queries and provides information about the slow queries.

Basic instance information

This section displays the instance ID, specifications, type, and the zone in which the instance is deployed.

Figure 1. Basic instance information
Basic instance information

Summary

This section displays the diagnostic results and the score of the instance health status. The highest score is 100. If your instance fails to achieve a score of 100, you can check the diagnostic items and details.

Figure 2. Summary
Summary

Performance level

Displays the statistics and status of key performance metrics related to the instance. You must pay attention to performance metrics that are in the Hazard state.

Note If your instance runs in a cluster architecture or read/write splitting architecture, you must check whether the performance metrics are skewed and check for skewed data nodes. In addition, we recommend that you focus on the data nodes with higher loads based on the curve charts of each performance metric in the Top 5 Nodes.
Figure 3. Performance level
Performance level
Performance metric Threshold Impact Possible cause and troubleshooting method
cpuUsage 60% When an ApsaraDB for Redis instance causes high CPU utilization, the throughput of the instance and the response time to clients are affected. In some cases, the clients may fail to respond.

Possible causes:

  • The instance runs commands that require high time complexity.
  • Hotkeys exist.
  • Connections are frequently established.

For more information about how to troubleshoot these issues, see Troubleshoot high CPU utilization on an ApsaraDB for Redis instance.

memoryUsage 80% When the memory usage of an ApsaraDB for Redis instance continuously increases, response time increases, queries per second (QPS) becomes unstable, and keys may be frequently evicted. This affects your business. Possible causes:
  • The memory is exhausted.
  • A great number of big keys exist.

For more information about how to troubleshoot these issues, see Troubleshoot the high memory usage of an ApsaraDB for Redis instance.

connectionUsage of data nodes 80% When the number of connections to a data node reaches the upper limit, new connection requests time out or fail.
Note
  • This metric is collected when clients connect to a cluster instance of ApsaraDB for Redis in direct connect mode.
  • This metric is not collected when clients connect to a cluster instance or read/write splitting instance by using the proxy nodes. In this case, you must monitor the number of connections on the proxy nodes. For more information, see Query monitoring data.

Possible causes:

  • User traffic spikes.
  • Idle connections are not released for a long period of time.

For more information about how to troubleshoot these issues, see Instance sessions.

inFlow 80% When the inbound or outbound traffic exceeds the maximum bandwidth of the instance specifications, the performance of applications is affected.

Possible causes:

  • Workloads spike.
  • Big keys are frequently read or written.

For more information about how to troubleshoot these issues, see Troubleshoot high data usage on an ApsaraDB for Redis instance.

outFlow 80%

If your instance runs in the cluster architecture or read/write splitting architecture, the system analyzes the preceding performance metrics. The system measures the overall access performance of the instance and displays the result in the diagnostic report. The following table describes the criteria used to determine skewed requests, possible causes, and troubleshooting methods.

Note If the diagnostic report indicates that the instance has skewed requests for a specific performance metric, you must check the nodes to which the skewed requests are directed.
Criteria Possible cause Troubleshooting method

The following conditions are met:

  • Peak values of performance metrics for all data nodes of an ApsaraDB for Redis instance are greater than the following thresholds:
    • CPU usage: 10%.
    • Memory usage: 20%.
    • Inbound and outbound traffic: 5 Mbit/s.
    • Connection usage: 5%.
  • The balance score is greater than 1.3, which is calculated by using the formula: max{average performance values of all data nodes}/the median performance value of all data nodes.

    For example, an ApsaraDB for Redis instance contains four data nodes and the average CPU usage of the nodes is 10%, 30%, 50%, and 60%. Then, the median value is 40% and the result is 1.5 from 60%/40%. The calculated value 1.5 is greater than 1.3. Therefore, the system considers the CPU usage of the instance skewed.

  • A data node has excessive big keys.
  • A data node has hotkeys.
  • The hash tags are improperly configured.
    Note If keys are configured with the same hash tag, the keys are stored on the same data node. If a large number of keys are configured with the same hash tag, the node is overwhelmed by these keys.

TOP 10 nodes that receive the greatest number of slow queries

This section displays the top 10 data nodes that receive the greatest number of slow queries and information about the corresponding slow queries. The statistics include:

  • The slow log data of data nodes is stored in the system audit log. The slow log data is retained for only four days.
  • The slow log data that is stored on the data node. Only the most recent 1,024 log entries are retained. You can use redis-cli to connect to the instance and run the SLOWLOG GET command to view the slow log.
Figure 4. Slow queries
Slow queries

You can analyze the slow queries and determine whether improper commands exist. This way, you can find the solutions to different issues.

Description Solution

Commands that implement a time complexity of O(N) or consume more resources, such as keys *.

Evaluate and disable commands that cause high risk and consume a large amount of resources, such as FLUSHALL, KEYS, and HGETALL. For more information, see Disable high-risk commands.
Big keys that are frequently read from and written to the data nodes. Analyze and evaluate the big keys. For more information, see Use the cache analytics feature to find big keys. Then, split the big keys based on your business requirements.