Troubleshoot and Resolve Out-of-Memory Issues - Tair (Redis® OSS-Compatible)

When a Tair (Redis OSS-compatible) instance runs out of memory, it can cause issues such as frequent key eviction, increased response times, and unstable QPS, which can affect your business operations. If your instance memory is full or you receive memory alerts, use this topic to determine if memory usage is consistently high, has suddenly increased, or is skewed. You can then resolve the issue by splitting large keys, setting an expiration policy, or upgrading the instance specifications.

Scenarios of high memory usage

High memory usage typically falls into one of the following three scenarios:

The memory usage remains at a high level for an extended period. Generally, you should take action if the memory usage exceeds 95%.
The memory usage is consistently low but suddenly spikes to a high level, sometimes reaching 100%.
The overall memory usage of the instance is low, but the memory usage of a specific data node is close to 100%.

To reduce memory usage, apply the solution that matches your scenario.

Solutions for consistently high memory usage

Check if your existing Keys meet business requirements, and promptly clean up any unnecessary Keys.
You can use the cache analysis feature to analyze the distribution of large keys and the TTL expiration policies for keys. For more information, see Offline key analysis.
1. Analyze whether the key has a reasonable TTL policy.
  Note
  In the following example, no expiration time is set for any keys. We recommend that you evaluate your business needs and set appropriate expiration times on the client side.
  Figure 1. Example of key expiration time distribution
2. Evaluate the large key and split it on the application side.
  On the cache analysis page, click the Large Key Analysis tab to view details. The table displays information such as key name, key type, and data size. You can click the Export button to download the analysis results to your local machine.
Set a proper eviction policy based on your business requirements by configuring the maxmemory-policy parameter. For specific instructions, see Set parameters.
Note
The default eviction policy for an instance is volatile-lru. For more information, see Eviction policies.
Set a reasonable execution frequency for actively deleting expired keys based on your business needs by adjusting the value of the hz parameter. For specific instructions, see Adjust the execution frequency of scheduled tasks.
Note
We recommend setting the hz parameter to a value less than 100. A larger value can significantly impact CPU usage. You can also enable automatic adjustment if your instance is compatible with Redis 5.0 or later. For more information, see Enable dynamic frequency control for background tasks.
If memory usage is still high after you perform the preceding optimizations, consider upgrading to a specification with more memory to handle more data and improve performance. For more information, see Change instance configuration.
Note
Before you upgrade your instance, you can purchase a pay-as-you-go instance to test whether the target specification meets your workload requirements. You can release the instance after the test is complete. For information about how to release an instance, see Release pay-as-you-go instances.

Solutions for sudden memory increases

Causes

A sudden increase in memory usage can be caused by the following:

A sudden memory spike has one of four causes. Use the key indicator in each row to identify which applies.

Cause	Key indicator
Large volume of new data written	Inbound traffic and write QPS spike alongside memory usage
Surge in new connections	Connection count spikes alongside memory usage
Burst traffic causing buffer backlog	Bandwidth usage reaches 100%; `clients.normal` memory is high
Slow client causing output buffer backlog	`big_client_buf` equals 1 in `MEMORY DOCTOR` output

Large volume of new data written

A rapid influx of new data directly increases memory consumption.

Diagnose

On the Performance Monitor page, check inbound traffic and write queries per second (QPS). If both metrics follow the same trend as memory usage, the spike is due to a large volume of new data.

Resolve

Set appropriate time-to-live (TTL) values on keys to automatically expire data that is no longer needed. Alternatively, delete unnecessary keys manually.
Increase the memory capacity of the instance. For details, see Change the configurations of an instance.
If your instance is a standard instance and memory usage remains high after increasing capacity, upgrade to a cluster instance. Cluster instances distribute data across multiple shards, reducing memory pressure on each shard. For details, see Change the configurations of an instance.

Surge in new connections

Each client connection reserves memory for its input and output buffers. A sudden surge in connections causes a corresponding rise in total memory usage.

Diagnose

On the Performance Monitor page, check the number of connections. If the connection count rises sharply alongside memory usage, a connection surge is the cause.

Resolve

Check for connection leaks in your application.
Configure a connection timeout to automatically close idle connections. For details, see Specify a timeout period for client connections.

Burst traffic causing buffer backlog

When burst access generates traffic that exceeds the instance's network bandwidth, incoming and outgoing data accumulates in the input and output buffers. This buffer backlog consumes additional memory on top of normal data storage.

Diagnose

Check whether inbound and outbound traffic usage has reached 100%.
Run MEMORY STATS and check whether clients.normal occupies an excessive amount of memory.
Note
clients.normal reports the total memory consumed by the input and output buffers across all normal client connections. A high value here confirms that buffer backlog is driving the memory spike.

Resolve

Identify the cause of the traffic burst.
Increase the network bandwidth of the instance. For details, see Manually increase the bandwidth of an instance and Enable bandwidth auto scaling.
Upgrade the instance specifications to provide sufficient buffer capacity. For details, see Change the configurations of an instance.

Slow client causing output buffer backlog

When a client cannot consume responses fast enough, undelivered data accumulates in the server-side output buffer. Redis allocates memory for each queued response, which can grow large when clients fall behind.

Diagnose

In redis-cli, run MEMORY DOCTOR and check the value of big_client_buf. If big_client_buf equals 1, at least one client has an oversized output buffer consuming significant memory.

Resolve

Run CLIENT LIST and look for clients with a high omem value. The omem field reports the memory used by a client's output buffer. Check whether the identified client has performance issues preventing it from consuming responses fast enough.

Solutions for high memory on a data node

Symptoms

For instances with a cluster architecture, high memory usage on a specific data node is indicated by the following symptoms:

You receive a memory usage alert from CloudMonitor. The alert message indicates that the memory usage of a specific data node has exceeded the threshold.
The instance diagnostic report shows a memory usage skew.
On the performance monitoring page, you notice that the overall instance memory usage is low, but a specific data node has high memory usage.

Cause

If overall instance memory usage is low while a specific data node's usage is high, this indicates a memory usage skew.

Solutions

Check for and split large keys

Find large keys

Use the offline key analysis feature to find any large keys.

For more methods to find large keys, see Large keys and hotkeys.

Split large keys

For example, split a HASH key with tens of thousands of members into multiple smaller HASH keys, each with a reasonable number of members. In a cluster instance, splitting large keys can significantly help balance memory usage across data shards.

Check for hash tag usage

If you are using a hash tag that causes data concentration, consider splitting it into multiple hash tags according to your application logic. This helps distribute data more evenly across different data shards.

Upgrade the instance specification

Upgrading the instance specification increases the memory available to each shard and can be a temporary solution for memory skew. For detailed steps, see Change instance configuration.

Important

The system initiates a precheck for data skew during instance specification change. If the instance type that you select cannot handle the data skew issue, the system reports an error. Select an instance type that has higher specifications and try again.
After you upgrade the instance specifications, memory usage skew may be alleviated. However, skew may also occur on bandwidth and CPU resources.

Appendix 1: Redis memory usage

A Redis instance's memory consists of three main components:

Memory component	Description
Link memory (dynamic)	This includes memory for the input buffer, output buffer, JIT overhead, fake Lua links, and the Lua execution cache. You can run the INFO command to check client cache information in the Clients section of the output. Note The input and output buffers are associated with each client connection and are typically small. However, when you perform range-based operations or when large keys are transferred slowly, the memory used by these buffers can increase, which affects the data area and can potentially cause an out-of-memory (OOM) error.
Data memory	This is the user data area, which stores the actual value information. It is usually the primary focus of analysis.
Management memory (static)	This area is small and relatively constant after startup. It consists of memory overhead from data management hash tables, the replication buffer, and the AOF buffer (approximately 32 MB to 64 MB). Note For a very large number of keys (for example, hundreds of millions), this area can consume a significant amount of memory.

Note

Ineffective dynamic memory management causes most OOM scenarios. For example, request queuing during rate limiting can cause dynamic memory to increase rapidly. Overly complex or poorly written Lua scripts can also lead to OOM. To prevent these issues, we recommend Tair (Enterprise Edition), which provides enhanced control over dynamic memory.

Appendix 2: Checking memory usage

MEMORY STATS command

In Redis, run the MEMORY STATS command to query memory usage details.

The memory overhead of a Redis instance consists of two main parts:

Memory overhead from business data. This is typically the primary focus of analysis.
Memory overhead from non-business data, such as the backlog buffer for master-replica replication and the memory consumed during Redis process initialization.

The following section shows a sample response and an explanation of each parameter:

Note

In the following response, memory values are in bytes.

 1) "peak.allocated" // The peak memory consumed by the Redis process since startup.
 2) (integer) 79492312
 3) "total.allocated" // The total number of bytes allocated by Redis, which is the current total memory usage.
 4) (integer) 79307776
 5) "startup.allocated" // The initial amount of memory consumed by Redis at startup.
 6) (integer) 45582592
 7) "replication.backlog" // The size of the replication backlog buffer.
 8) (integer) 33554432
 9) "clients.slaves" // The total size of read and write buffers for all replica nodes in master-replica replication.
10) (integer) 17266
11) "clients.normal" // The total size of read and write buffers for all regular clients (excluding replica nodes).
12) (integer) 119102
13) "aof.buffer" // The buffer used for AOF persistence and the buffer generated during AOF rewrite.
14) (integer) 0
15) "db.0"  // The number of business databases.
16) 1) "overhead.hashtable.main" // The total memory overhead for the hash tables of the current database, which is the metadata memory.
    2) (integer) 144
    3) "overhead.hashtable.expires" // The memory consumed to store key expiration times.
    4) (integer) 0
17) "overhead.total" // The sum of all overhead memory. Calculated as: startup.allocated + replication.backlog + clients.slaves + clients.normal + aof.buffer + db.X.
18) (integer) 79273616
19) "keys.count" // The total number of keys in the current Redis instance.
20) (integer) 2
21) "keys.bytes-per-key" // The average size of each key in the current Redis instance. Calculated as: (total.allocated - startup.allocated) / keys.count.
22) (integer) 16862592
23) "dataset.bytes" // The size of memory occupied by pure business data.
24) (integer) 34160
25) "dataset.percentage" // The percentage of memory occupied by pure business data. Calculated as: dataset.bytes * 100 / (total.allocated - startup.allocated).
26) "0.1012892946600914"
27) "peak.percentage" // The ratio of the current total memory to the historical peak memory. Calculated as: total.allocated * 100 / peak.allocated.
28) "99.767860412597656"
29) "fragmentation" // The memory fragmentation ratio.
30) "0.45836541056632996"

MEMORY DOCTOR command

In Redis, run the MEMORY DOCTOR command to obtain memory diagnostic suggestions.

Figure 2. Example of diagnostic results

r-bpxxxxxxxxxxxxxxxxxxxx.redis.rds.aliyuncs.com:6379> MEMORY DOCTOR
"Sam, I detected a few issues in this Redis instance memory implants:\n\n * High
 allocator RSS overhead: This instance has an RSS memory overhead is greater tha
n 1.1 (this means that the Resident Set Size of the allocator is much larger tha
n the sum what the allocator actually holds). This problem is usually due to a l
arge peak memory (check if there is a peak memory entry above in the report), yo
u can try the MEMORY PURGE command to reclaim it.\n\nI'm here to keep you safe,
 Sam. I want to help you.\n"

MEMORY DOCTOR provides memory diagnostic suggestions for your Redis instance based on the following dimensions. You can use these suggestions to formulate corresponding optimization strategies:

    int empty = 0;     /* Instance is empty or almost empty. */
    int big_peak = 0;       /* Memory peak is much larger than used memory. */
    int high_frag = 0;      /* High fragmentation. */
    int high_alloc_frag = 0;/* High allocator fragmentation. */
    int high_proc_rss = 0;  /* High process RSS overhead. */
    int high_alloc_rss = 0; /* High RSS overhead from the allocator. */
    int big_slave_buf = 0;  /* Replica buffers are too big. */
    int big_client_buf = 0; /* Client buffers are too big. */
    int many_scripts = 0;   /* Script cache has too many scripts. */

MEMORY USAGE command

In Redis, run the MEMORY USAGE command to query the memory consumed by a specified Key (in bytes).

Sample command:

MEMORY USAGE Key0089393003

Sample response:

(integer) 1000072