Tair (Redis® OSS-Compatible):Why memory alerts differ from monitored usage

Symptoms

Symptom 1:

You receive an instance memory alert indicating that the memory usage has exceeded the threshold (for example, the average value is >= 90% for three consecutive measurements), but the Performance Monitoring page in the console shows that the overall memory usage is low.

Symptom 2:

Your application returns a command not allowed when used memory > 'maxmemory' error, but the Performance Monitoring page shows that overall memory is not exhausted or that only a single data node has high memory usage.

Causes

Discrepancy between monitoring and alerts

A discrepancy between the memory usage in Performance Monitoring and a memory alert can occur if you are using a cluster instance. The alert often reflects the metrics of a single data node, while you are viewing the instance's overall metrics.

Check the details of the alert you received. If it contains the field nodeId = <Instance ID>-db-<Number>, it indicates that only the specific data node identified by <Instance ID>-db-<Number> has exceeded the memory threshold.

Follow these steps to confirm that the data node's memory usage matches the alert.

1. Log on to the console and go to the Instances page. In the top navigation bar, select the region in which the instance that you want to manage resides. Then, find the instance and click the instance ID.

2. In the left-side navigation pane, click Performance Monitoring.

3. Click the Data Node tab and select the data node that corresponds to <Instance ID>-db-<Number>. Check if its memory usage matches the value in the alert. You can view the monitoring charts for CPU Usage and Memory Usage for the selected Data Node.

Memory skew across data nodes

When one or more data nodes in a cluster instance have significantly higher memory usage than others, it indicates memory skew. You can use the instance diagnostics feature to quickly check your instance for skew issues.

What causes memory skew

The main causes of memory skew are:

• Presence of large keys.

  A cluster instance uses a cyclic redundancy check (CRC) algorithm to calculate which slot a key belongs to and then writes the data to the corresponding data shard.

  Even if keys are distributed evenly across data shards, a single key that stores too many or oversized fields can become a large key, leading to memory skew.

• Use of hash tags.

  When a hash tag is used, such as in user:{1000}:name, the instance performs the CRC calculation on the string within the curly braces ({}). Keys with the same hash tag are mapped to the same slot and stored on the same data shard. If a large number of keys share the same hash tag, data becomes concentrated on a single data shard, causing memory skew.

Solutions

Check for and split large keys

Find large keys

Use the offline key analysis feature to find any large keys.

For more methods to find large keys, see Large keys and hotkeys.

Split large keys

For example, split a HASH key with tens of thousands of members into multiple smaller HASH keys, each with a reasonable number of members. In a cluster instance, splitting large keys can significantly help balance memory usage across data shards.

Check hash tag usage

If you are using a hash tag that causes data concentration, consider splitting it into multiple hash tags according to your application logic. This helps distribute data more evenly across different data shards.

Upgrade the instance specification

Upgrading the instance specification increases the memory available to each shard and can be a temporary solution for memory skew. For detailed steps, see Change instance configuration.