ApsaraDB for Redis:Customize the size of macro nodes in streams

Prerequisites

• The engine version of the ApsaraDB for Redis instance is Redis 5.0 or later.
• The ApsaraDB for Redis instance uses the standard master-replica architecture.

Relationship between Redis streams and macro nodes

Stream is a new data type introduced in Redis 5.0. A Redis stream stores data on delta-compressed macro nodes that are connected in a radix tree. Each macro node stores multiple stream entries. You can use this data structure to access random elements, obtain elements within a specified range, and create capped streams with high efficiency. This data structure also significantly optimizes memory usage.

You can specify the stream-node-max-entries parameter to limit the maximum number of stream entries supported by each macro node. You can specify the stream-node-max-bytes parameter to limit the maximum amount of memory that can be consumed by each macro node.

• stream-node-max-entries: The default value is 100, which indicates that each macro node can store up to 100 stream entries. Valid values: 0 to 999,999,999,999,999. A value of 0 indicates that no limit exists. If the number of stream entries stored in a macro node reaches the upper limit, new stream entries are stored on a new macro node.
• stream-node-max-bytes: Unit: bytes. The default value is 4096, which indicates that each macro node can consume up to 4,096 bytes of memory. Valid values: 0 to 999,999,999,999,999. A value of 0 indicates that no limit exists.

Scenarios

You can specify the stream-node-max-entries parameter to adjust the length deviation of a fixed-length message queue.

If your application does not need to permanently store messages, you can use the MAXLEN parameter to specify the maximum number of messages that can be stored in a stream when you run the XADD command to add a message to the stream. Example:

XADD cappedstream MAXLEN 5000 * field value5001 //Add a value of value5001 to field1 of cappedstream and set the maximum number of messages to 5,000.

If you set the maximum queue length to an exact value such as 5,000 messages, performance is significantly degraded. To optimize memory usage, a Redis stream stores data on delta-compressed macro nodes that are connected in a radix tree. Each time ApsaraDB for Redis deletes a message, it must search the macro node for the message and mark the message as deleted. This mechanism is not optimal for high-throughput ApsaraDB for Redis services where messages are frequently added and deleted. The performance of ApsaraDB for Redis degrades if it frequently deletes messages. Therefore, we recommend that you add a tilde (~) in the XADD command to specify an approximate maximum length. Example:

XADD cappedstream MAXLEN ~ 5000 * field value1 //Add a value of value5001 to field1 of cappedstream and set the maximum number of messages to approximately 5,000.

This way, the actual length of the stream can be an approximate value greater than or equal to the specified value. For example, the stream may contain 5000, 5050, or 5060 messages. The deviation from 5000 depends on the number of macro nodes in the stream and the maximum number of messages that can be stored on each macro node. ApsaraDB for Redis calculates the approximate value based on the stream-node-max-entries parameter. This parameter specifies the maximum number of messages that can be stored on each macro node. If the number of messages stored in the stream exceeds this approximate value, ApsaraDB for Redis deletes the macro node that stores the earliest messages, instead of deleting specific messages.

The value of the stream-node-max-entries parameter determines the length deviation of a fixed-length message queue. To reduce the deviation, you can set the parameter to a proper smaller value.

Procedure

1. Log on to the ApsaraDB for Redis console and go to the Instances page. In the top navigation bar, select the region in which the instance that you want to manage resides. Then, find the instance and click its ID.
2. In the left-side navigation pane, click System Parameters.
3. On the page that appears, find the stream-node-max-bytes parameter and click Modify in the Actions column.
4. In the dialog box that appears, perform the following steps:
   1. Change the value of the stream-node-max-bytes parameter based on your business requirements.
   2. Click OK.
   

Related API operations