If a message fails to be consumed, message retry is automatically triggered. If the message still fails to be consumed after reaching the maximum number of retries, it becomes a dead-letter message. ApsaraMQ for RocketMQ lets you store these dead-letter messages in a specified topic for subsequent business recovery or retrospective analysis. This topic describes the scenarios, dead-letter policies, limits, configuration methods, and usage notes for dead-letter messages.
Scenarios
-
Typical dead-letter message handling scenario
After message retries fail, store dead-letter messages in a designated dead-letter topic. Create a separate consumer group to consume and process these messages, enabling error-path handling or dead-letter analysis.
-
Typical misuse of dead-letter messages
If you route dead-letter messages through multiple layers or dump them back into the original topic, they may re-enter the retry loop. This can trigger an avalanche effect.
Dead-letter policy
When messages become dead letters
If a message is not successfully consumed after reaching the maximum retry count, it stops being delivered and becomes a dead-letter message.
Dead-letter message retention rules
ApsaraMQ for RocketMQ By default, dead-letter messages are not retained and are discarded after they enter the dead-letter state.
You can enable dead-letter message retention in the console. Once enabled, dead-letter messages are stored in a specified topic known as the dead-letter topic. For details, see Configure dead-letter message retention rules.
Dead-letter messages are stored in the dead-letter topic as new messages. Their properties change as follows:
-
Message ID: A new message ID is generated when the dead-letter message is stored in the dead-letter topic.
-
User-defined attributes, message body, and other user-provided information remain unchanged.
-
The retention period for dead-letter messages starts counting from the time they enter the dead-letter topic. For example, if a message is sent to the server at 13:00:00 and fails consumption and retries two hours later (at 15:00:00), its retention period begins at 15:00:00.
Limits
-
The dead-letter topic must be a normal message topic or an ordered message topic. Transactional message topics and scheduled message topics cannot serve as dead-letter topics.
-
You cannot use the original message’s topic as the dead-letter topic (to prevent circular retries and avalanche effects). If the system detects that the dead-letter topic matches the original topic during transfer, the message is discarded.
-
Dead-letter messages from different topics can be stored in the same dead-letter topic.
-
Deleting a consumer group does not delete its associated dead-letter topic.
-
If a topic is referenced by a dead-letter policy, you must remove this policy before deleting the topic.
Configure dead-letter message retention rules
You can configure whether to retain dead-letter messages in the ApsaraMQ for RocketMQ console.
To access this setting:
-
On the Instances page, click the target instance name.
-
In the navigation pane on the left, click Groups. On the Groups page, click Create Group.
In the Create Group dialog box, expand the Advanced Settings section to configure the Consumption Retry Policy (default: exponential backoff), Maximum Retry Count (default: 16), and the Retain Dead-letter Messages toggle. When this toggle is off, messages exceeding the retry limit are discarded.
Observability metrics for dead-letter messages
Metric descriptions
|
Metric type |
Metric |
|
rocketmq_send_to_dlq_messages: Number of messages entering dead-letter status per minute |
|
|
Using metrics
ApsaraMQ for RocketMQ supports configuring alerts for dead-letter messages, which helps you detect anomalies before business feedback occurs and further identify their source by reviewing Dashboard metrics.
-
Scenario 1: View the number of messages entering dead-letter status per minute
In the Dashboard, check the gauge for Messages entering dead-letter status per minute. You can also use Cloud Monitor to set up an alert for Messages entering dead-letter status per minute (GroupId & Topic).
-
Scenario 2: Check how many dead-letter messages remain unprocessed
After dead-letter messages are stored in a designated topic, view the Consumption backlog metric for that dead-letter topic in the Dashboard. Alternatively, use Cloud Monitor to create an alert for Message backlog (GroupId & Topic).
Best practices
How consumers can retrieve the original topic information
-
Option 1: Map each dead-letter topic one-to-one with its original topic.
For example, if the original topic is testTopic, name the dead-letter topic DLQ-testTopic.
-
Option 2: Embed the original topic name in a custom message attribute. Example:
messageBuilder.addProperty("originalTopic","testTopic")
Separate dead-letter processing from main business logic
Dead-letter messages are those that failed even after retries in normal business flows. Handle them separately to avoid disrupting regular operations.
-
The dead-letter topic must differ from the original message topic. Reusing the original topic can cause dead-letter messages to re-enter the consumption loop, blocking normal consumer groups and triggering an avalanche effect.
-
Do not use consumer groups from your main production flow to consume dead-letter messages. This could interfere with normal message processing.