ApsaraMQ for RocketMQ:Consumption retries

What triggers a retry

A retry is triggered when either of the following occurs:

• The consumer returns a failure status or throws an unhandled exception.

• The message processing times out (including queuing timeouts for PushConsumer).

What happens after retries are exhausted

When a message exceeds the maximum retry count:

• Dead-letter enabled: The message is delivered to a dead-letter topic. Consume messages from this topic to investigate and recover failed operations. For details, see Dead-letter messages.

• Dead-letter disabled: The message is permanently discarded.

Retry state machine

A message consumed by a PushConsumer transitions through the following states:

Actual interval between delivery attempts

The retry interval is only one component of the total time between two consecutive delivery attempts. The actual interval also includes the processing time and the time spent in the Ready state before the consumer pulls the message.

Actual interval = Processing time + Retry interval + Ready-state duration

Example: A message spends 5 seconds in the Ready state and 6 seconds being processed before failing.

1. At 0 s, the message enters the Ready state for the first delivery.

2. At 5 s, the consumer pulls the message and begins processing.

3. At 11 s, processing fails and the consumer returns a failure status.

4. The retry interval elapses (10 seconds for the first retry).

5. At 21 s, the message returns to the Ready state.

6. At 26 s, the consumer pulls the message for the second attempt.

Total elapsed time between the two consumption attempts: 5 s (ready) + 6 s (processing) + 10 s (retry interval) = 21 seconds.

Retry interval schedule for unordered messages

Unordered messages use a tiered backoff schedule. The interval increases with each retry to prevent repeated failures from consuming excessive resources.

Retry interval for ordered messages

Ordered messages use a fixed retry interval instead of tiered backoff, because preserving message order requires predictable retry timing. For the specific interval value, see Parameter limits.

Maximum retries

• Default: 16

• Maximum: 1,000

The maximum retry count is set in the consumer group metadata. If the maximum is set to 3, a message can be delivered up to 4 times total: 1 original delivery + 3 retries.

To modify this value, see Configure the maximum retry count.

Code example

To trigger a retry for a PushConsumer, return a failure status from the message listener. The SDK also catches unhandled exceptions and treats them as failures.

// Return FAILURE from the message listener to trigger a retry.
// The server retries the message automatically until the maximum retry count is reached.
MessageListener messageListener = new MessageListener() {
    @Override
    public ConsumeResult consume(MessageView messageView) {
        System.out.println(messageView);
        // Return FAILURE to trigger a retry.
        return ConsumeResult.FAILURE;
    }
};

View retry logs for ordered messages

Retries for ordered message consumption by a PushConsumer occur on the consumer client. The server does not capture detailed retry logs. If a message trace shows a delivery result of "failed" for an ordered message, check the consumer client logs.

For the default log path, see Log configurations.

Search for these keywords to locate retry-related entries:

Message listener raised an exception while consuming messages
Failed to consume fifo message finally, run out of attempt times

Retry state machine

A message consumed by a SimpleConsumer transitions through the following states:

Unlike PushConsumer, SimpleConsumer has no WaitingRetry state. The retry interval is derived from the InvisibleDuration parameter set when receiving messages. This duration defines the maximum time allotted for processing. If the consumer does not acknowledge the message before the invisibility duration expires, the message becomes visible again and is redelivered.

Retry interval

Retry interval = InvisibleDuration - Actual processing time

Example: If InvisibleDuration is set to 30 ms and processing takes 10 ms before a failure response is returned, the next retry occurs after 20 ms. If processing exceeds the full 30 ms with no acknowledgment, the message times out and is retried immediately (interval = 0 ms).

Extend the invisibility duration

Because InvisibleDuration is set in advance, it may not match the actual processing time for every message. For example, if you set the maximum processing time to 20 ms but the message cannot be processed within that time, you can extend the invisibility duration using the API. This prevents the message from triggering a premature retry while processing is still in progress.

The change takes effect immediately. The new duration is calculated from the moment the API call is made.

Requirements for extending InvisibleDuration:

• The message has not yet timed out.

• The message has not been acknowledged (committed).

Maximum retries

• Default: 16

• Maximum: 1,000

The maximum retry count is set in the consumer group metadata. If the maximum is set to 3, a message can be delivered up to 4 times total: 1 original delivery + 3 retries.

To modify this value, see Configure the maximum retry count.

Code example

To trigger a retry for a SimpleConsumer, let the invisibility duration expire without acknowledging the message. The server then automatically redelivers it.

// Receive messages with a 30-second invisibility duration.
// To trigger a retry, skip the acknowledgment and let the invisibility duration expire.
List<MessageView> messageViewList = null;
try {
    messageViewList = simpleConsumer.receive(10, Duration.ofSeconds(30));
    messageViewList.forEach(messageView -> {
        System.out.println(messageView);
        // Skip acknowledgment to let the server retry after the invisibility duration expires.
    });
} catch (ClientException e) {
    // Handle receive failures (e.g., throttling). Initiate a new receive request.
    e.printStackTrace();
}

gRPC SDK

Use either of these methods:

• API: Call the UpdateConsumerGroup operation.

• Console:

  1. Go to the Instances page and click the target instance name.

  2. In the left navigation pane, click Groups.

  3. Click Create Group and configure the retry policy.

Remoting SDK

Set the maxReconsumeTimes property on the consumer object.

// Set the maximum number of consumption retries for this consumer.
consumer.setMaxReconsumeTimes(3);

Use retries for transient failures only

Consumption retries are a fault recovery mechanism, not a flow control tool. Route persistent or expected failures through normal processing logic.

Monitor dead-letter topics

When dead-letter messages are enabled, set up monitoring on the dead-letter topic. Messages in this topic represent processing failures that could not be recovered through retries. Investigate and reprocess these messages promptly to prevent data inconsistency.

Recommended monitoring actions:

• Configure alerts on the dead-letter topic message count. A spike in dead-letter messages often indicates a systemic issue rather than a transient failure.

• Track the retry count of messages arriving in the dead-letter topic. If most messages exhaust all retries, your retry count may be too low, or the failure is not transient.

• Regularly consume and reprocess dead-letter messages after the root cause is resolved.

How do I set the consumption timeout?

The timeout is configured on the consumer client. The available range depends on the protocol and consumer type.

// SimpleConsumer: InvisibleDuration bounds
private long minInvisiableTimeMillsForRecv = Duration.ofSeconds(10).toMillis();
private long maxInvisiableTimeMills = Duration.ofHours(12).toMillis();

// Remoting protocol: set consumption timeout in minutes
consumer.setConsumeTimeout(15); // Range: 1 to 180 minutes