Message Queue for Apache RocketMQ provides a troubleshooting feature that analyzes exceptions from log files, runs backend service diagnostics on instances, and provides actionable suggestions. The diagnostic process may probe your instance for analysis but does not affect your instance's configuration or services.
Region limitations
The troubleshooting feature is available only in the following regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), and China (Chengdu).
Prerequisites
A RAM user must have the required permissions to use the troubleshooting feature. For more information, see Example 4: Grant a RAM user all permissions on instance diagnostics.
Log diagnostics
This feature diagnoses issues from log files with the .log extension and provides suggestions in a diagnostic report. For information about specific issues and their suggested resolutions, see Log diagnostics suggestions.
Procedure
Log on to the Message Queue for Apache RocketMQ console. In the top navigation bar, select a region, for example, China (Hangzhou).
In the navigation pane on the left, choose .
On the Troubleshooting page, click the Log Diagnostics tab. In the Log section, upload a log file with the
.logextension, and then click Submit Diagnostics.NoteThe uploaded log file cannot exceed 64 MB.
On the Diagnostic Report page, wait a few minutes for the task to finish, and then view the results. Alternatively, you can click View Later and Exit. On the Diagnostic History page, after the status of the diagnostic task that you just created changes to Complete, click Details in the Actions column to return to the Diagnostic Report page and view the diagnostic results.
Backend service diagnostics
This feature diagnoses an instance's backend service and provides suggestions in a diagnostic report. For information about specific issues and their suggested resolutions, see Backend service diagnostics suggestions.
Procedure
Log on to the Message Queue for Apache RocketMQ console. In the top navigation bar, select a region, for example, China (Hangzhou).
In the navigation pane on the left, choose .
On the Troubleshooting page, click the Backend Service Diagnostics tab. Configure the parameters, and then click Submit Diagnostics.
5.x instances
Parameter
Description
Example
Instance
Select the ID of the Message Queue for Apache RocketMQ instance.
rmq-cn-vkl42*****
Time Range
Select the time range for the diagnostics.
2024-12-24 10:30:31 - 2024-12-24 11:30:31
4.x instances
Parameter
Description
Example
Instance
Select the ID of the Message Queue for Apache RocketMQ instance.
rmq-cn-vkl42*****
Topic Name
Select the topic to diagnose.
test_topic
SDK Type
Select the SDK type.
HTTP SDK
TCP SDK
TCP SDK
Time Range
Select the time range for the diagnostics.
2024-12-24 10:30:31 - 2024-12-24 11:30:31
On the Diagnostic Report page, wait a few minutes for the task to finish, and then view the results. Alternatively, you can click View Later and Exit. On the Diagnostic History page, after the status of the diagnostic task that you just created changes to Complete, click Details in the Actions column to return to the Diagnostic Report page and view the diagnostic results.
Network Connectivity Troubleshooting
If you experience connection issues between your client and the ApsaraMQ for RocketMQ instance, use the following scenario-based steps for targeted diagnosis.
VPC peering connection — Check whether the VPC route table of the ECS instance contains a route entry with the RocketMQ instance IP range as the destination and the peering connection ID as the next hop. Also verify that /etc/resolv.conf on the ECS instance is configured with Alibaba Cloud internal DNS servers (100.100.2.136 and 100.100.2.138).
Public network RemotingConnectException — Run nc -vz <endpoint> 8080 to test basic network connectivity. If the network is reachable but the error persists, compare your client code against the official SDK sample code to rule out configuration issues.
VPC internal network RemotingConnectException (Serverless instance) — Check whether the security group associated with the RocketMQ instance allows inbound traffic on ports 8080 and 8081. Also verify that the ECS security group's outbound rules allow traffic on those ports.
General port check — Verify the security group configurations of both the RocketMQ instance and the ECS instance, and confirm that the MQ allow-list includes the relevant ECS IP addresses.
Topic access topology
This feature shows the producers and consumers that access a specific topic on an instance during a selected time range.
This feature is supported only for Message Queue for Apache RocketMQ 5.x instances.
Procedure
Log on to the Message Queue for Apache RocketMQ console. In the top navigation bar, select a region, for example, China (Hangzhou).
In the navigation pane on the left, choose .
On the Troubleshooting page, click the Topic Access Topology tab. Configure the parameters, and then click Submit Diagnostics.
Parameter
Description
Example
Instance
Select the ID of the Message Queue for Apache RocketMQ instance.
rmq-cn-vkl42*****
Topic Name
Select the topic to query.
testTopic
Time Range
Select the time range for the diagnostics.
2024-12-24 10:30:31 - 2024-12-24 11:30:31
On the Diagnostic Report page, wait a few minutes for the task to finish, and then view the results. Alternatively, you can click View Later and Exit. On the Diagnostic History page, after the status of the diagnostic task that you just created changes to Complete, click Details in the Actions column to return to the Diagnostic Report page and view the diagnostic results.
Troubleshooting suggestions
Log diagnostics
Issue | Suggestion |
Unrecorded Logs Exist | The diagnostic system did not record certain logs. For assistance, submit a ticket. |
Signature Algorithm Not Found | Check your Java Development Kit (JDK) and operating system versions, and check for dependency conflicts. |
Consumption Exception | Check your consumer code. If the issue persists, submit a ticket. Messages not consumed (console shows "Processing" but no client-side logs) — Add a log statement at the consumer entry point to print the message ID, confirming whether messages are reaching the client. If the console shows no delivery trace and the client has no logs, focus your investigation on the client-side receive layer. Ordered messages stuck (manual ACK ineffective) — Ordered messages are processed sequentially within a queue. If one message is stuck, all subsequent messages in that queue are blocked. The server does not support manual ACK for ordered messages. Investigate your consumer-side business logic for excessive processing time or deadlocks. Ordered consumption repeated pull — This typically occurs when the consumer fails to process a message (throws an exception or does not return ACK before timeout), triggering a client-side retry. Check your local client logs at consumeTimeout parameter not taking effect — Verify that your SDK version supports the |
Failed to Send Heartbeat | Verify that the topic and group resources exist. If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. If the issue persists, submit a ticket. |
Failed to Acknowledge Consumption | If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. If the issue persists, submit a ticket. |
Failed to Pull Message | Verify that the topic and group resources exist. If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. If the issue persists, submit a ticket. |
Failed to Send Message | Verify that the topic resource exists. If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. If the issue persists, submit a ticket. Frequent producer instance creation causing send failures — If the error contains ONSClientException error — In addition to checking security group ports, the MQ allow-list, and telnet connectivity, verify that your code correctly sets the Namespace parameter: |
Failed to Obtain Route Information | Verify that the topic resource exists. If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. If the issue persists, submit a ticket. |
Failed to Disconnect Client | If this issue is accompanied by a network or backend service exception, follow the suggestions for those issues. You can ignore this error if it occurs when the application process terminates. |
Throttling Triggered on Broker | Log on to the Message Queue for Apache RocketMQ console. On the instance dashboard, check the producer and consumer TPS and the number of throttled requests to determine if an instance upgrade is needed. For more information, see Upgrade or downgrade an instance. |
Consumer Group Not Found | Verify that the group for the client subscription exists. Log on to the Message Queue for Apache RocketMQ console. On the Instances page, click the instance that the client accesses. On the Groups page, check if the group exists. |
Invalid AccessKey ID | Verify that the authentication credentials in your client are correct.
|
Invalid AccessKey Secret | Verify that the authentication credentials in your client are correct.
|
Invalid Scheduled Time for Scheduled Messages | Adjust the message delivery delay or upgrade your instance. For information about the supported delivery delay for each instance edition, see Quotas and limits. |
Invalid Message Attribute | Review the error log and check for conflicting message properties. For more information, see Internal message properties. |
Message Type Mismatch | Ensure that the message type configured for the topic in the console matches the message type your code sends. |
Full Local Message Cache | This indicates that the message consumption rate is too low. Review your consumption logic. |
Network Error | A network error occurred. For assistance, submit a ticket. |
No Online Consumer Client | Log on to the Message Queue for Apache RocketMQ console. On the Instances page, click the instance that the client accesses. Then, on the Groups page, click the group ID used by the client.
Messages not consumed (console shows "Processing" but no client-side logs) — Add a log statement at the consumer entry point to print the message ID, confirming whether messages are reaching the client. If the console shows no delivery trace and the client has no logs, focus your investigation on the client-side receive layer. Ordered messages stuck (manual ACK ineffective) — Ordered messages are processed sequentially within a queue. If one message is stuck, all subsequent messages in that queue are blocked. The server does not support manual ACK for ordered messages. Investigate your consumer-side business logic for excessive processing time or deadlocks. Ordered consumption repeated pull — This typically occurs when the consumer fails to process a message (throws an exception or does not return ACK before timeout), triggering a client-side retry. Check your local client logs at consumeTimeout parameter not taking effect — Verify that your SDK version supports the |
Local Request Missing | Possible causes:
|
Backend Service Exception | A broker-side exception occurred. For assistance, submit a ticket. |
Inconsistent Subscriptions | Log on to the Message Queue for Apache RocketMQ console. On the Instances page, click the instance that the client accesses. On the Groups page, click the group ID that the client uses. On the Group Details page, check the Subscriptions section for inconsistencies. Ensure all consumers in a group subscribe to the same topics. For more information, see Subscriptions. |
Subscription Does Not Exist | Check whether the consumer client is online and the network is functioning correctly. |
Topic Not Found | Verify that the topic your client uses exists. Log on to the Message Queue for Apache RocketMQ console. On the Instances page, click the instance that the client accesses. On the Topics page, check if the topic exists. |
Request Code Not Supported | A common example is error code 320, which indicates that batch messaging is not supported. |
Backend service diagnostics
Check item | Issue | Suggestion |
Instance | Instance Does Not Exist | Check the instance name. |
Instance Not Running | Check the instance status. | |
Topic | Topic Does Not Exist | Check if the topic was created in the console. |
Broker | Machine Failure | A machine in the backend broker cluster has failed. This may cause a temporary network error, which you can ignore. |
Broker Update Released | The backend broker cluster is undergoing maintenance, which might cause a brief network disruption and temporary errors in client logs. You can ignore these errors. If you did not receive a maintenance notification, submit a ticket. |
FAQ
Q: When the Broker processes heartbeat data (HeartbeatData), does it authenticate each consumer group independently, or does it authenticate all groups atomically?
A: The documentation does not commit to the specific atomicity or independence of the Broker's internal heartbeat processing logic. Use actual business behavior as the authoritative reference. If you encounter heartbeat-related exceptions, compare your client implementation against the standard sample code provided in the official documentation. This helps rule out issues caused by non-standard client implementations or differences in integration method.