This topic introduces the basic principles, scenarios, and use cases of RocketMQ message tracing.
Note: RocketMQ supports message tracing over TCP currently.
Definition: Message tracing shows the complete links of a message from its issuance by the producer to its consumption by a consumer, and consists of the time, location, and other information on every node.
Principles:In RocketMQ, a complete link of a message consists of the producer, service provider, and consumer. Each role adds relevant information to the trace when it processes the message. The information is aggregated to reflect the current status of a message, providing robust data support for troubleshooting in production environments.
Message trace data
|Producer information||Consumer information||Service provider information|
|Production client information||Consumption client information||The message topic.|
|Sending time||Delivery time, number of delivery retries||Message storage region|
|Sent or not||Consumed or not||Message key|
|Sending duration||Consumption duration||Message tag|
Message trace query rules:
|Message type||Time that can be queried||Query description|
|Normal message||After the message is sent||Sending trace is generated once the message is sent. If the message is not consumed yet, Not Consumed appears. Once it is consumed, the delivery and consumption information appears.|
|Ordered message||After the message is sent||Sending trace is generated once the message is sent. If the message is not consumed yet, Not Consumed appears. Once it is consumed, the delivery and consumption information appears.|
|Scheduled and delayed messages||After the current system time reaches the specified message consumption time||If the current system time does not reach the specified message consumption time, the trace can be queried while the message cannot.|
|Transactional message||After the message is sent||Sending trace is generated once the message is sent. Before the transaction is committed, the trace can be queried while the message cannot.|
You can use the message tracing tool for troubleshooting when messages are not sent and received as expected in the production environment. Query the trace of a message by its properties (message ID, message key, or topic) to check the sending and receiving status of the message, helping diagnose the problem.
Assume that you find out that you have not received a message as expected based on the log information. Perform the following steps to troubleshoot the issue with message tracing:
Collect the information about the message that you suspect went wrong, including the message ID, message key, topic, and approximate sending time.
Log on to the RocketMQ console, and create a query task with the collected information to query the message trace.
Check the results and analyze for the cause. If the trace shows that the message has not been consumed, you can go to the Consumers page to check whether there are any messages accumulated.
If the message has been consumed, locate the corresponding client and the message delivery time in the consumption information, and log on to the client to view the log.
The use of message tracing does not bring additional access costs to the service provider, but you must ensure that the client SDK version supports this feature. After a message is sent or received normally, you can query the message by the message attribute in the RocketMQ console.
The message tracing function is currently supported on Java clients (1.2.7 and later), C++, and .NET clients. For more client information, check the upgrade prompt in the console. Upgrade your client SDK to the latest version if necessary.
In the left-side navigation pane of the RocketMQ console, choose Message Tracing. On the page that is displayed, click Create Query Task in the upper-right corner.
The message tracing function supports three query types. Select a query type and specify the query conditions. The three query types are described as follows:
- By message ID: You must specify the unique message ID, topic name, and approximate sending time (time range) of a message.
- By message key: You must specify the message key, topic, and approximate sending time (time range) of a message. This query method is suitable for scenarios where the message ID is not recorded but the message key is.
- Query by topic: You must specify the topic and approximate sending time (time range) for batch query. This query method is suitable for scenarios where neither the message ID nor message key is recorded and the volume of messages is relatively small.
- Set the time range as accurately as possible to narrow the query scope and speed up the query.
- Query by message ID is recommended for its exact matching and fast speed.
- Query by message key is a fuzzy query, and is suitable for scenarios where the message ID is not recorded but a business-distinctive message key is set. A maximum of 1,000 traces can be displayed for a query by message key.
- Query by topic is a range query. It is not recommended because it is difficult to locate the needed message among a great number of messages within a time range with limited distinction.
After you created a query task, the RocketMQ backend executes the query task asynchronously, and sends the task status to the Query Task page. When the query task is completed, Query Complete appears in the Task Status column. Otherwise, Querying appears.
Depending on the task status, you can choose to view the trace or delete the query task.
After the query is complete, click Details to view the message trace. If no results appear, click the link in the dialog box to check the cause.
If the trace is queried, the brief trace information appears, including the message attributes and receiving status.
Click Details. On the page that is displayed, click View Trace to check the complete trace link.
The message trace link consists of three parts:
- Producer information
- Topic information
- Consumer information
In query by message key and query by topic, if multiple traces are displayed, you can page up and down to view the traces.
The following table describes relevant terms of message tracing:
|Producer||Sending Time||The timestamp when the message was sent from the sender.|
|Producer||Sending Duration||The time (in ms) taken by the sender to send a message by calling the send method.|
|Topic||Region||The region where the message is stored.|
|Consumer||Duration||The time that the receiver takes to execute the consumeMessage method after the message is pushed to the client.|
|Consumer||Delivery Time||The timestamp when the receiver executes the consumeMessage method to start message consumption.|
In the message tracing task list, click Details of a task. Sending Status and Consumption Status of the message appear. In addition, the message sending status and consumption status appear in the Producer and Consumer information areas in the message trace link. The statuses that may be displayed are as follows:
|Sending status||Sent||The message is sent and stored on the broker.|
|Sending status||Sending Failed||The message failed to be sent and is not stored on the broker. You need to try again.|
|Sending status||Message Standing By||The message is a scheduled or delayed message and it has not reached the delivery time.|
|Sending status||Transaction Uncommitted||The message is a transactional message and has not been submitted.|
|Sending status||Message Rolled Back||The message is a transactional message and has been rolled back.|
|Consumption status||All Succeeded||All deliveries of the message have been consumed.|
|Consumption status||Partially Succeeded||Certain deliveries of the message failed, or the consumption fails but the retry is successful.|
|Consumption status||All Failed||All delivered messages failed to be consumed.|
|Consumption status||Not Consumed||The message has not been delivered to any consumer.|
|Consumption status||Consumption Result Unreturned||The message consuming method has not returned any results yet, or has been interrupted, and therefore, the consumption status is not returned to the broker.|
|Consumption status||Consumed||The message has been consumed.|
|Consumption status||Consumption Failed||The message consuming method returned a failure result, or the consuming method threw an exception.|
If you have any questions about the result of a trace query, see Message tracing in the FAQ.