A message trace is the complete trace of a message that is sent from a producer to the Message Queue for Apache RocketMQ server and then consumed by a consumer. The message trace includes the time, status, and other information of each node that is involved in the preceding process. The trace provides robust data support for troubleshooting in production environments. This topic describes the scenarios, query procedure, and parameters of query results for message tracing.

Message trace data

In Message Queue for Apache RocketMQ, the complete trace of a message involves three roles: producer, server, and consumer. Each role adds relevant information to the trace when the role processes the message. The information is aggregated to indicate the status of the message.

Trace data

Usage notes

After a message is sent, you can query the trace of the message based on the attributes of the message in the Message Queue for Apache RocketMQ console. You must take note of the following items.

  • Time range of message tracing

    Messages are stored in Message Queue for Apache RocketMQ for three days by default. We recommend that you do not modify the default value. You can query only the traces that are generated within three days before the current query time. For example, if the current time is 15:09:48 on June 10, 2021, traces that have been generated starting from 15:09:48 on June 7, 2021 until the current time can be queried.

  • Impacts of consumption status on query results
    Message type Query description
    Normal messages If the message has not been consumed yet, Not Consumed is displayed. After the message is consumed, the delivery and consumption information is displayed.
    Ordered messages If the message has not been consumed yet, Not Consumed is displayed. After the message is consumed, the delivery and consumption information is displayed.
    Scheduled messages and delayed messages If the current system time does not reach the specified message consumption time, the trace can be queried but the message cannot.
    Transactional message Before the transaction is committed, the trace can be queried. However, the message cannot be queried.
  • Versions of SDKs supporting message tracing
    Protocol Programming language Version Release notes
    TCP Java V1.2.7 or later Release notes for the SDK for Java
    C/C++ V1.1.2 or later Release notes for the SDKs for C and C++
    .NET V1.1.2 or later Release notes for the SDK for .NET
    HTTP Node.js V1.0.2 or later Release notes for the HTTP client SDK
    Others V1.0.1 or later

Scenarios

You can use the message tracing feature for troubleshooting if a message is not sent or received as expected in your production environment. You can query the message trace by message ID, message key, or topic to check whether the message is sent and received as expected within the specified time range.

Message tracing scenarios

Examples

For example, you have not received a message as expected. You can identify the cause by viewing the related logs, and perform the following steps to troubleshoot the issue with message tracing:

  1. Record the information about the message that is suspected to be abnormal, including the message ID, message key, topic, and approximate sending time.
  2. Log on to the Message Queue for Apache RocketMQ console, and create a task to query the message trace based on the information that you have recorded.
  3. Check the query results and analyze the cause.
    • If Not Consumed is displayed in the trace, you can go to the Groups page to check the consumer status. Then, you can determine whether message accumulation is the reason why the message has not been consumed. For more information, see View the status of consumers.
    • If the message has been consumed, locate the consumer client and the message delivery time in the consumption information, and log on to the client to view the log.

Prerequisites

Procedure

  1. Log on to the Message Queue for Apache RocketMQ console.
  2. In the left-side navigation pane, click Instances.
  3. In the top navigation bar, select a region, such as China (Hangzhou).
  4. On the Instances page, find the instance, click More in the Actions column, and then select Message Trace from the drop-down list.
  5. On the Message Tracing page, click Create Query Task in the upper-left corner.
  6. In the Create Message Trace Query Task dialog box, select query criteria, enter information as prompted, and then click OK.
    Notice Set the time range as accurately as possible to narrow down the query scope and speed up the query.
    Query methods
    You can query a message trace with one of the following three methods:
    • Query by Message ID: Recommended. This method uses an exact match and allows you to query message traces in a short amount of time.
    • Query by Message Key: This method uses a fuzzy match. A maximum of 1,000 traces can be displayed for a query by message key. This query method is suitable for scenarios where the message ID is not recorded but a distinctive message key is specified.
    • Query by Topic: This method uses a range query. This query method is suitable for scenarios where no message ID or message key is recorded and the message volume is small. This query method is not recommended because a large volume of messages exist in a topic within the specified time range and the messages are not distinctive.
    After you create a query task, you can find it on the Message Trace page. If Status is Querying, the message trace cannot be viewed.
  7. View the status of the query task in the task list. If the Status is Querying, the message trace cannot be viewed. You need to click the refresh button in the upper right corner of the task list until the status changes to Query Completed.
  8. In the message tracing task list, select the specified query task and click Query Results in the Actions column.
    The basic information of the message trace is displayed. The information includes the attributes and sending and consumption statuses of the message. Query results
  9. On the Query Results page, select the message trace that you want to query and click Message Trace in the Actions column to view the detailed information of the trace.
    The following figure shows an example of the trace for a normal message that is queried by message ID in TCP mode. Tracing results
  10. In the displayed Message Trace page, the Consumer section lists consumers who subscribe to the specified message. You can view the details of a consumer by clicking the group ID of the consumer.

Query results

Table 1. Parameters
Parameter attribution Parameter name Description
Basic Information Message ID The globally unique identifier of the message, which is automatically generated by Message Queue for Apache RocketMQ.
Topic The name of the topic to which the message belongs.
Message Key The service identifier of the message, which is set by the message producer and uniquely identifies a service logic.
Message Tag The message tag that distinguishes one type of messages under a topic.
Producer IP The IP address of the message producer client.
Message production time The timestamp when the message is sent from the producer.
Sending RT The time taken to send the message, in milliseconds.
Sending Status See the Sending Status in descriptions about message statuses.
Consumer Current delivery result A message may have to be delivered for multiple times to a specified consumer before the message is successfully consumed. This parameter indicates the result of one delivery. For more information, see Consumption Status in descriptions about message statuses.
IP The IP address of the message consumer client.
Message Processing Started At The timestamp when the message starts to be consumed.
Message Processing Complete At The timestamp when the message is completely consumed.
Message Processing Duration The duration of message consumption.

In the message tracing task list, click Query Results in the Actions column of a task. Sending Status and Consumption Status of the message are displayed. The sending status and consumption status of the message are also displayed in the Producer and Consumer sections of the message trace. The following table describes the statuses.

Table 2. Message statuses
Status Parameter
Sending Status Sent
Send failed.
Scheduling the message.
Transactional Message Not Committed
Transactional Message Rolled Back
Consumption Status All succeeded.
Partially Successful
All failed.
Not Consumed
No consumption result is returned.
The message is consumed.
Failed to consume the message.

More operations

  • Delete a task

    Find the task to be deleted in the message tracing task list. Click More in the Actions column, and then select Delete from the drop-down list.

  • Delete multiple tasks

    Select all the tasks to be deleted in the message tracing task list, and then click Batch Delete Tasks above the list.

References

If you have any question about the query result, see Why is tracing data not found? in FAQs.