A message trace is the complete trace of a message that is sent from a producer to a Message Queue for Apache RocketMQ broker and then consumed by a consumer. The message trace includes time and status information about each step in the preceding process. The message trace can be used to help troubleshoot issues in production environments. This topic describes the scenarios for which message tracing is suitable. This topic also describes how to query a message trace and the parameters that are returned in query results.

Message trace data

In Message Queue for Apache RocketMQ, the complete trace of a message involves three roles: producer, broker, and consumer. Each role adds relevant information to the trace when the role processes the message. The information is aggregated to indicate the status of the message.

Trace data

Usage notes

After a message is sent, you can query the trace of the message based on the ID or key of the message in the Message Queue for Apache RocketMQ console. This feature takes effect on all types of messages. Before you use the message tracing feature, take note of the following points.

Query results

The trace data of a message may be incomplete because message tracing uses asynchronous links. The system may return no results for a query even if the query conditions that you specified are valid.

Time range for message trace queries

By default, messages are stored in Message Queue for Apache RocketMQ for three days. You can query the traces only for messages that are produced within the three days before the current point in time. For example, if the current point in time is 15:09:48 on June 10, 2021, you can query the traces of messages that were produced in a topic from 15:09:48 on June 7, 2021 to 15:09:48 on June 10, 2021.

Impacts of the consumption status on query results

Message type Description
Normal messages If a message is not consumed, Not Consumed is displayed. After the message is consumed, the sending information and consumption information are displayed.
Ordered message If a message is not consumed, Not Consumed is displayed. After the message is consumed, the sending information and consumption information are displayed.
Scheduled messages and delayed messages If the current system time is earlier than the specified point in time for consumption, the trace of the message can be queried but the message cannot be queried.
Transactional messages Before the local transaction for a message is committed, the trace of the message can be queried but the message cannot be queried.

Versions of SDKs that support message tracing

Protocol Programming language Version Description
TCP Java V1.2.7.Final and later Release notes for the SDK for Java
C/C++ V1.1.2 and later Release notes for the SDK for C and C++
.NET V1.1.2 and later Release notes for the SDK for .NET
HTTP Node.js V1.0.2 and later Usage notes for HTTP client SDKs
Others V1.0.1 and later

Prerequisites for using the enhanced message tracing feature

Compared with the basic message tracing feature, the enhanced message tracing feature allows you to view more trace data in the console. You can query the trace data that is generated during message consumption and the trace data of scheduled messages, delayed messages, and transactional messages. For more information about the parameters that are returned in trace query results, see Parameters. To use the enhanced message tracing feature, make sure that your instance and client SDK meet the following requirements:

  • TCP client SDK for Java:
    • The TCP client SDK for Java is upgraded to V2.X.X.Final.
    • The instances reside in one of the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Shenzhen), China (Chengdu), Germany (Frankfurt), and Indonesia (Jakarta).
  • TCP client SDK for C++:
    • The TCP client SDK for C++ is upgraded to V3.X.X.
    • The instances can reside in all regions.

Scenarios

You can use the message tracing feature for troubleshooting if a message is not sent or received as expected in your production environment. You can query the message trace by message ID, message key, or topic within a specified time range to check whether the message is sent and received as expected.

Scenarios for which message tracing is suitable

Example

In this example, a message is not received as expected, and you identify the issue based on the business log. To use the message tracing feature to troubleshoot this issue, perform the following steps:

  1. Collect the information about the message that is suspected to be abnormal. The information includes the message ID, message key, topic, and approximate point in time when the message was sent.
  2. Log on to the Message Queue for Apache RocketMQ console, and create a query task to query the message trace based on the collected information.
  3. Check the query results and analyze the cause.
    • If Not Consumed is displayed in the trace, you can go to the Groups page to check the consumer status. Then, you can determine whether the message is not consumed due to message accumulation. For more information, see View the status of consumers.
    • If the message is consumed, check the consumption information to find the corresponding consumer client and the time when the message was consumed. Then, log on to the consumer client to view the relevant log information.

Prerequisites

Procedure

  1. Log on to the Message Queue for Apache RocketMQ console, and in the left-side navigation pane, click Instances.
  2. In the top navigation bar, select a region such as China (Hangzhou), and in the list of instances, click instance name.
  3. In the left-side navigation pane, click Message Trace. In the upper-left corner of the Instance Details page, click Create Query Task.
  4. In the Create Message Trace Query Task panel, configure the query conditions based on your business requirements, and then click OK.
    Important Set the time range as accurately as possible to narrow down the query scope and speed up the query.
    Query method
    Message tracing supports the following three query methods:
    • Query by Message ID: This method uses an exact match and returns query results in a short period of time. We recommend that you use this method.
    • Query by Message Key: This method uses a fuzzy match. If you use this method, a maximum of 1,000 traces can be displayed for a query. This method applies only to scenarios in which the message ID is not recorded for a message but a message key that uniquely identifies the message is specified.
    • Query by Topic: This method uses a range query. This query method is suitable for scenarios in which no message ID or message key is recorded and the message volume is small. We recommend that you do not use this query method. In most cases, a large number of messages exist in a topic within a specified time range. If you use this method, you cannot identify a specific message among the messages that you queried.
    After you create a query task, you can view the query task on the Message Trace page. If Querying is displayed in the Status column, the message trace cannot be viewed.
  5. View the status of the query task in the task list. If Querying is displayed in the Status column, the message trace cannot be viewed. In this case, click the refresh icon in the upper-right corner until the status changes to Query Completed.
  6. In the task list, find the query task, and click Query Results in the Actions column.
    You can view the basic information about the queried message traces. The information includes the properties, sending status, and consumption status of each message. Query results
  7. On the Query Results page, find the message trace that you want to view and click Message Trace in the Actions column to view the detailed trace information.
  8. In the Message Trace panel, view the consumers who subscribe to the specified message in the Consumer section. Click the group ID that corresponds to the consumer that you want to query. Then, you can view the detailed information about the consumer.

Parameters

Note The following parameters can be viewed in the console only if your instance and client SDK version meet the prerequisites of using the enhanced message tracing feature. For more information, see the "Prerequisites for using the enhanced message tracing feature" section.
  • AccessKey in the Producer section
  • Arrive at Server At, Scheduled to Be Delivered At, Delivered At, and Committed/Rolled back At in the MQ Server section
  • Arrive at Consumer At and Wait Duration before Processing in the Consumer section
Table 1. Parameters
Section Parameter Description
Basic Information Message ID The globally unique identifier of the message. This identifier is automatically generated by the Message Queue for Apache RocketMQ system.
Topic The topic to which the message belongs.
Message Key The business identifier of the message. The message key is set by the message producer and uniquely identifies a business logic.
Message Tag The tag that is used to classify the message in a topic.
Producer AccessKey The AccessKey ID of your Alibaba Cloud account or RAM user. AccessKey IDs are used to verify user identities. When you use SDKs or API operations to call Message Queue for Apache RocketMQ resources, the AccessKey ID is required for identity verification.
Client IP Address The client IP address of the message producer.
Message Produced At The timestamp when the message was sent from the producer.
Sending RT The amount of time that is consumed to send the message. Unit: milliseconds.
Sending Status The status of the message sending task. For more information, see Sending Status in the Message status description section.
MQ Server Message Type
Note If the message is a scheduled message or a delayed message, Scheduled Message is displayed for Message Type.
Arrive at Server At The time when the message arrived at the Message Queue for Apache RocketMQ broker.
Scheduled to Be Delivered At The scheduled point in time for delivering a scheduled message.
Delivered At The time when the broker can deliver the message to the consumer.
Committed/Rolled back At The time when the transactional message is committed or rolled back.
Consumer AccessKey The AccessKey ID of your Alibaba Cloud account or RAM user. AccessKey IDs are used to verify user identities. When you use SDKs or API operations to call Message Queue for Apache RocketMQ resources, the AccessKey ID is required for identity verification.
Arrive at Consumer At The time when the message arrived at the consumer client.
Wait Duration before Processing The duration between the time when the message arrived at the consumer client and the time when the message started to be consumed.
Delivery Result A message may need to be delivered multiple times before it is consumed. This parameter indicates the result of one specific delivery. For more information, see Consumption Status in the Message status description section.
Client IP Address The IP address of the consumer client.
Message Processing Started At The timestamp when the message started to be consumed by the client.
Message Processing Complete At The timestamp when the consumer completed message consumption.
Message Processing Duration The amount of time that is used by the consumer to consume the message.

On the Message Trace page, find your query task and click Query Results in the Actions column. On the page that appears, you can view the sending status and consumption status of the queried messages in the Sending Status and Consumption Status columns. You can also view the sending status and consumption status of each message in the Producer and Consumer sections on the message trace details page. The following table describes the possible sending states and consumption states.

Table 2. Message status description
Type State
Sending Status Sent
Send failed.
Scheduling the message.
Transactional Message Not Committed
Transactional Message Rolled Back
Consumption Status All succeeded.
Partially Successful
All failed.
Not Consumed
No consumption result is returned.
The message is consumed.
Failed to consume the message.

What to do next

  • Delete a single query task

    To delete a single query task, find the task on the Message Trace page, click More in the Actions column, and then select Delete from the drop-down list.

  • Delete multiple query tasks at a time

    To delete multiple query tasks at a time, select the query tasks that you want to delete on the Message Trace page, and then click Batch Delete Tasks above the task list.

References

If you have questions about the query results, see Why is tracing data not found?