A message trace is the complete trace of a message that is sent from a producer to the Message Queue for Apache RocketMQ server and then consumed by a consumer. The message trace includes the time, status, and other information of each node in the preceding process. The message trace provides robust data support for troubleshooting in production environments. This topic describes the scenarios, query procedure, and parameters of query results for message tracing.

Message trace data

In Message Queue for Apache RocketMQ, the complete trace of a message involves three roles: producer, server, and consumer. Each role adds relevant information to the trace when the role processes the message. The information is aggregated to indicate the status of the message.

Trace data

Scenarios

You can use the message tracing feature for troubleshooting if a message is not sent or received as expected in your production environment. You can query the message trace by message ID, message key, or topic to check whether the message is sent and received as expected within the specified time range.

Scenarios

Example

Assume that you have not received a message as expected. You can identify the cause based on the related log, and perform the following steps to troubleshoot the issue by querying the message tracing feature:

  1. Collect the information of the message that is suspected to be abnormal, including the message ID, message key, topic, and approximate sending time.
  2. Log on to the Message Queue for Apache RocketMQ console, and create a query task to query the message trace based on the available information.
  3. Check the query results and analyze the cause.
    • If Not Consumed is displayed in the trace, you can go to the Groups page to check the consumer status. Then, you can determine whether message accumulation is the reason why the message has not been consumed. For more information, see View the status of consumers.
    • If the message has been consumed, locate the consumer client and the message delivery time in the consumption information and log on to the client to view the log.

Usage notes

No extra fees are incurred when you use the message tracing feature. After a message is sent, you can query the trace of the message based on the ID or key of the message in the Message Queue for Apache RocketMQ console. You must take note of the following items.

Message type Query description
Normal messages If the message is not consumed, Not Consumed is displayed. After the message is consumed, the delivery and consumption information is displayed.
Ordered messages If the message is not consumed, Not Consumed is displayed. After the message is consumed, the delivery and consumption information is displayed.
Scheduled messages and delayed messages If the current system time does not reach the specified message consumption time, the trace can be queried but the message cannot be queried.
Transactional messages Before the transaction is committed, the trace can be queried but the message cannot be queried.

Query a message trace

Messages are stored in Message Queue for Apache RocketMQ for three days by default. We recommend that you do not modify the default value. You can query only the traces that are generated within three days before the current query time. For example, if the current time is 15:09:48 on June 10, 2019, traces that were generated from 15:09:48 on June 7, 2019 can be queried.

Prerequisites

  • Make sure that the version of your SDK supports the message tracing feature. You can use the following versions of SDKs:
    • In TCP mode, traces can be queried by message ID, message key, or topic within the specified time range.
      • SDK for Java: version 1.2.7 and later. For more information, see Release notes.
      • SDK for C/C++: version 1.1.2 and later. For more information, see Release notes.
      • SDK for .NET: version 1.1.2 and later. For more information, see Release notes.
    • In HTTP mode, you can query traces only by message ID.

      Message tracing is supported by SDK for Node.js of version 1.0.2 or later and SDKs for other languages of version 1.0.1 or later. For more information, see Message Queue for Apache RocketMQHTTP SDK Repositories.

  • The message has been sent from the producer.
Procedure
  1. Log on to the Message Queue for Apache RocketMQ console.
  2. In the left-side navigation pane, click Instances.
  3. In the top navigation bar, select a region. In this example, select China (Hangzhou).
  4. On the Instances page, find the instance, click More in the Actions column, and then select Message Trace from the drop-down list.
  5. On the Message Tracing page, click Create Query Task in the upper-left corner.
  6. In the Create Message Trace Query Task dialog box, select query criteria, enter information as prompted, and then click OK.
    Notice Set the time range as accurately as possible to narrow down the query scope and speed up the query.
    Message tracing supports the following three query methods:
    • Query by Message ID: Recommended. This method uses an exact match and allows you to query message traces fast.message_trace_query_by_messageid
    • Query by Message Key: This method uses a fuzzy match. A maximum of 1,000 traces can be displayed for a query by message key. This query method is suitable for scenarios where the message ID is not recorded but a business-distinctive message key is specified.message_trace_query_by_messagekey
    • Query by Topic: This method uses a range query. This query method is suitable for scenarios where no message ID or message key is recorded and the message volume is small. This query method is not recommended because a large volume of messages exist in a topic within the specified time range and the messages are not business-distinctive.message_trace_query_by_topic

    After you create a query task, you can find it on the Message Trace page. If Status is Querying, the message trace cannot be viewed.

  7. In the upper-right corner of the message tracing task list, click refresh until the status changes to Query is complete.. Click Query Results in the Actions column.

    The basic information of the message trace is displayed. The information includes the attributes and statuses of the message.

    message_trace_state
  8. Click Message Trace to view the complete trace.

    The following figure shows an example of the trace for a normal message that is queried by message ID in TCP mode.

    message_trace_detail

    When you query traces by message key or topic, multiple traces may be displayed. You can page up and down to view and compare the traces.

Query results

The following table describes the terms related to message tracing.

Table 1. Message tracing terms
Role Log field Description
Producer Sending Time The timestamp when the message was sent from the producer.
Sending Time Consumed The time that was taken by the producer to send a message by calling the Send method. Unit: milliseconds.
Consumer General Sending Count The number of deliveries performed by a producer before a message is consumed by a consumer.

On the Message Tracing page, find the task and click Query Results in the Actions column. Sending Status and Consumption Status of the message are displayed. The sending status and consumption status of the message are also displayed in the Producer and Consumer sections of the message trace. The following table describes the statuses.

Table 2. Statuses
Status Log field Description
Sending Status Sent. The message is sent and stored in the server.
Send failed. The message fails to be sent and is not stored in the server. In this case, you must try again.
Scheduling the message. The message is a scheduled or delayed message and it is not the time to deliver the message.
Transactional Message Not Committed The message is a transactional message and has not been committed.
Transactional Message Rolled Back The message is a transactional message and has been rolled back.
Consumption Status All succeeded. All deliveries of the message have been consumed.
Partially Successful The message fails to be consumed in specific deliveries, or the consumption fails but the retry succeeds.
All failed. The message still fails to be consumed after all delivery retries.
Not Consumed The message is not delivered to consumers.
No consumption result is returned. No results are returned for the message consumption method or the method is interrupted. Therefore, the consumption status is not returned to the server.
The message is consumed. The message is consumed.
Failed to consume the message. A failure result is returned for the message consumption method, or the method threw an exception.

References

  • To delete a query task, find the task on the Message Tracing page, click the More in the Actions column, and then select Delete from the drop-down list.
  • If you have any question about the query result, see Why is tracing data not found?