A message trace shows the complete trace of a message from its being produced and sent by a producer to the Message Queue for Apache RocketMQ broker to its being consumed by a consumer, and consists of the time, status, and other information on each node. The trace provides robust data support for troubleshooting in production environments. This topic describes the scenarios, query procedures, and parameter description of query results for message tracing.

Message trace data

In Message Queue for Apache RocketMQ, a complete trace of a message involves the producer, broker, and consumer. Each role adds relevant information to the trace when it processes the message. The information is aggregated to reflect the current status of a message.

Trace data

Scenarios

You can use the message tracing tool for troubleshooting when messages are not sent and received as expected in the production environment. You can query the trace of a message by message ID, message key, or topic to check the sending and receiving status of the message.

Scenarios

Example

Assume that you are looking for the reason why you have not received a message as expected based on the log information. Complete the following steps to troubleshoot the issue with message tracing:

  1. Collect information about the message that is suspected to be abnormal, including the message ID, message key, topic, and approximate sending time.
  2. Log on to the Message Queue for Apache RocketMQ console, and create a query task based on the available information to query the message trace.
  3. Check the results and perform analysis to determine the cause.
    • If Not consumed appears in the trace, you can go to the Group Management page to check the consumer status and check whether the message has not been consumed due to accumulation. For more information, see View the consumer status.
    • If the message has been consumed, locate the corresponding client and the message delivery time in the consumption information. Then, log on to the client to view the log.

Instructions

No additional access costs will be created when you use the message tracing feature. After any message is sent normally, you can query the sending trace of the message by the message property in the Message Queue for Apache RocketMQ console. Note the following for the message trace.

Message type Query description
Normal messages If the message has not been consumed yet, Not consumed appears. Once it is consumed, the delivery and consumption information appears.
Ordered messages If the message has not been consumed yet, Not consumed appears. Once it is consumed, the delivery and consumption information appears.
Scheduled messages and delayed messages If the current system time does not reach the specified message consumption time, the trace can be queried but the message cannot.
Transactional messages Before the transaction is committed, the trace can be queried but the message cannot.

Query a message trace

Messages are stored in Message Queue for Apache RocketMQ for three days by default, and we recommend that you do not modify the value. That is, only the messages that are generated within three days before the current query time can be queried. For example, if the current time is 15:09:48 on June 10, 2019, traces of the messages that were generated as early as 15:09:48 on June 7, 2019 can be queried.

Prerequisites

  • Ensure that your SDK version supports message tracing. The supported SDK versions are as follows:
    • In TCP mode, traces of messages in the specified time range can be queried by the message ID, message key, or topic.
    • In HTTP mode, you can only query message traces by message ID.

      Message tracing is supported by Node.js SDK V1.02 or later and SDKs of V1.0.1 or later in other languages. For more information, see Message Queue for Apache RocketMQHTTP SDK Repositories.

  • The message has been sent by the producer.
Procedure
  1. Log on to the Message Queue for Apache RocketMQconsole. In the top navigation bar, select the region of the instance where the message you want to query is located, for example, China (Hangzhou).
  2. On the Instances page, find the target instance and click Details in the Actions column.
  3. In the left-side navigation pane, choose Message Tracing > Create Query Task.
  4. In the Create Query Task dialog box, select query criteria, enter information as prompted, and then click OK.
    Notice Set the time range as accurately as possible to narrow the query scope and speed up the query.
    Message tracing supports three query methods that are detailed as follows:
    • By Message ID: This method is recommended for its exact matching and fast speed.By message ID
    • By Message Key: This method is fuzzy matching. A maximum of 1,000 traces can be displayed for a query by message key. This query method is suitable for scenarios where the message ID is not recorded but a business-distinctive message key is set.By message key
    • By Topic: This method is a range query. It is suitable for scenarios where no message ID or message key is recorded and the number of messages is small. This query method is not recommended because there are many messages within the time range and it is not business-distinctive.By topic

    After creating a query task, you can find the query task on the Message Tracing page. If Task Status is Querying, the message trace cannot be viewed.

  5. In the upper-right corner of the message tracing task list, click refresh, until the status switches to Query completed.. Click Add.

    The displayed brief message trace information includes the properties, sending status, and receiving status of the message.

    Sending and receiving status
  6. In the section that appears, click View Trace to view the complete trace.

    The following is an example of querying the trace of a normal message in TCP mode.

    Message tracing

    When you query by message key or message topic, if multiple traces are displayed, you can page up and down to view the traces.

Query results

The following table describes the terms related to message tracing:

Table 1. Message tracing terms
Role Field Description
Producer Sending Time The timestamp when the message was sent from the producer.
Sending Duration The time (in ms) taken by the producer to send a message by calling the Send method.
Topic Region The region where the message is stored.
Consumer Duration The time that the consumer takes to execute the consumeMessage method after the message is pushed to the client.
Delivery Time The timestamp when the consumer executes the consumeMessage method to start message consumption.

On the Message Tracing page, click the Add icon of a task, Sending Status and Consumption Statusof the message appear. In addition, the message sending status and consumption status appear in the Producer and Consumer sections in the message trace. The statuses that may be displayed are as follows:

Table 2. Status information
Category Status Description
Sending Status Sent. The message is sent and stored on the broker.
Send failed. The message failed to be sent and is not stored on the broker. In this case, you need to try again.
The message is being scheduled. The message is a scheduled or delayed message and has not reached the delivery time.
Transaction not committed The message is a transactional message and has not been committed.
Transaction rolled back. The message is a transactional message and has been rolled back.
Consumption Status All succeeded. The message still failed to be consumed after all delivery retries.
Partially successful. The message failed to be consumed in certain deliveries, or the consumption of the message failed but the retry was successful.
All failed. The message still failed to be consumed after all delivery retries.
Not consumed The message has not been delivered to any consumer.
No consumption result is returned. The message consumption method has not returned any results yet or has been interrupted, so the consumption status is not returned to the broker.
The message is consumed. The message has been consumed.
Failed to consume the message. The message consumption method returned a failure result, or the consumption method threw an exception.

More information

  • To delete a query task, find the target task on the Message Tracing page, click the Delete icon in the Actions column, and delete the query task as prompted.
  • If you have any question about the result of a message trace query, see Why is tracing data not found? in FAQ.