Integrate OpenTelemetry trace data from RocketMQ for end-to-end tracing analysis - ApsaraMQ for RocketMQ

In a distributed system, a single request can pass through dozens of services before completing. When a message gets stuck, a consumer times out, or latency spikes, pinpointing the root cause across the producer-broker-consumer chain requires visibility into message flow.

ApsaraMQ for RocketMQ generates trace data that follows the OpenTelemetry semantic conventions for messaging. Export this data to Simple Log Service or Application Real-Time Monitoring Service (ARMS) to track message delivery, measure end-to-end latency, and diagnose failures.

Limitations

Before you set up tracing, review these constraints:

Java SDK only: Only clients that use the SDK for Java can report trace data. For version details, see Release notes.
Transactional messages: Only production-related data is traced for transactional messages.
Push consumers only: Only messages consumed by push consumers generate trace data. See Consumer types.

Export trace data

ApsaraMQ for RocketMQ exports OpenTelemetry-compliant trace data to a tracing analysis server. Use an Alibaba Cloud managed service or build your own backend.

Simple Log Service

Simple Log Service provides dashboards for monitoring message sending delay, send success rate, consumption success rate, and end-to-end latency. Drill down into individual requests by message ID or trace ID.

Setup: Import trace data from Java applications to Simple Log Service by using OpenTelemetry SDK for Java
Demo dashboards:
- Tracing metrics dashboard -- View message sending delay, success rates, and end-to-end latency.
- Analysis dashboard -- Investigate specific requests by message ID or trace ID.

For more information about Simple Log Service, see Usage notes.

ARMS

Application Real-Time Monitoring Service (ARMS) provides distributed tracing through its Tracing Analysis component.

Setup: Connect OpenTelemetry to Tracing Analysis

For more information about ARMS Tracing Analysis, see What is Tracing Analysis?

How tracing works

ApsaraMQ for RocketMQ records each stage of a message lifecycle as a span. Each span captures timing data, message metadata, and the operation type.

Span types

Three span types capture the message lifecycle:

Span type	What it captures	Lifecycle
send	A message leaving the producer.	Starts when the producer begins sending and ends when the send succeeds, fails, or throws an exception. If retries occur, each attempt generates a separate span.
receive	A consumer's long polling request.	Matches the lifecycle of the long polling operation. Disabled by default.
process	A push consumer processing a message.	Starts when the `MessageListener` callback begins and ends when it returns.

Enable receive spans

Receive spans are disabled by default. To enable them, add the following JVM parameter:

-Dotel.instrumentation.messaging.experimental.receive-telemetry.enabled=true

Span relationships

The relationship between spans depends on whether receive spans are enabled:

Receive spans disabled (default): The process span is a direct child of the send span.
Receive spans enabled: The process span is a child of the receive span and is linked to the send span.

Span relationships

Messaging attributes

Each span includes attributes defined by the OpenTelemetry semantic conventions for messaging spans.

Standard attributes

messaging.message.id: Unique identifier for the message.
messaging.destination: Topic or queue that the message is delivered to.
messaging.operation: Operation type: send, receive, or acknowledge.

RocketMQ-specific attributes

ApsaraMQ for RocketMQ adds the following attributes. For the full list, see Apache RocketMQ attributes.

Attribute	Type	Description
messaging.rocketmq.client_group	string	Consumer group used for load balancing.
messaging.rocketmq.client_id	string	Unique identifier for the client instance.
messaging.rocketmq.message.delivery_timestamp	int	Scheduled delivery time for scheduled and delayed messages.
messaging.rocketmq.message.group	string	Message group for ordered messages.
messaging.rocketmq.message.type	string	Message type: `normal`, `fifo`, `delay`, or `transaction`. See Normal messages, Ordered messages, Scheduled and delayed messages, and Transactional messages.
messaging.rocketmq.message.tag	string	Tag used to filter messages on the consumer side.
messaging.rocketmq.message.keys	string[]	Business keys for looking up specific messages.