Dashboard metrics overview and query methods - ApsaraMQ for RocketMQ

By leveraging the metric storage and visualization capabilities of Alibaba Cloud's Managed Service for Prometheus and Managed Service for Grafana, Message Queue for Apache RocketMQ 5.0 provides a dashboard feature. This feature helps you centrally collect, observe, and visualize multi-dimensional metrics to quickly assess the operational status of your business. This topic describes the use cases, background, metric details, billing, and query methods for the dashboard.

Use cases

Troubleshoot online message consumption exceptions, receive timely alerts, and quickly locate issues when messages are not processed promptly.
Investigate message delivery through the pipeline when you encounter issues with online orders.
Analyze message traffic trends, distribution patterns, or message volume to support business trend analysis and planning.
View and analyze the upstream and downstream dependency topology of your applications to upgrade or optimize your system architecture.

Background

In the message sending and receiving process of Alibaba Cloud Message Queue for Apache RocketMQ, the queue backlog, buffering status, and the time consumed at each stage of message processing directly reflect the performance of your services and the operational status of the server. Therefore, the key metrics for Alibaba Cloud Message Queue for Apache RocketMQ are primarily related to the following business scenarios.

Consumer lag scenarios

The following figure shows the status of messages in a specific queue of a topic.

队列消息状态

The figure above shows the status of messages in a queue for a specific topic. Message Queue for Apache RocketMQ collects statistics on the number of messages and the time consumed at different processing stages. These metrics directly reflect the processing rate and accumulation of messages in the queue. By observing these metrics, you can make a preliminary judgment on whether your service consumption is abnormal. The specific definitions and calculation formulas for these metrics are as follows:

Category	Metric	Description	Formula
Message count metrics	inflight messages	Messages that are being processed by a consumer client but have not yet been acknowledged.	Offset of the latest pulled message - Offset of the latest acknowledged message
	ready messages	Messages that are ready on the Message Queue for Apache RocketMQ 5.0 broker and are visible and available for consumption. If tag filtering is configured for the consumer group, ready messages include only the messages that match the filter conditions.	Maximum message offset - Offset of the latest pulled message
	consumer lag	The total number of messages that have not been fully processed.	Number of inflight messages + Number of ready messages
Message latency metrics	Ready time	For a normal message or an ordered message: The time when the message is stored on the broker. For a scheduled/delayed message: The time when the scheduled or delayed period ends. For a transactional message: The time when the transaction is committed.	N/A
	Ready message queue time	The time difference between the current time and the ready time of the earliest ready message. This duration reflects the timeliness of message pulling by the consumer.	Current time - Ready time of the earliest ready message
	Consumer lag time	The time difference between the current time and the ready time of the earliest unacknowledged message. This duration reflects the timeliness of message processing by the consumer.	Current time - Ready time of the earliest unacknowledged message

When the consumer lag keeps decreasing and approaches 0, and the consumer lag time keeps shortening and approaches 0, the consumption speed has exceeded the production speed, and the backlog is being cleared. When both metrics approach 0, the consumer group has caught up with the production speed, and the backlog has returned to normal.

Push consumer scenarios

A Push consumer encapsulates message fetching, threading, and retry logic. Register a message listener during initialization, and the SDK handles the rest.

pushconsumer

For more information about the principles, see Push consumers.

In a push consumer scenario, the following metrics measure the state of the local buffer queue:

Number of messages in the local buffer queue: The total number of messages in the local buffer queue.
Size of messages in the local buffer queue: The total size of all messages in the local buffer queue.
Message wait time: The duration for which messages are temporarily stored in the local buffer queue.

Metric details

Important

Message TPS, API call counts, and message counts are based on a 4 KB normal message. To calculate the final value, multiply this base by factors corresponding to the actual message size and type. For more information about the calculation rules, see How computing specifications work.

The following table describes the fields related to metrics.

Field	Value
Metric type	Counter: A cumulative metric whose value only increases. For example, the number of produced messages. Gauge: A metric whose value can both increase and decrease. It represents the current value of a monitored object. For example, the TPS of API calls. Histogram: A representation of the distribution of numerical data. For example, the distribution of message sizes.
Label	instance_id: The ID of the Message Queue for Apache RocketMQ 5.0 instance. topic: The topic in Message Queue for Apache RocketMQ 5.0. message_type: The message type. `normal` indicates a normal message, `fifo` indicates an ordered message, `transaction` indicates a transactional message, and `delay` indicates a scheduled/delayed message. fifo_enable: Specifies whether the broker delivers messages for consumption in order. `true` indicates ordered delivery, and `false` indicates concurrent delivery. uid: The ID of your Alibaba Cloud account. client_id: The ID of the Message Queue for Apache RocketMQ 5.0 client. invocation_status: The result of the API call to send a message. `success` indicates a successful call, and `failure` indicates a failed call.

Broker metrics

Metric type	Metric name	Unit	Description	Labels
Gauge	rocketmq_instance_requests_max	count/s	The maximum TPS for sending and receiving messages in the instance per minute. This metric excludes throttled requests. The system collects one sample per second over a 1-minute period. The maximum value among the 60 samples is used.	uid instance_id
Gauge	rocketmq_instance_requests_in_max	count/s	The maximum TPS for sending messages in the instance per minute. This metric excludes throttled requests. The system collects one sample per second over a 1-minute period. The maximum value among the 60 samples is used.	uid instance_id
Gauge	rocketmq_instance_requests_out_max	count/s	The maximum TPS for consuming messages in the instance per minute. This metric excludes throttled requests. The system collects one sample per second over a 1-minute period. The maximum value among the 60 samples is used.	uid instance_id
Gauge	rocketmq_topic_requests_max	count/s	The maximum TPS for sending messages to a topic in the instance per minute. This metric excludes throttled requests. The system collects one sample per second over a 1-minute period. The maximum value among the 60 samples is used.	uid instance_id topic
Gauge	rocketmq_group_requests_max	count/s	The maximum TPS for consuming messages in a consumer group within the instance per minute. This metric excludes throttled requests. The system collects one sample per second over a 1-minute period. The maximum value among the 60 samples is used.	uid instance_id consumer_group
Gauge	rocketmq_instance_requests_in_threshold	count/s	The throttling threshold for sending messages in the instance.	uid instance_id
Gauge	rocketmq_instance_requests_out_threshold	count/s	The throttling threshold for consuming messages in the instance.	uid instance_id
Gauge	rocketmq_throttled_requests_in	count	The number of throttled requests for sending messages.	uid instance_id topic message_type
Gauge	rocketmq_throttled_requests_out	count	The number of throttled requests for consuming messages.	uid instance_id topic fifo_enable consumer_group
Gauge	rocketmq_instance_elastic_requests_max	count/s	The maximum elastic TPS for sending and receiving messages in the instance.	uid instance_id
Counter	rocketmq_requests_in_total	count	The number of API calls related to sending messages.	uid instance_id topic message_type
Counter	rocketmq_requests_out_total	count	The number of API calls related to consuming messages.	uid instance_id topic consumer_group fifo_enable
Counter	rocketmq_messages_in_total	message	The number of messages sent from producers to the broker.	uid instance_id topic message_type
Counter	rocketmq_messages_out_total	message	The number of messages delivered from the broker to consumers. This includes messages that are being processed, have been processed successfully, or failed to be processed.	uid instance_id topic consumer_group fifo_enable
Counter	rocketmq_throughput_in_total	byte	The message throughput from producers to the broker.	uid instance_id topic message_type
Counter	rocketmq_throughput_out_total	byte	The message throughput from the broker to consumers. The message count includes messages that are being processed, have been processed successfully, or failed to be processed.	uid instance_id topic consumer_group fifo_enable
Counter	rocketmq_internet_throughput_out_total	byte	The outbound public network traffic consumed by sending and receiving messages.	uid instance_id topic message_type
Histogram	rocketmq_message_size	byte	The distribution of message sizes, recorded upon successful sending. Distribution buckets are as follows: le_1_kb: ≤1 KB le_4_kb: ≤4 KB le_512_kb: ≤512 KB le_1_mb: ≤1 MB le_2_mb: ≤2 MB le_4_mb: ≤4 MB le_overflow: >4 MB	uid instance_id topic message_type
Gauge	rocketmq_consumer_ready_messages	message	Number of ready messages. The number of messages that are ready on the broker and available for consumption. This metric reflects the volume of messages that consumers have not yet pulled.	uid instance_id topic consumer_group
Gauge	rocketmq_consumer_inflight_messages	message	The number of messages that consumer clients are processing but have not yet acknowledged. The total number of messages that the consumer client is processing but has not yet returned a consumption result for.	uid instance_id topic consumer_group
Gauge	rocketmq_consumer_queueing_latency	ms	Queuing latency of ready messages. The time difference between the current time and the ready time of the earliest ready message. This metric indicates how promptly consumers pull messages.	uid instance_id topic consumer_group
Gauge	rocketmq_consumer_lag_latency	ms	Consumption processing latency. The time difference between the current time and the ready time of the earliest unconsumed message. This metric indicates how promptly consumers process messages.	uid instance_id topic consumer_group
Counter	rocketmq_send_to_dlq_messages	message	The number of messages sent to the dead-letter queue (DLQ) per minute. A message is sent to the DLQ if it fails consumption more times than the configured maximum redelivery attempts. Depending on the dead-letter policy configured for the consumer group, these undelivered messages are saved to a specified topic or discarded.	uid instance_id topic consumer_group
Gauge	rocketmq_storage_size	byte	The amount of storage space used by the instance, including the size of all files.	uid instance_id

Producer metrics

Metric type

Metric name

Unit

Description

Labels

Histogram

rocketmq_send_cost_time

The distribution of time taken for successful calls to the message sending API.

Distribution buckets are as follows:

le_1_ms
le_5_ms
le_10_ms
le_20_ms
le_50_ms
le_200_ms
le_500_ms
le_overflow

uid
instance_id
topic
client_id
invocation_status

Consumer metrics

Metric type	Metric name	Unit	Description	Label
Histogram	rocketmq_process_time	ms	The distribution of time taken by a push consumer to process messages. This includes both successful and failed processing attempts. `rocketmq_process_time = process end time - process start time` The distribution intervals are as follows: le_1_ms le_5_ms le_10_ms le_100_ms le_10000_ms le_60000_ms le_overflow	uid instance_id consumer_group topic client_id invocation_status
Gauge	rocketmq_consumer_cached_messages	message	The number of messages in the local cache queue of a push consumer.	uid instance_id consumer_group topic client_id
Gauge	rocketmq_consumer_cached_bytes	byte	The total size of messages in the local cache queue of a push consumer.	uid instance_id consumer_group topic client_id
Histogram	rocketmq_await_time	ms	The distribution of time that messages wait in the local cache queue of a push consumer. `rocketmq_await_time = process start time - arrival time` The distribution intervals are as follows: le_1_ms le_5_ms le_20_ms le_100_ms le_1000_ms le_5000_ms le_10000_ms le_overflow	uid instance_id consumer_group topic client_id

Billing

Dashboard metrics for ApsaraMQ for RocketMQ are basic metrics in Alibaba Cloud Managed Service for Prometheus. No fees are charged for basic metrics. Therefore, the dashboard feature is also free of charge.

For more information, see Metrics and Pay-as-you-go.

Prerequisites

Activate Managed Service for Prometheus
Create a service-linked role
- Role name: AliyunServiceRoleForOns
- Policy name: AliyunServiceRolePolicyForOns
- Permissions: Allows Message Queue for Apache RocketMQ to use this role to access your services, such as Cloud Monitor and Application Real-Time Monitoring Service (ARMS), to enable monitoring, alerting, and dashboard features.
- For more information, see Service-linked roles.

View dashboards

ApsaraMQ for RocketMQ lets you view dashboards from the following pages:

Dashboard page: View metrics for all topics and groups in an instance.
Instance Details page: Displays a producer overview, billing metrics, and throttling metrics for a specific instance.
Topic Details page: Displays production metrics and producer client metrics for a specific topic.
Group Details page: Displays message backlog metrics and consumer client metrics for a specific group.

Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region, such as China (Hangzhou). On the Instances page, click the name of the instance that you want to manage.
View the dashboard in one of the following ways.
- Instance Details page: On the Instance Details page, click the Dashboard tab.
- Dashboard page: In the navigation pane on the left, click Overview.
- Topic Details page: In the navigation pane on the left, click Topics. In the topic list, click the name of the target topic. Then, on the Topic Details page, click the Dashboard tab.
- Group Details page: In the navigation pane on the left, click Groups. In the group list, click the name of the target group. Then, on the Group Details page, click the Dashboard tab.

Dashboard FAQ

How do I get dashboard metric data?

Use your main account to log on to the ARMS console.
In the navigation pane on the left, click Integration Center.
On the Integration Center page, enter RocketMQ in the search box and click the search icon.
In the search results, select the cloud service to integrate, such as Alibaba Cloud RocketMQ (5.0) Service. For more information, see Step 1: Integrate cloud service monitoring data.
After the integration is successful, in the navigation pane on the left, click Integration Management.
On the Integration Management page, click the Cloud Service Region Environment tab.
In the Cloud Service Region Environment list, click the name of the target environment to go to the cloud service environment details page.
On the Component Management tab, in the Basic Information section, click the cloud service region after Prometheus Instance.
On the Settings tab, you can get different data access methods.

How do I integrate dashboard metric data into a self-managed Grafana instance?

All metric data for ApsaraMQ for RocketMQ is saved to your Managed Service for Prometheus instance. Follow the steps in How do I get dashboard metric data? to integrate the cloud service and get the environment name and HTTP API address. You can then use the API to integrate the dashboard metric data for ApsaraMQ for RocketMQ into your self-managed Grafana instance. For more information, see Use an HTTP API address to connect Grafana or a self-managed application to Prometheus for data access.

What does the max TPS value for an instance mean?

Max TPS: The value is calculated over a 1-minute period, with one sample taken per second. The final result is the maximum value among these 60 samples.

For example:

Assume that an instance produces 60 messages in 1 minute (all are normal messages, and each is 4 KB in size). The production rate of the instance is 60 messages per minute.

If all 60 messages are sent in the first second, the per-second TPS values for the instance during that minute are 60, 0, 0, ..., 0.
Max instance TPS = 60 TPS.
If 40 messages are sent in the first second and 20 messages are sent in the second second, the per-second TPS values for the instance during that minute are 40, 20, 0, 0, ..., 0.
Max instance TPS = 40 TPS.

Why is the production rate much higher than the consumer lag?

The production rate is calculated at the topic level and counts all messages, regardless of tag. The consumer lag is calculated at the consumer group level and is affected by tag filtering: only messages that match the consumer group's filter conditions are counted. If most messages in a topic do not match the consumer group's tag filter conditions, the consumer lag can be much smaller than the production rate.