ApsaraMQ for RocketMQ allows you to configure alert rules by using Cloud Monitor to monitor the running status and key business metrics of your instances in real time, receive timely alert notifications for exceptions, and enable early risk warnings for your production environment.
Background information
ApsaraMQ for RocketMQ is a fully managed messaging service. Each instance specification comes with a clear Service Level Agreement (SLA). After you purchase an instance, its performance for metrics such as messaging TPS, message storage, and network throughput is guaranteed to meet its specifications.
You do not need to worry about instance performance. However, in a production environment, you should monitor if your actual business consumption and scale are approaching your instance's specification limits. ApsaraMQ for RocketMQ, in conjunction with CloudMonitor, provides a free, out-of-the-box monitoring and alerting service that addresses the following issues:
Early warning for instance specification watermarks
If your resource usage exceeds the instance's specification limits, ApsaraMQ for RocketMQ enforces throttling. By configuring alerts for specification watermarks in advance, you can detect the risk of exceeding limits early and promptly upgrade instance configurations to avoid business failures caused by throttling.
Early warning for business logic errors
You may encounter errors when sending and receiving messages. Configuring alerts for invocation errors allows you to detect exceptions before they are reported by your business. This helps you identify the source of the error and fix it promptly.
Early warning for business performance metrics
If your message pipeline has specific performance requirements, such as response time (RT) or message latency, configuring alerts for these business metrics helps you manage business risks proactively.
Alert configuration principles
ApsaraMQ for RocketMQ provides a rich set of metrics and alert items. These items can be categorized into three types: running watermarks, messaging performance, and exception events.
Based on extensive experience with production environments, we recommend configuring the following alerts:
The following core monitoring items are basic recommendations. ApsaraMQ for RocketMQ provides a comprehensive set of monitoring metrics. You can configure more fine-grained and comprehensive alerts based on your business needs. For more information, see Monitoring and Alerts.
Alert category | Key alert items | When to configure | Target role |
Instance resource watermarks and consumption metrics |
|
| Resource O&M engineers |
Messaging performance metrics |
|
|
|
Messaging exception events |
|
|
|
Alert configuration entry point
Log on to the ApsaraMQ for RocketMQ console, and in the left navigation bar, click Instances.
In the top menu bar, select a region, such as China (Hangzhou), and then in the instance list, click the target instance name.
In the left-side navigation pane, click Monitoring and Alerts, and then click Create Alert Rule.
Best practices
Configure peak TPS alerts for instance API calls
Background: An ApsaraMQ for RocketMQ 5.0 instance specification defines a base TPS limit for sending and receiving messages, which is the API call rate. If an instance's peak API call TPS exceeds this limit, it is throttled. For more information about the base TPS limits, see Quotas and limits.
Risk if not configured: Without this alert, you receive no warning before the API call rate exceeds the limit, which causes some message send and receive requests to fail once throttling begins.
When to configure: We recommend configuring this alert after the instance is created and you have set the send-to-receive request ratio. To adjust the ratio, follow these steps:
On the Instance Details page, click the Basic Information tab.
Click Edit to open the Modify Configurations panel, where you can adjust the send-to-receive request ratio.
Instance send API calls
Recommended threshold: Set the threshold to 70% of the instance's peak send TPS limit. For example, if the peak send TPS limit is 5,000, set the threshold to 3,500.
Professional Edition and Enterprise Platinum Edition instances support elastic TPS for traffic bursts. If you enable this feature, set your threshold to 70% of the elastic specification limit (Peak Send TPS + Elastic Send TPS).
A serverless instance supports adaptive elasticity. Set your threshold to 70% of the instance's elastic peak send TPS.
You can view the peak send TPS limit and elastic send TPS of an instance on the Instance Details page in the console.
Response: When you receive an alert for the instance send API call rate, follow these steps:
On the Instance Details page, click the Dashboard tab.
In the Throttling-related Metrics section, check the Production TPS MAX Value curve in the Production TPS Watermark chart to identify when the threshold was reached.
In the Instance Overview section, check the Rate of Messages Sent by Producer to Server (messages/min) chart. Correlate the time the threshold was reached with the topic activity to find the source of the abnormal traffic, and analyze its curve to determine if the traffic change is expected.
If the traffic change is unexpected, contact the business team for further analysis.
If the traffic change is expected, the current instance specification is insufficient. Immediately upgrade instance configurations to adjust the messaging compute specification.
Instance receive API calls
Recommended threshold: Set the threshold to 70% of the instance's peak receive TPS limit. For example, if the peak receive TPS limit is 5,000, set the threshold to 3,500.
Professional Edition and Enterprise Platinum Edition instances support elastic TPS for traffic bursts. If you enable this feature, set your threshold to 70% of the elastic specification limit (Peak Receive TPS + Elastic Receive TPS).
A serverless instance supports adaptive elasticity. Set your threshold to 70% of the instance's elastic peak receive TPS.
You can view the peak receive TPS limit and elastic receive TPS of an instance on the Instance Details page in the console.
Response: When you receive an alert for the instance receive API call rate, follow these steps:
On the Instance Details page, click the Dashboard tab.
In the Throttling-related Metrics section, check the Consumption TPS MAX Value curve in the Consumption TPS Watermark chart to identify when the threshold was reached.
In the Instance Overview section, check the Rate of Messages Delivered from Server to Consumer (messages/min) chart. Correlate the time the threshold was reached with the consumer group activity to find the source of the abnormal consumption, and analyze its curve to determine if the traffic change is expected.
If the traffic change is unexpected, contact the business team for further analysis.
If the traffic change is expected, the current instance specification is insufficient. Immediately upgrade instance configurations to adjust the messaging compute specification.
Configure alerts for messages per minute
Background: ApsaraMQ for RocketMQ allows you to monitor message sending and receiving TPS at the topic and consumer group levels. By setting up alerts for these metrics, you can proactively monitor the traffic volume of specific business services.
Risk if not configured: The TPS for a topic's message sending and receiving reflects the business's call frequency. Without this alert, a sudden drop to zero or an unexpected spike in traffic may go unnoticed, potentially leading to business risks.
When to configure: We recommend that you configure this alert after your business goes live and traffic has stabilized.
Messages sent by producers
Recommended threshold: Estimate the alert threshold based on the actual traffic during the stable period after your business goes live.
Response: When you receive a message sending TPS alert, follow these steps:
On the Topics page, click the name of the topic specified in the alert rule.
On the Topic Details page, click the Dashboard tab.
Check the Production curve in the Message Volume (messages/min) chart. Based on your business model, determine if the curve's fluctuation is reasonable and analyze any anomalies.
Messages received by consumers
Recommended threshold: Estimate the alert threshold based on the actual traffic during the stable period after your business goes live.
Response: When you receive a message receiving TPS alert, follow these steps:
On the Groups page, click the ID of the consumer group specified in the alert rule.
On the Group Details page, click the Dashboard tab.
Check the Consumption Rate (messages/min) curve in the Message Production and Consumption Rate Trend (messages/min) chart. Based on your business model, determine if the curve's fluctuation is reasonable and analyze any anomalies.
Configure alerts for Internet outbound bandwidth
Background: ApsaraMQ for RocketMQ 5.0 series instances support Internet access, but this access is limited by the Internet outbound bandwidth. Exceeding the bandwidth limit of your specification impairs Internet access.
Risk if not configured: Without this alert, you will not be warned if your instance's Internet traffic exceeds the bandwidth limit, which can lead to packet loss, client call timeouts, or failures.
When to configure: Configure this alert after a non-serverless instance is created and Internet access is enabled.
NoteA serverless instance supports elastic bandwidth, so you do not need to configure this alert.
Recommended threshold: Set the threshold to 35% of your specification limit. We recommend this value because the target is to alert at 70% of the limit, and the monitoring tool collects data that represents about 50% of the actual traffic (70% × 50% = 35%). For example, if you purchase an instance with a bandwidth of 1 Mbit/s, the recommended alert threshold is 43,750 B/s. You can find your instance's Internet bandwidth information in the Running Information section of the Basic Information tab on the Instance Details page.
NoteWhen estimating the threshold, convert Mbit/s to B/s before calculating. For example:
1 Mbit/s = 1×10^6 bits/s = (1×10^6)/8 B/s = 125,000 B/s. The recommended threshold is125,000 B/s × 0.7 × 0.5 = 43,750 B/s.Response: When you receive an alert for Internet outbound bandwidth, follow these steps:
On the Instance Details page, click the Dashboard tab.
In the Billing Metrics Overview section, check the Outbound Bandwidth curve in the Internet Outbound Traffic Bandwidth chart to identify when the threshold was reached. Ensure that the units in the chart match the units of your alert threshold.
In the Instance Overview section, check the Rate of Messages Sent by Producer to Server (messages/min) and Rate of Messages Delivered from Server to Consumer (messages/min) charts. Correlate the time the threshold was reached to find the topic or consumer group with abnormal data, and analyze the curve to determine if the traffic change is expected.
If the traffic change is unexpected, contact the business team for further analysis.
If the traffic change is expected, the current instance specification is insufficient. Immediately upgrade instance configurations to adjust the Internet outbound bandwidth specification.
Configure alerts for message accumulation
Message accumulation statistics can fluctuate and contain inaccuracies. We do not recommend setting an alert threshold for an accumulation of a few dozen messages. If your business is highly sensitive to even small backlogs, monitor consumption delay time instead.
Background: ApsaraMQ for RocketMQ allows you to monitor message accumulation at the consumer group level, which can alert you to downstream consumption backlog scenarios.
Risk if not configured: Although message accumulation is a standard feature of ApsaraMQ for RocketMQ, for real-time processing, you must monitor and control the volume of unprocessed messages to avoid business impact from consumption backlogs.
When to configure: We recommend that you configure this alert after your business goes live and traffic has stabilized.
Recommended threshold: Estimate the alert threshold based on your business tolerance after it goes live.
Response: When you receive an alert for message accumulation, follow these steps:
On the Groups page, click the ID of the consumer group specified in the alert rule.
On the Group Details page, click the Dashboard tab.
Check the Accumulated Messages curve in the Accumulation-related Metrics chart. Analyze the trend of the accumulation to find when it first started.
Based on business changes and application logs, analyze the factors at the initial time of accumulation to find the cause. For more information about the principles of message consumption, see Consumer types.
Based on the cause, decide whether to scale out the consumer application or fix defects in the consumption logic.
Configure alerts for consumption delay time
The consumption delay time is calculated based on the oldest message among all unconsumed messages in the current consumer group, making it a cumulative and sensitive metric. When you receive an alert for consumption delay time, you must first determine whether the delay is caused by a few stuck messages or a global consumption delay.
Background: ApsaraMQ for RocketMQ allows you to monitor consumption delay time at the consumer group level, providing a more specific metric for analyzing consumption backlog scenarios.
Risk if not configured: While message accumulation is a standard feature of ApsaraMQ for RocketMQ, for real-time processing, you must monitor and control the delay of backlogged messages to prevent business impact from consumption delays.
When to configure: We recommend that you configure this alert after your business goes live and traffic has stabilized.
Recommended threshold: Estimate the alert threshold based on your business tolerance after it goes live.
Response: When you receive an alert for consumption delay time, follow these steps:
On the Groups page, click the ID of the consumer group specified in the alert rule.
On the Group Details page, click the Dashboard tab.
Check the Accumulated Messages curve in the Accumulation-related Metrics chart. Analyze the trend of the accumulation to find when it first started.
Based on business changes and application logs, analyze the factors at the initial time of accumulation to find the cause. For more information about the principles of message consumption, see Consumer types.
Based on the cause, decide whether to scale out the consumer application or fix defects in the consumption logic.
Configure alerts for throttling events
Background: ApsaraMQ for RocketMQ monitors throttling events for a specific instance. By monitoring the number of throttling events, you can understand the impact on your current business.
Risk if not configured: A high number of throttling events indicates that instance specifications have been significantly exceeded. You should promptly upgrade the instance configurations.
When to configure: After your business goes live and traffic has stabilized.
Instance-level throttling count: Configure alerts after the instance is created.
Topic- and group-level throttling count: Configure alerts after your business goes live and traffic has stabilized.
Recommended threshold: Estimate the alert threshold based on your business tolerance after it goes live.
Response: When you receive an alert for throttling events, follow these steps:
On the Instance Details page, click the Dashboard tab.
In the Throttling-related Metrics section, check the Throttled Request Distribution (Production) chart to analyze the timing and patterns of throttling events.
In the Instance Overview section, review the Rate of Messages Sent by Producer to Server (messages/min) metric. Based on the timing of the throttling events, identify the topic with abnormal data and check its curve to determine if the traffic increase is expected.
Based on this analysis, if the traffic increase is expected, upgrade the instance configurations. If it is not expected, investigate the source of the abnormal traffic.