You can use ApsaraMQ for RocketMQ with Cloud Monitor to configure alert rules. This lets you monitor the real-time status of your instances and key business metrics, get notified promptly of anomalies, and proactively manage risks in your production environment.
Background information
ApsaraMQ for RocketMQ provides a fully managed messaging service. Each instance specification comes with a clear SLA guarantee, ensuring that after you purchase an instance, metrics such as messaging TPS and message storage meet their specifications.
Although you do not need to manage instance performance, you must monitor whether your business workload is approaching the instance's specification limits. ApsaraMQ for RocketMQ integrates with Cloud Monitor to provide a free, out-of-the-box monitoring and alerting service to help you address the following issues:
-
Proactive alerts for instance specification watermarks
If your actual usage exceeds the instance's specification limits, ApsaraMQ for RocketMQ enforces strict throttling. By configuring alerts for instance specification watermarks in advance, you can identify the risk of exceeding these limits and upgrade your instance promptly to avoid business disruptions from throttling.
-
Proactive alerts for business logic errors
You may encounter errors when sending or receiving messages. Configuring alerts for call errors allows you to detect anomalies before users report them, enabling you to promptly identify and fix the source of the errors.
-
Proactive alerts for business performance metrics
If your message-oriented middleware has specific performance requirements, such as response time (RT) or message latency, configuring alerts for these business metrics helps you proactively manage business risks.
Alert configuration principles
ApsaraMQ for RocketMQ provides a rich set of metrics and monitoring items for alerts. These items can be categorized into three types of alerts: operational watermarks, send/receive performance, and error events.
Based on extensive production experience, we recommend configuring alerts according to the following principles:
The following key monitoring items are baseline recommendations. ApsaraMQ for RocketMQ provides a comprehensive set of monitoring metrics, allowing you to configure more granular and extensive alerts based on your business needs. For more information, see Monitoring and Alerting.
|
Category |
Key alert |
Recommendation |
Role |
|
Instance operational watermark and consumption metrics |
Instance API call frequency |
|
Operations staff |
|
Message send/receive performance metrics |
|
|
|
|
Message send/receive error events |
|
|
|
Access alert rule configuration
Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region, such as China (Hangzhou). On the Instances page, click the name of the instance that you want to manage.
-
In the left-side navigation pane, click Monitoring and Alerts and then click Create Alert Rule.
Best practices
Configure an alert for instance API call frequency
-
Background: Each ApsaraMQ for RocketMQ instance has a defined TPS limit for message sending and receiving API calls. For example, a Standard Edition instance supports 5,000 API calls per second. If the API call frequency exceeds this limit, the instance will be throttled.
-
Risk if not configured: Without this alert, you won't be warned before your instance's API call limit is exceeded, which will cause some message requests to fail due to throttling.
-
When to configure: We recommend configuring this alert immediately after the instance is created.
Set the Rule Name to Instance API call frequency alert. For Metric Type, select Basic Metrics, and for Metric, select Instance / Instance API call frequency(Instance). Configure three alert levels: Critical (notifies by phone, SMS, email, and webhook), Warn (notifies by SMS, email, and webhook), and Info (notifies by email and webhook). For each level, set the condition to trigger after 3 consecutive periods (1 period = 1 minute) if Sum >= Threshold (unit: count/s). For the Critical level, enter your desired threshold.
-
Recommended threshold: We recommend setting the threshold to 70% of the instance's peak TPS. For example, if your instance has a peak TPS of 10,000, set the alert threshold to 7,000. You can find the peak TPS of the instance on the Instance Details page in the console.
-
Alert handling: If you receive an instance API call frequency alert, follow these steps:
-
On the Instance Details page, click the Dashboard tab.
-
In the Instance Message Volume Overview section, check the TPS Max value curve under Instance Request Count Metrics (Production + Consumption) to determine the time pattern when the alert threshold is reached.
-
In the Message Business Metrics Overview section, review the Top 20 Topics by Message Production Rate (messages/minute) and Top 20 GroupIDs by Message Consumption Rate (messages/minute) metrics. Based on the time pattern when the threshold was reached, identify the topic and group with abnormal data and analyze their curves to determine if the traffic change is expected.
-
If the business change is abnormal, contact the business team to further analyze the cause of the anomaly.
-
If the traffic increase is expected, this indicates the instance specification is insufficient. Upgrade the instance immediately. For instructions, see Upgrade or downgrade an instance.
-
Configure alerts for message production and consumption rates
-
Background: ApsaraMQ for RocketMQ supports monitoring of message sending and receiving TPS at the topic and group levels. This allows you to set up alerts for the TPS of specific business applications, helping you understand your business scale in a timely manner.
-
Risk if not configured: Without this alert, traffic drops or surges can occur without warning, leading to unexpected business risks.
-
When to configure: We recommend configuring this alert after your business goes live and traffic has stabilized.
Producer send rate
In the rule settings dialog box, set the Rule Name to Producer messages sent per minute alert. For Metric Type, select the Basic Metrics tab, and for Metric, select Producer / Number of messages sent per minute(Topic). Configure three alert levels: Critical (notifies by phone, SMS, email, and webhook), Warn (notifies by SMS, email, and webhook), and Info (notifies by email and webhook). Set the condition for each level to trigger after 3 consecutive periods (1 period = 1 minute) if Sum >= Threshold (unit: count/m), and enter the appropriate threshold based on your business needs. For Dimension, select topic.
-
Recommended threshold: After your business goes live, estimate the alert threshold based on the stable traffic volume.
-
Alert handling: If you receive a message sending and receiving TPS alert, follow these steps:
-
On the Topics page, click the name of the topic specified in the alert rule.
-
On the Topic Details page, click the Dashboard tab.
-
In the Message Volume (messages/minute) chart, check the Production curve. Based on your business model, determine if the curve's changes are reasonable and analyze for any anomalies.
-
Consumer receive rate
In the Set Rule Description dialog, set the Rule Name to Consumer messages received per minute alert. For Metric Type, select Basic Metrics, and for Metric, select Consumer / Number of messages received per minute(Group). Configure three alert levels: Critical, Warn, and Info. Select Sum as the aggregation method. Set the condition to trigger after 3 consecutive periods (1 period = 1 minute) if the value is greater than or equal to the threshold, and enter the appropriate threshold. For Dimension, select groupId and specify the target consumer group.
-
Recommended threshold: After your business goes live, estimate the alert threshold based on the stable traffic volume.
-
Alert handling: If you receive a message sending and receiving TPS alert, follow these steps:
-
On the Groups page, click the Group ID specified in the alert rule.
-
On the Group Details page, click the Dashboard tab.
-
In the Trend of Message Production and Consumption Rate (messages/minute) chart, check the Consumption curve. Based on your business model, determine if the curve's changes are reasonable and analyze for any anomalies.
-
Configure an alert for message accumulation
Message accumulation statistics can be volatile and prone to error. We do not recommend setting a monitoring threshold for an accumulation of only a few dozen messages. If your business is highly sensitive to even small accumulations, we suggest using the consumption latency threshold for monitoring instead.
-
Background: ApsaraMQ for RocketMQ supports monitoring message accumulation at the consumer group level, which can be used to alert for downstream consumption backlog issues.
-
Alerting for message accumulation is a key capability of ApsaraMQ for RocketMQ. However, for scenarios requiring real-time message processing, it is crucial to monitor and control the number of unprocessed messages to avoid business impact from consumption backlogs.
-
When to configure: We recommend configuring this alert after your business goes live and traffic has stabilized.
In the Set Rule Description dialog, set the Rule Name to Message accumulation alert. For Metric Type, select Basic Metrics, and for Metric, select Consumer / Number of accumulated messages(Group). Configure three alert levels: Critical (notifies by phone, SMS, email, and webhook), Warn (notifies by SMS, email, and webhook), and Info (notifies by email and webhook). For each level, set the condition to trigger after 3 consecutive periods (1 period = 1 minute) if Sum >= Threshold and the statistical method is count. For Dimension, select groupId and specify the corresponding consumer group.
-
Recommended threshold: After your business goes live, estimate the alert threshold based on your acceptable tolerance levels.
-
Alert handling: If you receive a message accumulation alert, follow these steps:
-
On the Groups page, click the Group ID specified in the alert rule.
-
On the Group Details page, click the Dashboard tab.
-
In the Accumulation-related Metrics (messages) chart, check the Accumulated Messages curve. Analyze the trend of the accumulation and find when the accumulation started.
-
Based on business changes and application logs, analyze the factors at the initial accumulation time to find the root cause. For more information about the causes of message accumulation, see How to handle message accumulation.
-
Based on the root cause, decide whether to scale out the consumer application or fix defects in the consumption logic.
-
Configure an alert for consumption latency
Because this metric is cumulative (calculated from the oldest unconsumed message in the consumer group), it is highly sensitive. If you receive a consumption latency alert, you first need to determine whether the business impact is due to a few stuck messages or a global consumption delay.
-
Background: ApsaraMQ for RocketMQ supports monitoring consumption latency at the consumer group level, providing a more specific metric for analyzing message accumulation scenarios.
-
Alerting for message accumulation is a key capability of ApsaraMQ for RocketMQ. However, for scenarios requiring real-time message processing, it is crucial to monitor and control the latency of accumulated messages to avoid business impact from consumption delays.
-
When to configure: We recommend configuring this alert after your business goes live and traffic has stabilized.
In the rule settings dialog, set the Rule Name to Message processing latency alert. For Metric Type, select Basic Metrics, and for Metric, select Consumer / Message processing latency(GroupId). Configure three alert levels: Critical (notifies by phone, SMS, email, and webhook), Warn (notifies by SMS, email, and webhook), and Info (notifies by email and webhook). For each level, set the condition to trigger after 3 consecutive periods (1 period = 1 minute) if Maximum >= Threshold (unit: ms), and enter the appropriate thresholds for each level based on your business needs. For Dimension, select the corresponding groupId.
-
Recommended threshold: After your business goes live, estimate the alert threshold based on your acceptable tolerance levels.
-
Alert handling: If you receive a consumption latency alert, follow these steps:
-
On the Groups page, click the Group ID specified in the alert rule.
-
On the Group Details page, click the Dashboard tab.
-
In the Accumulation-related Metrics (messages) chart, check the Accumulated Messages curve. Analyze the trend of the accumulation and find when the accumulation started.
-
Based on business changes and application logs, analyze the factors at the initial accumulation time to find the root cause. For more information about the causes of message accumulation, see How to handle message accumulation.
-
Based on the root cause, decide whether to scale out the consumer application or fix defects in the consumption logic.
-
Configure an alert for dead-letter messages
-
Background: ApsaraMQ for RocketMQ supports dead-letter messages. Messages that fail consumption and exceed the maximum number of retries are delivered to a dead-letter queue for you to handle manually. Monitoring the number of messages entering the dead-letter queue helps you promptly detect unexpected and indeterminate issues in your business.
-
Risk if not configured: Ignoring dead-letter messages can lead to incomplete business processing.
-
When to configure: We recommend configuring this alert after your business goes live and traffic has stabilized.
In the Set Rule Description dialog, set the Rule Name to Dead-letter message alert. For Metric Type, select Basic Metrics, and for Metric, select Consumer / Number of dead-letter messages generated per minute(Group). Under Thresholds and Alert Levels, configure three levels: Critical (notifies by phone, SMS, email, and webhook), Warn (notifies by SMS, email, and webhook), and Info (notifies by email and webhook). For each level, set the condition to trigger after 3 consecutive periods (1 period = 1 minute), the aggregation method to Sum, and the condition to Value >= Threshold (unit: count/m). For Dimension, select groupId and set its value to the corresponding consumer group.
-
Recommended threshold: We recommend estimating the alert threshold based on your acceptable tolerance levels after your business goes live and stabilizes.
-
Alert handling: If you receive an alert for the number of dead-letter messages, follow these steps:
-
Query the dead-letter messages and analyze the list of original messages. For instructions, see Dead-letter queue.
-
Based on the Topic and Message ID of the original message, query the message trace to analyze the cause of the consumption failure. For instructions, see Query a message trace.
-
Based on the cause of the message consumption failure, determine the appropriate corrective measures.
-
Configure an alert for throttling count
-
Background: ApsaraMQ for RocketMQ allows you to use throttling events on a specified instance as a monitoring item. By monitoring the throttling count, you can understand the extent of the impact on your business.
-
Risk if not configured: A higher throttling count indicates a more severe breach of instance specifications. You should upgrade the instance specification in a timely manner.
-
When to configure: After your business goes live and traffic has stabilized.
-
Instance-level throttling count: We recommend configuring this alert after the instance is created.
-
Topic- and group-level throttling count: We recommend configuring this alert after your business goes live and traffic has stabilized.
-
In the Set Rule Description dialog, set the Rule Name to Throttling count alert. For Metric Type, select Basic Metrics, and for Metric, select Producer / Throttled messages per minute(Instance). In the Thresholds and Alert Levels section, set the condition for each level to trigger after 3 consecutive periods (1 period = 1 minute), the aggregation method to Sum, and the condition to Value >= Threshold (unit: count/m). The notification methods are: Critical (phone, SMS, email, and webhook), Warn (SMS, email, and webhook), and Info (email and webhook).
-
Recommended threshold: After your business goes live, estimate the alert threshold based on your acceptable tolerance levels.
-
Alert handling: If you receive a throttling count alert, follow these steps:
-
On the Instance Details page, click the Dashboard tab.
-
In the Instance Message Volume Overview section, check the Number of Throttled Requests curve to analyze the timing and pattern of throttling events.
-
In the Message Business Metrics Overview section, review the Top 20 Topics by Message Production Rate (messages/minute) metric. Based on the timing and pattern of the throttling, identify the topic with abnormal data and check its curve to determine if the traffic increase aligns with business expectations.
-
Based on this analysis, if the traffic increase is expected, upgrade the instance specification. If it is not expected, investigate the source of the abnormal traffic.
-