You can create a Prometheus alert rule to monitor specific metrics and trigger alert events when specified conditions are met. Configure a notification policy to receive alerts through SMS, email, phone calls, DingTalk, WeCom, or webhooks.
Prerequisites
An integrated Prometheus instance is required. For more information, see the following topics:
Procedure
Log on to the ARMS console.
-
In the left-side navigation pane, choose .
-
On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.
Create a Prometheus alert rule based on a static threshold
The static threshold check type provides predefined alert metrics. Select a metric to quickly create an alert rule.
-
On the Create Prometheus Alert Rule page, set the following parameters.
Parameter
Description
Example
Alert name
The alert rule name.
Production cluster - container CPU usage alert
Check type
Select Static Threshold.
Static Threshold
Prometheus instance
Select the Prometheus instance for which you want to create the alert rule.
Production cluster
Alert group
Select an alert group.
Different Prometheus types support different alert groups. The available alert group options vary depending on the selected Prometheus instance type.
Kubernetes workload
Alert metric
Select the metric for which you want to configure the alert. The available metrics vary based on the selected alert group.
Container CPU usage
Alert condition
Set the conditions that trigger an alert event.
The alert condition is met when the container CPU usage is
greater than80%.Filter conditions
Defines the scope of the alert rule. An alert event is generated only when a resource meets both the filter and alert conditions.
The following filter conditions are available:
-
Traverse: Applies the alert rule to all resources in the current Prometheus instance. This is the default condition.
-
Equal To: Applies the alert rule to a single, specified resource. You cannot specify multiple resources.
-
Not Equal To: Applies the alert rule to all resources except for the one specified. You cannot specify multiple resources.
-
Match Regular Expression: Applies the alert rule to all resources whose names match the specified regular expression.
-
Do Not Match Regular Expression: Applies the alert rule to all resources except for those matching the specified regular expression.
Note-
After you configure the filter conditions, the Data Preview section appears.
-
The filter condition must not exceed 300 characters.
Traverse
Data preview
The Data Preview section shows the corresponding PromQL query and a time series chart of the metric.
By default, it displays the real-time value for a single resource. Use the filters to view different resources and time ranges.
Note-
The alert threshold is represented by a red horizontal line in the chart. Time series data that meets the threshold is shown in dark red, while data that does not meet the threshold is shown in blue.
-
Hover over the time series curve to view resource details at a specific point in time.
-
Select a time range on the chart to zoom in and view the corresponding time series curve.
--
Duration
-
Triggers an alert if a single data point meets the threshold.
-
Triggers an alert only if the condition persists for a specified duration (N minutes).
The duration parameter does not support sub-minute (second-level) configurations. This is a product-level mechanism limitation, not abnormal behavior.
1
Alert level
Define a custom alert level. The default level is Default. The severity increases from Default, P4, P3, P2, to P1.
Default
Alert message
The notification content. You can use Go template variables to customize the alert message.
Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU usage {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, Current value: {{ printf "%.2f" $value }}%
Alert notification
-
Simple Mode: Allows you to configure Notification Receiver, Notification Period, and Whether to Resend Notifications.
-
Standard Mode:
-
Do Not Specify Notification Policy: Select this option to link this alert to a notification policy later. After creating the alert rule, go to the Notification Policies page to create a policy that matches this rule by name or other conditions. For more information, see Notification policies.
-
Select a policy from the drop-down list: ARMS automatically adds a matching rule to the selected notification policy. The matching rule uses the alert rule ID (displayed as the alert rule name) to ensure that alert events from this rule are matched by the selected policy.
ImportantSpecifying a notification policy here only ensures that alert events from the current rule are matched by the selected policy. Events from this rule may also be matched by other policies that use fuzzy matching. The relationship between alert rules and notification policies is many-to-many.
-
Do Not Specify Notification Policy
Advanced Settings
Alert check cycle
The check interval in minutes. The minimum and default is 1 minute. Even if you enter a value less than 1 minute (for example, 15 seconds) in the interface, the system still performs checks at 1-minute intervals. This is a product-level mechanism limitation, not abnormal behavior.
1
Data Completion Check
-
Yes
-
No
Yes
Tags
Tags for the alert rule. Tags can be used as matching conditions in notification policies.
--
Annotations
Annotations for the alert rule.
--
-
-
After you complete the settings, click Save. The Prometheus Alert Rules page shows the status of the current alert rule.
If the Status of the alert rule is Automatic Interruption, edit the alert rule based on the provided reason for the interruption and click Start. Then, in the dialog box that appears, click Confirm. If you encounter a rule interruption issue that you cannot resolve, contact the ARMS alert service on DingTalk (ID: d9j_rg9e4062f) for assistance.
An alert rule might be automatically interrupted for the following reasons:
-
The number of query results exceeds 1,500.
-
No notification object is configured in Alert Management.
-
The Prometheus instance is uninstalled or unavailable.
-
Create a Custom PromQL alert rule
To monitor metrics not covered by static threshold options, use the Custom PromQL check type.
-
On the Create Prometheus Alert Rule page, set the following parameters.
Parameter
Description
Example
Alert name
The alert rule name.
Pod CPU usage is greater than 8%
Check type
Set to Custom PromQL.
Custom PromQL
Prometheus instance
Select the Prometheus instance for which you want to create the alert rule.
--
Reference alert group
Select an alert group.
Different Prometheus types support different alert groups. The available alert group options vary depending on the selected Prometheus instance type.
Kubernetes workload
Reference metrics
Optional. Reference metrics provide Custom PromQL configurations for common metrics. Select a similar metric to populate the query, then modify it as needed.
The Reference Metrics parameter automatically filters the supported alert metrics based on the selected Prometheus instance type.
Pod disk usage alert
Custom PromQL
The PromQL expression for the alert rule.
max(container_fs_usage_bytes{pod!="", namespace!="arms-prom",namespace!="monitoring"}) by (pod_name, namespace, device)/max(container_fs_limit_bytes{pod!=""}) by (pod_name,namespace, device) * 100 > 90
Data preview
The Data Preview section shows the corresponding PromQL query and a time series chart of the metric.
By default, it displays the real-time value for a single resource. Use the filters to view different resources and time ranges.
Note-
Hover over the time series curve to view resource details at a specific point in time.
-
Select a time range on the chart to zoom in and view the corresponding time series curve.
--
Duration
-
Triggers an alert if a single data point meets the threshold.
-
Triggers an alert only if the condition persists for a specified duration (N minutes).
The duration parameter does not support sub-minute (second-level) configurations. This is a product-level mechanism limitation, not abnormal behavior.
1
Alert level
Define a custom alert level. The default level is Default. The severity increases from Default, P4, P3, P2, to P1.
Default
Alert message
The notification content. You can use Go template syntax to customize the alert message with parameter variables.
Example (disk usage alert): Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value: {{ printf "%.2f" $value }}%
Example (Pod restart alert): Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}} restarted more than {{ $labels.metrics_params_value}} times in {{$labels.metrics_params_time}} minutes, current restarts: {{ $value }}
Use the Pod restart alert template to generate readable notification content for Pod restart scenarios.
Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value: {{ printf "%.2f" $value }}%
Alert notification
-
Simple Mode: Allows you to configure Notification Receiver, Notification Period, and Whether to Resend Notifications.
-
Standard Mode:
-
Do Not Specify Notification Policy: Select this option to link this alert to a notification policy later. After creating the alert rule, go to the Notification Policies page to create a policy that matches this rule by name or other conditions. For more information, see Notification policies.
-
Select a policy from the drop-down list: ARMS automatically adds a matching rule to the selected notification policy. The matching rule uses the alert rule ID (displayed as the alert rule name) to ensure that alert events from this rule are matched by the selected policy.
ImportantSpecifying a notification policy here only ensures that alert events from the current rule are matched by the selected policy. Events from this rule may also be matched by other policies that use fuzzy matching. The relationship between alert rules and notification policies is many-to-many.
-
Do Not Specify Notification Policy
Advanced settings
Alert check cycle
The check interval in minutes. The minimum and default is 1 minute. Even if you enter a value less than 1 minute (for example, 15 seconds) in the interface, the system still performs checks at 1-minute intervals. This is a product-level mechanism limitation, not abnormal behavior.
1
Data Completion Check
-
Yes
-
No
Yes
Tags
Tags for the alert rule. Tags can be used as matching conditions in notification policies.
--
Annotations
Annotations for the alert rule.
--
-
-
Click Save. The Prometheus Alert Rules page shows the status of the current alert rule.
If the Status of the alert rule is Automatic Interruption, edit the rule based on the provided reason, click Start, and then click Confirm in the dialog box. If you encounter an issue that you cannot resolve, contact the ARMS alert service on DingTalk (ID: d9j_rg9e4062f) for assistance.
An alert rule might be automatically interrupted for the following reasons:
-
The number of query results exceeds 1,500.
-
No notification object is configured in Alert Management.
-
The Prometheus instance is uninstalled or unavailable.
-
Manage alert rules
-
For static threshold and Custom PromQL rules created in the Managed Service for Prometheus console, you can perform actions such as edit, start, stop, delete, copy, and view historical alert events.
-
For rules originating from other Alibaba Cloud services, you can view historical alert events and navigate to the source service's alert list.
FAQ
Q1: Why is there a delay in alert recovery notifications?
Alert recovery notifications may be delayed for the following reasons:
-
Root cause: The system has a maximum tolerated delay mechanism. To prevent missed alerts caused by Agent reporting delays, the alert engine looks back at the last 10 minutes of data on each check. Even if metrics no longer meet the alert condition, the system waits for the lookback window to end before confirming recovery.
-
Timing: Recovery notifications are typically sent approximately 10 minutes after the data stops triggering the alert.
-
Mechanism note: Prometheus itself has no built-in automatic recovery logic. ARMS relies on its own detection mechanism to determine when an alert has recovered.
Q2: What should I do if the notification policy is lost after applying an alert rule template?
-
Root cause: Alert rule templates do not contain notification policy configurations. Reapplying a template may cause the rule ID to change, which invalidates the original notification policy matching based on
_aliyun_arms_alert_rule_id. -
Solution: Go to , find the corresponding policy, and update the Dispatch Conditions to use more stable labels — such as
alertnameor the cluster namecluster— instead of the rule ID for matching.
Q3: What should I do if I still receive duplicate alerts after changing the alert expression to increase?
-
Root cause: If the notification policy is configured with "Repeat notification every N minutes" combined with "Manual recovery", the system continuously generates new alert events as long as metrics keep meeting the new expression condition.
-
Solution: Modify the notification policy to change the repeat notification setting to "First notification only", or switch to "Automatic recovery" mode. Also confirm that the new expression has been correctly saved and is in effect.
Q4: How do I troubleshoot when a Prometheus alert rule shows no data or does not trigger?
Follow these steps:
-
Check components: Verify whether the ACK cluster has the kube-state-metrics component installed; install it if missing. Check the Prometheus component status and reinstall the ack-arms-prometheus component if necessary.
-
Verify trigger records: Click Alert History next to the rule to check for records.
-
No records: The rule conditions were never met. Common causes include incorrect metric names, empty data, or an incorrect query time range.
-
Records exist but no notifications sent: Check whether the notification policy dispatch rules (labels, cluster name, etc.) match the alert rule.
-
-
Recommendation: Use the latest Alibaba Cloud-verified alert templates to avoid false positives or missed triggers from older templates.
-
Syntax check: If the page displays an error, verify that the PromQL regex syntax is valid. For example,
{pvc=~}(empty regex) causes an error; change it to{pvc=~".*"}or specify a concrete value.
Q5: Is Prometheus monitoring Pod memory alerting for ACS clusters free of charge?
ACS clusters support Pod memory metrics monitoring and alerting. SMS and voice call alerts incur additional charges, but DingTalk bot and Webhook alerts are free of charge.