Application Real-Time Monitoring Service:Create alert rules for a Prometheus instance

Entry

1. Log on to the ARMS console.

2. In the navigation pane on the left, choose Prometheus Monitoring > Prometheus Alert Rules.

3. On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.

CreatePrometheus an alert rule based on a static threshold

The static threshold check type provides predefined alert metrics. You can select these metrics to quickly create alert rules.

1. On the Create Prometheus Alert Rule page, configure the following alert parameters.

   Parameter	Description	Example
   Alert Name	The name of the alert.	prod-cluster-container-cpu-alert
   Check Type	Select Static threshold.	Static threshold
   Prometheus Instance	Select the Prometheus instance for which you want to create the alert.	Production Cluster
   Alert Group	Select an alert group. Different Prometheus types support different alert groups. The available alert group options change based on the selected Prometheus instance type.	Kubernetes Workloads
   Alert Metric	Select the metric for which you want to configure the alert. Each alert group corresponds to different metrics.	Container CPU utilization
   Alert Condition	Set the conditions that trigger an alert event based on the predefined content of the alert metric.	The alert condition is met when container CPU utilization is greater than 80%.
   Filter Condition	Define the scope of the alert rule based on the alert metric. An alert event is triggered when any resource that meets the filter conditions satisfies the alert rule. The following filter conditions are available: Traverse: The alert rule applies to all resources in the current Prometheus instance. Traverse is the default filter condition. Equal To: After you select this condition, enter a specific resource name. The alert rule applies only to that resource. You cannot enter multiple resource names. Not Equal To: After you select this condition, enter a specific resource name. The alert rule applies to all resources except the specified one. You cannot enter multiple resource names. Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule applies to all resources that match the regular expression. Do Not Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule filters out all resources that match the regular expression. Note After you set the filter conditions, the Data Preview area appears. The filter condition cannot exceed 300 characters.	Traverse
   Data Preview	The Data Preview area displays the Prometheus Query Language (PromQL) statement that corresponds to the alert condition. It also shows the values of the monitoring metric as a time series curve. By default, only the real-time value of one resource is displayed. You can select a target resource and a time range in the filter box of this area to view values for different resources and time ranges. Note The alert threshold appears as a red straight line on the time series curve. The part of the curve that meets the alert threshold is displayed in dark red, and the part that does not is displayed in blue. Hover over the time series curve to view the resource details at a specific point in time. Select a time range on the time series curve to view the curve for that period.	None
   Duration	If an alert condition is met, an alert event is triggered immediately: An alert event is triggered if any data point meets the threshold. An alert event is triggered only after an alert condition persists for N minutes: An alert event is triggered only if the time that the threshold is met is greater than or equal to N minutes. You cannot configure the duration in seconds. This is a product limitation and is expected behavior.	1
   Alert Level	Customize the alert level. The default alert level is Default. The severity increases from Default, P4, P3, P2, to P1.	Default
   Alert Content	The alert information that users receive. You can use Go template syntax to customize alert parameter variables in the alert content.	Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, Current value: {{ printf "%.2f" $value }}%
   Alert Notification	Simple Mode: You can set the Notification Receiver, Notification Period, and Whether to Resend Notifications. Standard Mode: Do not specify a notification policy: If you select this option, after you create the alert rule, you can create a new notification policy on the Notification Policy page and specify matching rules and matching conditions, such as the alert rule name, to match the alert rule. When the alert rule is triggered and generates an alert event, the alert information is sent to the contacts or contact groups specified in the notification policy. For more information, see Notification policies. Specify a notification policy: If you select this option, ARMS automatically adds a matching rule to the corresponding notification policy. The matching rule content is the alert rule ID, which is presented as the alert rule name. This ensures that alert events generated by the current alert rule are matched by the selected notification policy. Important Quickly specifying a notification policy only ensures that alert events from the current alert rule are matched by the selected notification policy and that corresponding alerts are generated. However, events from the current alert rule may also be matched by other notification policies that are configured with fuzzy matching, which also generates alerts. The relationship between alert events and notification policies is a many-to-many mapping.	Do not specify a notification rule
   Advanced Settings
   Alert Check Interval	The interval at which the system checks the alert rule to determine if the data meets the alert conditions. The default is 1 minute, and the minimum is 1 minute. Even if you enter a value less than 1 minute, such as 15 seconds, the system still performs the check every 1 minute. This is a product limitation and is expected behavior.	1
   Check after data is complete	Yes No	Yes
   Tags	Set tags for the alert. The tags can be used as options for notification policy matching rules.	None
   Annotations	Set annotations for the alert.	None

2. After you complete the configuration, click Save. On the Prometheus alert rule list page, you can view the status of the current alert rule.

   If the Status of the alert rule is Automatic Interruption, edit the alert rule to address the cause of the interruption. Then, click Start, and in the dialog box that appears, click Confirm. If you cannot resolve the issue, contact the ARMS alerting feature team on DingTalk (ID: d9j_rg9e4062f) for help.

   An alert rule may be automatically interrupted for the following reasons:

   • The rule query returns more than 1,500 results.

   • No notification recipient is configured in the alert management system.

   • The Prometheus instance is uninstalled or unavailable.

Create a Prometheus alert rule using a custom PromQL query

To monitor metrics that are not covered by pre-configured static thresholds, create an alert rule with a custom Prometheus Query Language (PromQL) query.

1. On the Create Prometheus Alert Rule page, set the following alert parameters.

   Parameter	Description	Example
   Alert Name	The name of the alert.	Pod CPU usage is greater than 8%
   Check Type	Set to Custom PromQL query.	Custom PromQL query
   Prometheus Instance	Select the Prometheus instance for which you want to create the alert.	None
   Reference Alert Group	Select an alert group. Different Prometheus types support different alert groups. The available alert group options change based on the selected Prometheus instance type.	Kubernetes Workload
   Reference Alert Metric	Optional. Reference metrics provide custom PromQL configurations for common metrics. Select a similar metric to pre-fill the fields. Then, modify the configuration as needed. The Reference Metric parameter automatically filters the supported alert metrics based on the selectedPrometheus instance type.	Pod disk usage alert
   Custom PromQL Statement	Use a PromQL statement to set the alert rule expression.	Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value {{ printf "%.2f" $value }}%max(container_fs_usage_bytes{pod!="", namespace!="arms-prom",namespace!="monitoring"}) by (pod_name, namespace, device)/max(container_fs_limit_bytes{pod!=""}) by (pod_name,namespace, device) * 100 > 90
   Data Preview	The Data Preview area displays the Prometheus Query Language (PromQL) statement that corresponds to the alert condition. It also shows the values of the monitoring metric as a time series curve. By default, only the real-time value of one resource is displayed. You can select a target resource and a time range in the filter box of this area to view values for different resources and time ranges. Note Hover the mouse over the time series curve to view resource details for a specific data point. Select a time range on the curve to view the time series for that period.	None
   Duration	An alert is triggered immediately if any data point meets the threshold. An alert is triggered only if the condition persists for N consecutive minutes. Configuring the duration in seconds is not supported. This is a product limitation and is expected behavior.	1
   Alert Level	Customize the alert level. The default alert level is Default. The severity increases from Default, P4, P3, P2, to P1.	Default
   Alert Content	The alert information that users receive. Use Go template syntax to customize parameter variables in the alert content. The following example shows a template for a pod restart alert. This helps you configure readable notification content: Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}} restarted more than {{ $labels.metrics_params_value}} times in {{$labels.metrics_params_time}} minutes. Current restarts: {{ $value }}	Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} usage exceeds 90%, current value {{ printf "%.2f" $value }}%
   Alert Notification	Simple Mode: You can set the Notification Receiver, Notification Period, and Whether to Resend Notifications. Standard Mode: Do not specify a notification policy: If you select this option, after you create the alert rule, you can create a new notification policy on the Notification Policy page and specify matching rules and matching conditions, such as the alert rule name, to match the alert rule. When the alert rule is triggered and generates an alert event, the alert information is sent to the contacts or contact groups specified in the notification policy. For more information, see Notification policies. Specify a notification policy: If you select this option, ARMS automatically adds a matching rule to the corresponding notification policy. The matching rule content is the alert rule ID, which is presented as the alert rule name. This ensures that alert events generated by the current alert rule are matched by the selected notification policy. Important Quickly specifying a notification policy only ensures that alert events from the current alert rule are matched by the selected notification policy and that corresponding alerts are generated. However, events from the current alert rule may also be matched by other notification policies that are configured with fuzzy matching, which also generates alerts. The relationship between alert events and notification policies is a many-to-many mapping.	Do not specify a notification rule
   Advanced Settings
   Alert Check Period	The interval, in minutes, for checking the alert rule. The minimum value is 1 minute. If you enter a value less than 1 minute, such as 15 seconds, the system still checks every 1 minute. This is a product limitation and is expected behavior.	1
   Check After Data Is Complete	Yes No	Yes
   Tags	Set tags for the alert. The tags can be used as options for notification policy matching rules.	None
   Annotations	Set annotations for the alert.	None

2. After you configure the parameters, click Save. On the Prometheusalert rules list page, you can view the status of the alert rule.

   If the Status of the alert rule is Automatic Interruption, edit the rule again based on the provided reason. Then, click Start and click Confirm in the dialog box that appears. If you cannot resolve the interruption, contact the ARMS alerting feature service account (d9j_rg9e4062f) for help.

   An alert rule can be automatically interrupted for the following reasons:

   • The rule query returns more than 1,500 results.

   • No notification recipient is configured in the alert management system.

   • The Prometheusinstance is uninstalled or unavailable.

Manage alerting rules

• For alerts created on the Alerting Rules page of the Managed Service for Prometheusconsole, such as static threshold and custom Prometheus Query Language (PromQL) alerts, you can edit, enable (for alerts in the Stopped state), disable (for alerts in the Running state), delete, and copy the alerts, along with view their event history.

• For alerts generated in the consoles of other Alibaba Cloud products, you can view the alert event history or return to the alert list of the corresponding product.

After you modify an alerting rule, any existing unrecovered alert events continue to trigger based on the previous rule. The new rule takes effect for subsequent alert triggers only after you claim and resolve the corresponding unrecovered events on the Alert Sending History page.

FAQs