Create and manage Prometheus alert rule templates - Application Real-Time Monitoring Service

You can use Prometheus alert rule templates to centrally manage alert rules for multiple Prometheus instances across different regions. This topic describes how to create and manage Prometheus alert rule templates to ensure rule consistency and efficiency.

Background information

If you have multiple cross-region Prometheus instances, creating alerting rules for each Prometheus instance individually is labor-intensive and makes synchronized management of the rules for each Prometheus instance difficult. To solve this problem, Managed Service for Prometheus provides the alerting rule template feature. This feature helps you quickly create alerting rules for multiple Prometheus instances and manage them in a unified way, which reduces the cost of managing alerting rules across multiple Prometheus instances.

Create Prometheus alerting rule template

Log on to the ARMS console.
In the navigation pane on the left, choose Managed Service for Prometheus > Prometheus Alert Rule Template.
In the upper-right corner of the page, click Create Prometheus Alert Rule Template.

On the Create Prometheus Alert Rule page, configure the following parameters.

You can create a Prometheus alert rule template using a static threshold or a custom PromQL.

The static threshold check type provides preset alert metrics. You can select a preset metric to quickly create an alert rule.
To monitor metrics that are not preset, you can use the custom PromQL check type to create an alert rule.

Table 1. Static threshold check types

Parameter	Description	Example
Alert rule template name	The name of the alert rule template.	Production cluster - Container CPU utilization alert
Alert rule template description	Optional. Describe the template. You can include its purpose, applicable scenarios, or other notes.	None
Check type	Select Static Threshold.	Static Threshold
Alert group	Select an alert group.	Kubernetes Workloads
Alert metric	Select the metric for which you want to configure an alert. Each alert group has different metrics.	Container CPU utilization
Alert condition	Set the conditions for triggering an alert event based on the preset content of the alert metric.	The alert condition is met when the container CPU utilization is `greater than` `80`%.
Filter condition	Based on the alert metric, set the scope for the alert rule. An alert event is triggered for any resource that meets both the filter conditions and the alert condition. The available filter conditions are: Traverse: The alerting rule applies to all resources in the current Prometheus instance. By default, the filter condition is set to Traversal. Equal To: After selecting this condition, enter a specific resource name. The alert rule will apply only to that resource. You cannot enter multiple resources. Is not equal: After selecting this condition, enter a specific resource name. The alert rule will apply to all resources except that one. You cannot enter multiple resources. Match Regular Expression: After selecting this condition, enter a regular expression to match resource names. The alert rule will apply to all resources that match the expression. Do Not Match Regular Expression: After selecting this condition, enter a regular expression to match resource names. The alert rule will filter out all resources that match the expression.	Instance IP: Traverse
Duration	Trigger an alert event as soon as the alert condition is met: An alert event is triggered if any data point meets the threshold. Trigger an alert event only when the alert condition persists for N minutes: An alert event is triggered only if the threshold is met for N minutes or more.	1
Alert severity	The severity of the alert. The default severity is Default. The severity levels, from lowest to highest, are Default, P4, P3, P2, and P1.	Default
Alert content	The alert information that users receive. You can use Go template syntax to customize alert parameter variables in the alert content.	`Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value {{ printf "%.2f" $value }}%`
Advanced Settings
Labels	Set alert labels. The labels can be used as options for matching rules in notification policies.	None
Annotations	Set annotations for the alert.	None

Table 2. Custom PromQL check types

Parameter	Description	Example
Alert rule template name	The name of the alert rule template.	Pod CPU utilization is greater than 80%
Alert rule template description	Optional. Describe the template. You can include its purpose, applicable scenarios, or other notes.	None
Check type	Set to Custom PromQL.	Custom PromQL
Custom PromQL statement	Use a PromQL statement to set the alert rule expression.	`max(container_fs_usage_bytes{pod!="", namespace!="arms-prom",namespace!="monitoring"}) by (pod_name, namespace, device)/max(container_fs_limit_bytes{pod!=""}) by (pod_name,namespace, device) * 100 > 90`
Duration	Trigger an alert event as soon as the alert condition is met: An alert event is triggered if any data point meets the threshold. Trigger an alert event only when the alert condition persists for N minutes: An alert event is triggered only if the threshold is met for N minutes or more.	1
Alert severity	The severity of the alert. The default severity is Default. The severity levels, from lowest to highest, are Default, P4, P3, P2, and P1.	Default
Alert content	The alert information that users receive. You can use Go template syntax to customize alert parameter variables in the alert content.	`Namespace: {{$labels.namespace}}/Pod: {{$labels.pod_name}}/Disk device: {{$labels.device}} utilization exceeds 90%, current value {{ printf "%.2f" $value }}%`
Advanced Settings
Labels	Set alert labels. The labels can be used as options for matching rules in notification policies.	None
Annotations	Set annotations for the alert.	None

Application template

After you create a Prometheus alert template, you can apply it to create alert rules for a Prometheus instance. You can also use the template to update existing alert rules for a Prometheus instance.

On the Prometheus Alert Rule Template page, click Apply Template for the template that you want to apply.
On the Apply Template page, on the Prometheus Instance Selection Mode tab, select a Prometheus instance and click OK.

Note
You can filter Prometheus instances associated with this alert template by Prometheus instance name, region, and Prometheus instance type.
In the dialog box that appears, select whether to update existing alert rules and then click OK.
ARMS uses the current alert rule template to create alert rules in the selected Prometheus instances.
- If you do not select Update Existing Alert Rules: If the selected Prometheus instance already has alert rules that were created from the current alert template, the message Alert Rules Have Already Been Created By Using This Template And Will Not Be Updated appears. The existing alert rules are not updated.
- Update Existing Alert Rules: If the selected Prometheus instance already contains alert rules created from the current alert template, the Prometheus alert rules are updated to match the latest content of the alert template.
  
  Important
  If an alert rule was modified individually but its mapping to the template was retained, the individual modifications are overwritten and lost.

Delete the template

If a Prometheus alert template is no longer needed, you can delete the Prometheus alert template. When you delete a template, you can specify whether to keep the alert rules that were created from it.

On the Prometheus Alert Rule Template page, click Delete to the right of the target alert rule template.
In the dialog box that appears, select whether to also delete the alert rules created from the template and click Confirm.
- Select Delete Alert Rules Created from Template: The Prometheus alert rules associated with the template are deleted. If a Prometheus alert rule that was created from the template was manually edited and you selected Do Not Retain The Mapping Between The Template And The Rule when you saved it, the rule is not deleted.
- If you do not select Delete Alert Rules Created from Template, the Prometheus alert rules created from the template are retained.

View alert rules created from a template

You can view and manage alert rules that were created from a template in batches.

On the Prometheus Alert Rule Template page, click View Rules Created From Template to the right of the target alert rule template.
You can manage alert rules in batches on the Alert Rules Created from Template page.
- To enable alerts in batches: Select the desired alert rules and click Enable Alert Rules.
- To disable alerts in batches: Select the desired alert rules and click Disable Alert Rules.
- To delete alerts in batches: Select the desired alert rules and click Delete Alert Rules.

Notes

When you edit and save a Prometheus alert rule that was created from a Prometheus alert rule template, you must choose whether to retain the mapping between the template and the rule.

Retain the mapping between this alert rule and the alert rule template: If you retain this mapping, your current edits may be overwritten if you apply the corresponding alert rule template to the Prometheus instance for this rule and select the force update option.
Remove the mapping between this alert rule and the alert rule template: After the mapping is disabled, this alert rule is treated as an independent rule. We recommend that you change the alert rule name. Otherwise, if you apply the corresponding alert rule template to the Prometheus instance again, the creation fails because of a duplicate alert rule name.