The new version of Managed Service for Prometheus is integrated with Cloud Monitor, which lets you create Prometheus alert rules directly in Cloud Monitor. By default, alert events generated by these rules are integrated into the ARMS alert management system. This topic describes how to create Prometheus alert rules and use the ARMS alert management system to send alert notifications to channels such as DingTalk and text messages.
Prerequisites
Your cloud products are connected to Prometheus.
ARMS is activated.
Create an alert rule
Log on to the Cloud Monitor console.
In the navigation pane on the left, choose .
On the Alert Rule page, select a destination region, and then click Create Alert Rule.
In the Create Alert Rule panel, set the alert parameters.
Parameter
Description
Example
Rule Name
The name of the alert rule.
Production cluster - container CPU utilization alert
Data Source
Select the Prometheus instance for which you want to create an alert rule.
Production cluster
Detection Condition Definition Method
Configure based on a preset metric: Prometheus provides a set of preset alert metrics. You can select a preset metric and configure an alert rule for the metric.
Custom PromQL: Enter a Prometheus Query Language (PromQL) statement directly.
Configure based on a preset metric
Based on Predefined Metrics - Metric Group
Select a metric group.
Kubernetes Load
Based on Predefined Metrics - Metric
Select the metric for which you want to configure an alert rule. Different metric groups correspond to different metrics.
Container CPU utilization
Based on Predefined Metrics - Filter Condition
Specify the scope of the alert rule based on the alert metric. An alert event is generated for any resource that meets the filter condition and the alert condition.
The following filter conditions are available:
Traverse: The alert rule applies to all resources in the current Prometheus instance. This is the default filter condition.
Equal To: After you select this condition, enter a specific resource name. The created alert rule applies only to the specified resource. You cannot specify multiple resources.
Not Equal To: After you select this condition, enter a specific resource name. The created alert rule applies to all resources except the specified one. You cannot specify multiple resources.
Regular Match: After you select this condition, enter a regular expression to match resource names. The alert rule applies to all resources that match the regular expression.
Regular Not Match: After you select this condition, enter a regular expression to match resource names. The alert rule filters out all resources that match the regular expression.
Traverse
Based on Predefined Metrics - Detection Condition
Set the conditions for generating alert events based on the preset content of the alert metric.
The alert condition is met when the container CPU utilization is
greater than80%.Based on Predefined Metrics - PromQL Preview
The PromQL statement that corresponds to the alert condition. You can use the PromQL statement to directly create an alert using the custom PromQL method, or create a dashboard using this PromQL statement.
-
Custom PromQL - Custom PromQL
Use a PromQL statement to set the alert rule expression.
100 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100 > 80
Query Result Preview
Displays the values of the metric configured in the current alert rule as a time series curve.
None
Severity
The default severity is P4. The severity levels in ascending order are P4, P3, P2, and P1.
P4
Duration
An alert event is generated only when the threshold is met for N minutes or longer.
1
Alert Check Cycle
The interval at which the alert rule checks whether the data meets the alert condition. The unit is minutes.
1
Detect After Data Completion
Specifies whether to determine whether the alert condition is triggered based on complete data after all monitoring data in the current epoch has been collected and processed.
Yes
Alert Content
Use Go template syntax to customize alert parameter variables in the alert content.
For the Configure based on a preset metric method, the system provides preset templates for different metrics. If a preset template does not meet your needs, you can modify it.
Example:
Node {{ $labels.instance }} CPU utilization{{$labels.metrics_params_opt}} {{$labels.metrics_params_value}}%, current CPU utilization {{ printf "%.2f" $value }}%You can also add custom tags to the alert content. Example:
When Go Template renders a template, it fills the template based on data from the context. The namespace
{{$labels.namespace}}and pod{{$labels.pod_name}}are taken from the results of the PromQL query:
To add custom tags and annotations, introduce them using
{{$labels.custom_label_key}}.
Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU utilization{{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, current value{{ printf "%.2f" $value }}%
Tags
Set tags for the alert. The tags can be used as options for notification policy matching rules.
None
Annotations
Set annotations for the alert.
None
Alert Notification
Alert events are sent to the ARMS alert management system by default. You can go to the ARMS alert management platform to configure notification policies to send alert notifications.
Yes
After you are finished, click OK. You can view the status of the new alert rule on the Alert Rules page.
Configure alert notifications
Log on to the ARMS console. In the left-side navigation pane, choose .
On the Notification Policies page, click Create Notification Policy. On the Create Notification Policy page, enter a name for the policy.
In the Match Rules section, set the match rules for alert events.
Set Data Source to ARMS-Prometheus Monitoring.
Set the match rule expression. You can define custom tags or select existing tags, and then select the corresponding values. If a required value is not in the drop-down list, you can enter it directly.
Existing tags include system tags and tags from the metrics in the alert rule.
Specify a Prometheus instance
To specify a Prometheus instance, set
clusternameto the name of the Prometheus instance.
Configure based on metric tags
Notification rules in ARMS can route notifications based on the tags of alert events. By default, the labels used for grouping in the PromQL expression of a Prometheus alert rule become the tags for the resulting alert events.
For example, if the alert expression is:
AliyunEcs_CPUUtilization{}>90Then all labels of the
AliyunEcs_CPUUtilizationmetric are used as tags for the alert event.If the alert expression is:
avg(AliyunEcs_CPUUtilization) by (instanceId,instanceName) > 90Then only the
instanceIdandinstanceNamelabels are used as tags for the alert event.
Based on the tags of alert events, you can configure different notification policies. For example, to send alert events for which
instanceIdisabcdexxxxxxto a specific DingTalk group, configure the rule asinstanceIdequalsabcdexxxxxx.
Click Next.
In the Event Group step, select an option to specify whether to group alert events, and click Next.
If you specify that alert events are not grouped, all alert events are sent to the contacts in an alert notification.
If you specify that alert events are grouped, you must also specify a field. Alert events that have the same field value are grouped. Each group of alert events is sent to contacts in one alert notification.
In the Notification Objects step, configure the following parameters.
Click + Add Notification Object to select contacts.
Contact types:
Contact: When you select a contact, select a notification method, such as by phone call, text message, or email.
Contact group: When you select a contact group, select a notification method, such as by phone call, text message, or email.
Shift: When you select a shift, select a notification method, such as by phone call, text message, or email.
Instant messaging (IM) applications: You can send alert notifications to contacts from IM applications, including DingTalk, Lark, and WeCom.
Webhook: You can use a webhook to send alert notifications.
Specify whether to send clearing notifications after an alert is cleared.
If you choose to send clearing notifications after an alert is cleared, the status of an alert automatically changes to Resolved if all events related to the alert are handled. When an alert is cleared, the system sends clearing notifications to the contacts.
Specify a notification template. For more information, see Configure a notification template and a webhook template.
Specify the period of time during which alert notifications are sent.
(Optional)Select the ticket system to which alert notifications are sent. For more information, see Integration.
Click Next.
In the Policy to Repeat/Escalate/Clear Alerts step, specify whether to send repeated notifications or use an escalation policy, and click Next.
Whether to Resend Notifications: To send repeated notifications, specify the interval at which alert notifications are sent. Alert notifications are continuously sent at the specified interval until the alert is cleared.
Whether to configure an escalation policy for alerts:
If you do not use an escalation policy, an alert notification is sent only once before an alert is cleared.
If you use an escalation policy, an alert notification is sent to other notification objects based on the escalation policy until the alert is cleared.
Whether to enable manual recovery of alerts: After you enable manual recovery, an alert is not automatically cleared if an alert event is not triggered within the automatic recovery time that is set on the Alert Integrations tab of the Integrations page.
In the Action Integration step, specify the automatic action after an alert is triggered or cleared. For more information, see Create an action integration to execute an alerting plan.