ARMS Prometheus alerting lets you add metric trend charts to the alert cards of a DingTalk group. Viewing these charts provides more information about related alerts, which helps you troubleshoot and locate problems. This topic describes how to configure Prometheus alerts to add metric trend charts to notification cards.
Prerequisites
You have created a DingTalk group. For more information, see DingTalk Robot.
You have configured a notification policy that specifies the DingTalk group as the alert contact. For more information, see Notification policies.
Limits
You can add metric trend charts only to the alert cards of a DingTalk group.
Configure a metric trend chart
Log on to the ARMS console.
In the navigation pane on the left, choose .
On the Prometheus Alert Rules page:click Create Prometheus Alert Rule in the upper-right corner of the page.
On the create or edit page, perform the following operations:
In the Alert Notification section, select Simple Mode and then select the notification policy that uses the DingTalk group as the contact.
Expand Advanced Settings. In the Annotations section, click Create Annotation to add the following two annotations.
Key
Value
Example
_aliyun_display_promql
The PromQL query statement that is used to display the metric trend chart. This statement is usually the PromQL statement for the alert rule with the threshold condition removed. For example, you must change
cpu_usage > 80tocpu_usage. (You can query metric data of up to 512 KB.)_aliyun_display_promql:cpu_usage_aliyun_display_name
The name of the metric in the metric trend chart.
_aliyun_display_name:Container CPU utilizationModify other parameters as needed. For more information about the parameters, see Prometheus alert rules.
Click Save. Then, click Complete.
View the metric trend chart
After the configuration is complete, when an alert is triggered, you can view the metric trend chart on the alert card in the DingTalk client.
For more information about the details on the alert card, see Handle alerts in a notification group.
FAQ
Why do I need to manually configure the PromQL query statement for the chart instead of using the PromQL query statement from the alert rule?
The PromQL query statement for an alert rule is different from the PromQL query statement for the chart. The query statement for an alert rule contains threshold conditions and logical operators such as AND and OR. You must remove these parts from the query statement that is used for the chart.
Why is the chart not displayed after I configure the _aliyun_display_promql annotation?
The chart is rendered only if the labels of the time series returned by the _aliyun_display_promql query are identical to the labels of the time series returned by the alert rule's PromQL statement. A PromQL statement can return multiple time series. The system selects one of the time series based on its labels to render the trend chart. For example, if the metric returned by the
cpu_usage > 80statement contains the pod_name and container labels, the result returned by the _aliyun_display_promql query must also contain these two labels.