When a Prometheus alert fires, responders often switch to the ARMS console to check the metric trend. Adding a metric trend chart to the DingTalk alert card gives responders immediate visual context directly in the notification, without leaving DingTalk.
Two custom annotations on the alert rule control which metric to plot and what name to display on the chart.
Metric trend charts are supported only on DingTalk group alert cards. Other notification channels (email, webhooks) do not support this feature.
How it works
A Prometheus alert rule evaluates a PromQL expression with a threshold condition, such as cpu_usage > 80. To render a trend chart, ARMS needs a separate PromQL expression -- the same query without the threshold -- to plot the raw metric values over time.
You provide this query and a display name through two annotations:
| Annotation key | Purpose | Example value |
|---|---|---|
_aliyun_display_promql | PromQL expression that returns the metric data to plot. Remove any threshold conditions or logical operators (AND, OR) from the alert expression. | cpu_usage |
_aliyun_display_name | Display name for the metric shown on the chart. | Container CPU utilization |
The chart renders only when the labels in the _aliyun_display_promql query result match the labels in the alert rule's PromQL result. For details, see Why is the chart missing?
The _aliyun_display_promql query can return up to 512 KB of metric data.Prerequisites
Before you begin, make sure you have:
A DingTalk group configured as a notification channel. See DingTalk chatbot
A notification policy that routes alerts to the DingTalk group. See Notification policies
Create or edit an alert rule with a trend chart
Log on to the ARMS console.
In the left-side navigation pane, choose Managed Service for Prometheus > Prometheus Alert Rules.
On the Prometheus Alert Rules page, click Create Prometheus Alert Rule in the upper-right corner.
In the Alert Notification section, select Simple Mode, then select the notification policy that uses your DingTalk group as the contact.
Expand Advanced Settings, find the Annotations section, and click Create Annotation. Add the following two annotations:
Set the key to
_aliyun_display_promqland the value to your PromQL expression with the threshold removed. For example, if the alert rule usescpu_usage > 80, set the value tocpu_usage.Click Create Annotation again. Set the key to
_aliyun_display_nameand the value to a descriptive metric name, such asContainer CPU utilization.
Configure other alert rule parameters as needed. See Prometheus alert rules.
Click Save, then click Complete.
Verify the chart
After the alert rule triggers, open the DingTalk group and check the alert card. The metric trend chart appears inline, showing the metric values leading up to the alert.
To learn how to act on alerts from DingTalk, see Handle alerts in a notification group.
FAQ
Why do I need a separate PromQL expression for the chart?
An alert rule's PromQL expression includes threshold conditions and logical operators (for example, cpu_usage > 80). These filter the result set rather than returning the raw metric values needed for a chart. The _aliyun_display_promql annotation takes the same query minus the threshold, so ARMS can retrieve the underlying time-series data and plot it.
Why is the chart missing after I set the annotations?
The chart renders only when the labels in the _aliyun_display_promql query result match the labels in the alert rule's PromQL result. A PromQL expression can return multiple time series, and ARMS selects the matching one based on its labels.
For example, if cpu_usage > 80 returns time series with pod_name and container labels, the _aliyun_display_promql query must also return time series that include both of these labels. If the labels differ, ARMS cannot match the results, and the chart does not render.
To troubleshoot:
Run the
_aliyun_display_promqlquery and the alert rule's PromQL expression separately in the Prometheus query editor.Compare the label sets of the returned time series.
Adjust the
_aliyun_display_promqlexpression so that its result labels match those of the alert rule.