Use Kubernetes labels to dispatch alerts - Application Real-Time Monitoring Service

This topic describes how to dispatch alerts based on labels attached to pods. In the following example, alerts are generated when the CPU utilization of pods exceeds a predefined threshold.

Background information

Kubernetes allows you to classify and manage resources based on labels. You can attach different labels to applications, such as env:prod and service:iam. The env:prod label indicates that the application is in the production environment, and the service:iam label indicates that the application provides the Identity and Access Management (IAM) service. You can use labels to manage applications. You can use labels to manage applications. For example, you can use the env:prod label to identify applications in the production environment, or use the service:iam label to check resource usage of the IAM service.

The Alert Management sub-service of Application Real-Time Monitoring Service (ARMS) allows you to attach Kubernetes labels to alerts based on Prometheus data sources. When alerts are generated for pods, you can attach the labels of the pods to the alerts, and use notification policies to dispatch alerts based on the labels.

Working mechanism

kube-state-metrics in Prometheus generates metrics about the labels of each Kubernetes resource, such as kube_pod_labels and kube_node_labels.
You can configure an annotation for an alert rule. In this example, the key of the annotation is _aliyun_arms_enrich_desc, and the value of the annotation is an executable Prometheus Query Language (PromQL) statement. When an alert is generated for a pod, ARMS executes the PromQL statement to query the labels of the pod, and then attaches the labels to the alert.
The PromQL statement that you configured as the annotation value can be used to query the label metrics provided by kube-state-metrics for the pod. This way, ARMS can obtain the pod labels and attach the labels to the alert.

Procedure

Create a Prometheus instance to monitor a Container Service for Kubernetes (ACK) cluster. For more information, see Create a Prometheus instance to monitor an ACK cluster.
Add a label to an application in the ACK cluster.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters. On the Clusters page, find the cluster where the application you want to manage resides and click Applications in the Actions column.
2. On the Deployments page, choose More > View in YAML in the Actions column of the application that you want to manage.
3. On the View in YAML page, add the custom label app:arms-prom-ack-arms-prometheus.
4. Click Update.
Create a Prometheus alert rule.
1. Log on to the ARMS console. In the left-side navigation pane, choose Managed Service for Prometheus > Prometheus Alert Rules.
2. In the upper-right corner of the Prometheus Alert Rules page, click Create Prometheus Alert Rule.
3. On the Create Prometheus Alert Rule page, set Alert Name to EnrichDemo and Alert Condition to When Container CPU Usage Greater than 2%, Meet the alarm conditions.
4. Configure other parameters and click Save. For more information about how to configure the parameters, see Create an alert rule for a Prometheus instance.
Add a label to an alert.
On the Create Prometheus Alert Rule page, click the arrow next to Advanced Settings to add an annotation.
Description:
- Key: _aliyun_arms_enrich_desc.
- Value: an executable PromQL statement. You can use ${xxx} to reference labels of the Kubernetes resource. kube_pod_labels{pod_name=${pod_name}} is used in the example, and ${pod_name} is used to reference a pod name.
Important
- The value of an annotation must be an executable PromQL statement.
- The query results of a PromQL statement can contain only one time series.
- All labels that are returned after a PromQL statement is executed are attached to the alert.
- kube_pod_labels adds the label_ prefix to the keys of all labels. Therefore, you must manually add this prefix to the label key that you configure as a condition in the Match Alert Event Rules section of the Notification Policies page.
- If more than 10 timelines are queried, you can only enrich tags for the first 10 timelines.
Configure a notification policy.
In the Match Alert Event Rules section of the Notification Policies page, set the condition to label_app Equal To arms-prom-ack-arms-prometheus. This way, you can identify alerts based on their labels. For more information about how to create a notification policy, see Notification policies.
On the Alert Sending History page, view the alerts with the specified label. For more information, see View historical alerts.