Container Service for Kubernetes:Monitor an ACK Edge cluster using Managed Service for Prometheus

Step 1: Create a contact

1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Notification Objects.

2. On the Contacts tab, click Create Contact.

3. In the Create Contact dialog box, configure the parameters and click Confirm.

   Parameter	Description
   Name	The name of the contact.
   Phone Number	Allows the contact to receive alert notifications by phone call and text message. Note Only verified phone numbers can be used for phone call notifications in a notification policy. To learn how to verify a phone number, see Verify a phone number.
   Email	Allows the contact to receive alert notifications by email.

   Important

   You can create up to 100 contacts.

Step 2: Create a Prometheus alert rule

Create a static threshold alert rule

The static threshold check type provides preset metrics. By selecting an existing metric, you can quickly create an alert rule.

1. Log on to the ARMS console.

2. In the navigation pane on the left, choose Prometheus Monitoring > Prometheus Alert Rules.

3. On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.

4. On the Create Prometheus Alert Rule page, configure the alert parameters and click Save.

   Parameter	Description	Example
   Alert Name	The name of the alert.	prod-cluster-container-cpu-alert
   Check Type	Select Static threshold.	Static threshold
   Prometheus Instance	Select the Prometheus instance for which you want to create the alert.	Production Cluster
   Alert Group	Select an alert group. Different Prometheus types support different alert groups. The available alert group options change based on the selected Prometheus instance type.	Kubernetes Workloads
   Alert Metric	Select the metric for which you want to configure the alert. Each alert group corresponds to different metrics.	Container CPU utilization
   Alert Condition	Set the conditions that trigger an alert event based on the predefined content of the alert metric.	The alert condition is met when container CPU utilization is greater than 80%.
   Filter Condition	Define the scope of the alert rule based on the alert metric. An alert event is triggered when any resource that meets the filter conditions satisfies the alert rule. The following filter conditions are available: Traverse: The alert rule applies to all resources in the current Prometheus instance. Traverse is the default filter condition. Equal To: After you select this condition, enter a specific resource name. The alert rule applies only to that resource. You cannot enter multiple resource names. Not Equal To: After you select this condition, enter a specific resource name. The alert rule applies to all resources except the specified one. You cannot enter multiple resource names. Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule applies to all resources that match the regular expression. Do Not Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule filters out all resources that match the regular expression. Note After you set the filter conditions, the Data Preview area appears. The filter condition cannot exceed 300 characters.	Traverse
   Data Preview	The Data Preview area displays the Prometheus Query Language (PromQL) statement that corresponds to the alert condition. It also shows the values of the monitoring metric as a time series curve. By default, only the real-time value of one resource is displayed. You can select a target resource and a time range in the filter box of this area to view values for different resources and time ranges. Note The alert threshold appears as a red straight line on the time series curve. The part of the curve that meets the alert threshold is displayed in dark red, and the part that does not is displayed in blue. Hover over the time series curve to view the resource details at a specific point in time. Select a time range on the time series curve to view the curve for that period.	None
   Duration	If an alert condition is met, an alert event is triggered immediately: An alert event is triggered if any data point meets the threshold. An alert event is triggered only after an alert condition persists for N minutes: An alert event is triggered only if the time that the threshold is met is greater than or equal to N minutes. You cannot configure the duration in seconds. This is a product limitation and is expected behavior.	1
   Alert Level	Customize the alert level. The default alert level is Default. The severity increases from Default, P4, P3, P2, to P1.	Default
   Alert Content	The alert information that users receive. You can use Go template syntax to customize alert parameter variables in the alert content.	Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, Current value: {{ printf "%.2f" $value }}%
   Alert Notification	Simple Mode: You can set the Notification Receiver, Notification Period, and Whether to Resend Notifications. Standard Mode: Do not specify a notification policy: If you select this option, after you create the alert rule, you can create a new notification policy on the Notification Policy page and specify matching rules and matching conditions, such as the alert rule name, to match the alert rule. When the alert rule is triggered and generates an alert event, the alert information is sent to the contacts or contact groups specified in the notification policy. For more information, see Notification policies. Specify a notification policy: If you select this option, ARMS automatically adds a matching rule to the corresponding notification policy. The matching rule content is the alert rule ID, which is presented as the alert rule name. This ensures that alert events generated by the current alert rule are matched by the selected notification policy. Important Quickly specifying a notification policy only ensures that alert events from the current alert rule are matched by the selected notification policy and that corresponding alerts are generated. However, events from the current alert rule may also be matched by other notification policies that are configured with fuzzy matching, which also generates alerts. The relationship between alert events and notification policies is a many-to-many mapping.	Do not specify a notification rule
   Advanced Settings
   Alert Check Interval	The interval at which the system checks the alert rule to determine if the data meets the alert conditions. The default is 1 minute, and the minimum is 1 minute. Even if you enter a value less than 1 minute, such as 15 seconds, the system still performs the check every 1 minute. This is a product limitation and is expected behavior.	1
   Check after data is complete	Yes No	Yes
   Tags	Set tags for the alert. The tags can be used as options for notification policy matching rules.	None
   Annotations	Set annotations for the alert.	None

Create an alert rule with custom PromQL

To monitor metrics not available in the static threshold list, use the custom PromQL check type to create an alert rule.

On the Create Prometheus Alert Rule page, configure the following alert parameters and click Save.

How do I check the ack-arms-prometheus add-on version?

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons .

3. On the Add-ons page, click the Logs and Monitoring tab and find the ack-arms-prometheus add-on.

   The current version is displayed on the component card. If a new version is available, click Upgrade to update the component.

   Note

   The Upgrade button is displayed only if the installed component is not the latest version.

ACK Edge cluster: How does it obtain monitoring data?

In edge computing scenarios, edge nodes are typically located in relatively isolated, on-premises data center environments. This means the cloud-based VPC and the edge nodes operate in separate networks. The Prometheus Agent deployed in the cloud cannot directly access the endpoints of edge-side components like Node Exporter and GPU Exporter to collect metrics. Starting from ack-arms-prometheus version 1.1.4, the built-in cloud-native O&M communication component Tunnel in ACK Edge clusters allows ack-arms-prometheus to automatically establish a data collection channel between the cloud and the edge.

Why does the GPU monitoring deployment fail?

GPU monitoring deployment can fail if the GPU node has taints. To resolve this issue, first check the node's taints.

1. Run the following command to check the taints of the target GPU node.

   If the GPU node has custom taints, you can find the related entries. This example uses a taint with a key of test-key, a value of test-value, and an effect of NoSchedule:

   kubectl describe node cn-beijing.47.100.***.***

   Expected output:

   Taints:test-key=test-value:NoSchedule

2. Handle the GPU node taints in one of the following two ways:

   • Run the following command to remove the taint from the GPU node.

     kubectl taint node cn-beijing.47.100.***.*** test-key=test-value:NoSchedule-

   • Declare a toleration for the taint to allow pods to be scheduled to the node.

     # 1. Run the following command to edit the ack-prometheus-gpu-exporter DaemonSet.
     kubectl edit daemonset -n arms-prom ack-prometheus-gpu-exporter
     
     # 2. Add the following fields to the YAML file to declare the toleration for the taint.
     # Other fields are omitted.
     # The `tolerations` field is added above the `containers` field and at the same level.
     tolerations:
     - key: "test-key"
       operator: "Equal"
       value: "test-value"
       effect: "NoSchedule"
     containers:
     # Other fields are omitted.

How to completely remove ARMS-Prometheus resources

Deleting only the namespace of Managed Service for Prometheus leaves residual configurations after the resources are deleted. This affects reinstallation. You can perform the following operations to completely and manually delete the residual ARMS-Prometheus configurations.

• Delete the arms-prom namespace.

  kubectl delete namespace arms-prom

• Delete the ClusterRoles.

  kubectl delete ClusterRole arms-kube-state-metrics
  kubectl delete ClusterRole arms-node-exporter
  kubectl delete ClusterRole arms-prom-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-prometheus-oper3
  kubectl delete ClusterRole arms-prometheus-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-pilot-prom-k8s
  kubectl delete ClusterRole gpu-prometheus-exporter
  kubectl delete ClusterRole o11y:addon-controller:role
  kubectl delete ClusterRole arms-aliyunserviceroleforarms-clusterrole

• Delete the ClusterRoleBindings.

  kubectl delete ClusterRoleBinding arms-node-exporter
  kubectl delete ClusterRoleBinding arms-prom-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding arms-prometheus-oper-bind2
  kubectl delete ClusterRoleBinding arms-kube-state-metrics
  kubectl delete ClusterRoleBinding arms-pilot-prom-k8s
  kubectl delete ClusterRoleBinding arms-prometheus-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding gpu-prometheus-exporter
  kubectl delete ClusterRoleBinding o11y:addon-controller:rolebinding
  kubectl delete ClusterRoleBinding arms-kube-state-metrics-agent
  kubectl delete ClusterRoleBinding arms-node-exporter-agent
  kubectl delete ClusterRoleBinding arms-aliyunserviceroleforarms-clusterrolebinding

• Delete the Roles and RoleBindings.

  kubectl delete Role arms-pilot-prom-spec-ns-k8s
  kubectl delete Role arms-pilot-prom-spec-ns-k8s -n kube-system
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s -n kube-system

After you delete the resources, return to the ACK console, choose Operations > Add-ons, and reinstall the ack-arms-prometheus add-on.