Container Service for Kubernetes:Monitor ACK Edge clusters with Managed Service for Prometheus

Step 1: Create a contact

1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Notification Objects.

2. On the Contacts tab, click Create Contact.

3. In the Create Contact dialog box, configure the parameters and click Confirm.

   Parameter	Description
   Name	Specify a custom name for the contact.
   Mobile phone number	After you specify a mobile phone number, the contact can receive alert notifications by phone call and text message. Note Only verified mobile phone numbers can be used for phone call notifications in a notification policy. For information about verifying a mobile phone number, see Verify a phone number.
   Email address	After you specify an email address, the contact can receive alert notifications by email.

   Important

   You can add up to 100 contacts.

Step 2: Create a Prometheus alert rule

Create a Prometheus alert rule by using a static threshold

The static threshold check type provides system-preset alert metrics. By selecting an existing alert metric, you can quickly create an alert rule for that metric by using descriptive options.

1. Log on to the ARMS console.

2. In the navigation pane on the left, choose Prometheus Monitoring > Prometheus Alert Rules.

3. On the Prometheus Alert Rules page, click Create Prometheus Alert Rule.

4. On the Create Prometheus Alert Rule page, configure the following alert parameters and click Save.

   Parameter	Description	Example
   Alert Name	The name of the alert.	prod-cluster-container-cpu-alert
   Check Type	Select Static threshold.	Static threshold
   Prometheus Instance	Select the Prometheus instance for which you want to create the alert.	Production Cluster
   Alert Group	Select an alert group. Different Prometheus types support different alert groups. The available alert group options change based on the selected Prometheus instance type.	Kubernetes Workloads
   Alert Metric	Select the metric for which you want to configure the alert. Each alert group corresponds to different metrics.	Container CPU utilization
   Alert Condition	Set the conditions that trigger an alert event based on the predefined content of the alert metric.	The alert condition is met when container CPU utilization is greater than 80%.
   Filter Condition	Define the scope of the alert rule based on the alert metric. An alert event is triggered when any resource that meets the filter conditions satisfies the alert rule. The following filter conditions are available: Traverse: The alert rule applies to all resources in the current Prometheus instance. Traverse is the default filter condition. Equal To: After you select this condition, enter a specific resource name. The alert rule applies only to that resource. You cannot enter multiple resource names. Not Equal To: After you select this condition, enter a specific resource name. The alert rule applies to all resources except the specified one. You cannot enter multiple resource names. Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule applies to all resources that match the regular expression. Do Not Match Regular Expression: After you select this condition, enter a regular expression to match resource names as needed. The alert rule filters out all resources that match the regular expression. Note After you set the filter conditions, the Data Preview area appears. The filter condition cannot exceed 300 characters.	Traverse
   Data Preview	The Data Preview area displays the Prometheus Query Language (PromQL) statement that corresponds to the alert condition. It also shows the values of the monitoring metric as a time series curve. By default, only the real-time value of one resource is displayed. You can select a target resource and a time range in the filter box of this area to view values for different resources and time ranges. Note The alert threshold appears as a red straight line on the time series curve. The part of the curve that meets the alert threshold is displayed in dark red, and the part that does not is displayed in blue. Hover over the time series curve to view the resource details at a specific point in time. Select a time range on the time series curve to view the curve for that period.	None
   Duration	If an alert condition is met, an alert event is triggered immediately: An alert event is triggered if any data point meets the threshold. An alert event is triggered only after an alert condition persists for N minutes: An alert event is triggered only if the time that the threshold is met is greater than or equal to N minutes.	1
   Alert Level	Customize the alert level. The default alert level is Default. The severity increases from Default, P4, P3, P2, to P1.	Default
   Alert Content	The alert information that users receive. You can use Go template syntax to customize alert parameter variables in the alert content.	Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} CPU utilization {{$labels.metrics_params_opt_label_value}} {{$labels.metrics_params_value}}%, Current value: {{ printf "%.2f" $value }}%
   Alert Notification	Simple Mode: You can set the Notification Receiver, Notification Period, and Whether to Resend Notifications. Standard Mode: Do not specify a notification policy: If you select this option, after you create the alert rule, you can create a new notification policy on the Notification Policy page and specify matching rules and matching conditions, such as the alert rule name, to match the alert rule. When the alert rule is triggered and generates an alert event, the alert information is sent to the contacts or contact groups specified in the notification policy. For more information, see Notification policies. Specify a notification policy: If you select this option, ARMS automatically adds a matching rule to the corresponding notification policy. The matching rule content is the alert rule ID, which is presented as the alert rule name. This ensures that alert events generated by the current alert rule are matched by the selected notification policy. Important Quickly specifying a notification policy only ensures that alert events from the current alert rule are matched by the selected notification policy and that corresponding alerts are generated. However, events from the current alert rule may also be matched by other notification policies that are configured with fuzzy matching, which also generates alerts. The relationship between alert events and notification policies is a many-to-many mapping.	Do not specify a notification rule
   Advanced Settings
   Alert Check Interval	The interval at which the system checks the alert rule to determine if the data meets the alert conditions. The default is 1 minute, and the minimum is 1 minute.	1
   Check after data is complete	Yes No	Yes
   Tags	Set tags for the alert. The tags can be used as options for notification policy matching rules.	None
   Annotations	Set annotations for the alert.	None

Create a Prometheus alert rule by using custom PromQL

To monitor metrics other than the system-preset metrics provided by the static threshold check type, use the custom PromQL check type to create an alert rule.

On the Create Prometheus Alert Rule page, configure the following alert parameters and click Save.

How do I view the ack-arms-prometheus add-on version?

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons .

3. On the Add-ons page, click the Logs and Monitoring tab and find the ack-arms-prometheus add-on.

   The current version appears below the add-on. If a new version is available, click Upgrade on the right of the version to upgrade the add-on.

   Note

   The Upgrade option appears only when the installed add-on version is not the latest version.

ACK Edge cluster: How is monitoring data obtained?

In edge computing, edge nodes are located in relatively isolated on-premises data centers. The cloud-side Virtual Private Cloud (VPC) and the edge side are on different network planes. The Prometheus Agent deployed in the cloud cannot directly access the Node Exporter and GPU Exporter on the edge side through endpoints to obtain the corresponding monitoring metrics. Starting with ack-arms-prometheus version 1.1.4, the ACK Edge cluster's built-in Overview of the cloud-edge O&M communication component Tunnel enables ack-arms-prometheus to automatically establish a monitoring data collection path between the cloud and the edge.

Why does GPU monitoring fail to deploy?

If a GPU node has taints, GPU monitoring may fail to deploy. Use the following steps to view the taints on the GPU node.

1. Run the following command to view the taints on the target GPU node.

   If the GPU node has custom taints, you can find the taint-related entries. This example uses a taint with key set to test-key, value set to test-value, and effect set to NoSchedule:

   kubectl describe node cn-beijing.47.100.***.***

   Expected output:

   Taints:test-key=test-value:NoSchedule

2. Use one of the following two methods to handle the taints on the GPU node.

   • Run the following command to remove the taint from the GPU node.

     kubectl taint node cn-beijing.47.100.***.*** test-key=test-value:NoSchedule-

   • Declare a toleration for the taint on the GPU node to allow pods to be scheduled to the tainted node.

     # 1. Run the following command to edit ack-prometheus-gpu-exporter.
     kubectl edit daemonset -n arms-prom ack-prometheus-gpu-exporter
     
     # 2. Add the following fields to the YAML file to declare a toleration for the taint.
     # Other fields are omitted.
     #tolerations field is added above the containers field, at the same level as the containers field.
     tolerations:
     - key: "test-key"
       operator: "Equal"
       value: "test-value"
       effect: "NoSchedule"
     containers:
     # Other fields are omitted.

Manually deleting resources may cause the reinstallation of Managed Service for Prometheus to fail. How do I completely delete ARMS-Prometheus manually?

Deleting only the namespace of Managed Service for Prometheus leaves residual configuration after the resources are deleted and affects reinstallation. Perform the following operations to completely delete the residual ARMS-Prometheus configuration manually.

• Delete the arms-prom namespace.

  kubectl delete namespace arms-prom

• Delete the ClusterRoles.

  kubectl delete ClusterRole arms-kube-state-metrics
  kubectl delete ClusterRole arms-node-exporter
  kubectl delete ClusterRole arms-prom-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-prometheus-oper3
  kubectl delete ClusterRole arms-prometheus-ack-arms-prometheus-role
  kubectl delete ClusterRole arms-pilot-prom-k8s
  kubectl delete ClusterRole gpu-prometheus-exporter
  kubectl delete ClusterRole o11y:addon-controller:role
  kubectl delete ClusterRole arms-aliyunserviceroleforarms-clusterrole

• Delete the ClusterRoleBindings.

  kubectl delete ClusterRoleBinding arms-node-exporter
  kubectl delete ClusterRoleBinding arms-prom-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding arms-prometheus-oper-bind2
  kubectl delete ClusterRoleBinding arms-kube-state-metrics
  kubectl delete ClusterRoleBinding arms-pilot-prom-k8s
  kubectl delete ClusterRoleBinding arms-prometheus-ack-arms-prometheus-role-binding
  kubectl delete ClusterRoleBinding gpu-prometheus-exporter
  kubectl delete ClusterRoleBinding o11y:addon-controller:rolebinding
  kubectl delete ClusterRoleBinding arms-kube-state-metrics-agent
  kubectl delete ClusterRoleBinding arms-node-exporter-agent
  kubectl delete ClusterRoleBinding arms-aliyunserviceroleforarms-clusterrolebinding

• Delete the Roles and RoleBindings.

  kubectl delete Role arms-pilot-prom-spec-ns-k8s
  kubectl delete Role arms-pilot-prom-spec-ns-k8s -n kube-system
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s
  kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s -n kube-system

After you manually delete the ARMS-Prometheus resources, in ACK console, go to Operations > Add-ons and reinstall the ack-arms-prometheus add-on.