Container Service for Kubernetes:Set up alerting for a registered cluster

Using onectl

1. Install and configure onectl on your local machine. For more information, see Manage registered clusters using onectl.

2. Grant RAM permissions to the Cloud Monitor component.

   onectl ram-user grant --addon alicloud-monitor-controller

   Expected output:

   Ram policy ack-one-registered-cluster-policy-alicloud-monitor-controller granted to ram user ack-one-user-ce313528c3 successfully.

Using the console

Before installing an add-on in a registered cluster, create a RAM user with the required permissions and obtain an AccessKey pair to access Alibaba Cloud services.

1. Create a RAM user.

2. Create a custom policy. Example:

   {
               "Action": [
                   "log:*",
                   "arms:*",
                   "cms:*",
                   "cs:UpdateContactGroup"
               ],
               "Resource": [
                   "*"
               ],
               "Effect": "Allow"
   }

3. Manage RAM user permissions.

4. Create an AccessKey for the RAM user.

   Warning

   For enhanced security, configure a network access control policy for the AccessKey to restrict access to trusted network environments and improve security. For more information, see AccessKey-based network access restriction policies.

5. Use the AccessKey pair to create a Secret named alibaba-addon-secret in the registered cluster.

   The Cloud Monitor component uses this AccessKey pair to access cloud resources.

   kubectl -n kube-system create secret generic alibaba-addon-secret --from-literal='access-key-id=<your access key id>' --from-literal='access-key-secret=<your access key secret>'

   Note

   Replace <your access key id> and <your access key secret> with your AccessKey pair.

Using onectl

Install the Cloud Monitor component.

onectl addon install alicloud-monitor-controller

Expected output:

Addon alicloud-monitor-controller, version **** installed.

Using the console

The console checks alerting requirements and guides you to activate, install, or upgrade the component.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Alerts.

3. On the Alerts page, follow the on-screen instructions to install or upgrade the required components.

4. After installation or upgrade, go to the Alerts page to configure alerts.

   Tab	Description
   Alert Rules	Status: Enable or disable an alert rule set. Edit Contact Group: Set the contact group for alert notifications. Notifications are sent to contact groups only. Create contacts and groups first. To notify an individual, create a dedicated group for that contact.
   Alert History	View up to 100 alert records from the last 24 hours. Click a link in the Alert Rule column to view rule configurations in the corresponding monitoring system. Click Details to navigate to the anomaly-related resource page. Click Intelligent Analytics for AI-powered issue analysis and troubleshooting.
   Alert Contacts	Create, edit, or delete contacts. Contact methods: Phone call/SMS: Set a mobile number for a contact to receive alerts by phone and SMS. Only verified mobile numbers can receive phone call notifications. See Verify a mobile phone number. Email: Set an email address for a contact to receive alert notifications. Chatbots: DingTalk chatbots, WeCom chatbots, and Lark chatbots. For DingTalk chatbots, add security keywords: alert, dispatch. Verify email and chatbot notifications in the CloudMonitor console under Alerts > Alert Contacts before configuring them.
   Alert Contact Groups	Create, edit, or delete contact groups. If no contact group exists, the console creates a default group from your Alibaba Cloud account.

Console

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Operations > Alerts.

3. On the Alert Rules tab, click Configure Alert Rule in the upper-right corner. Then, in the Actions column of the target rule, click YAML to view the AckAlertRule CRD resource configuration.

4. Modify the YAML file. See Default alert rule template for parameter details.

   The following example shows the YAML configuration of an alert rule:

   Alert rule YAML configuration

   apiVersion: alert.alibabacloud.com/v1beta1
   kind: AckAlertRule
   metadata:
     name: default
   spec:
     groups:
       # The following is a sample configuration for a cluster event alert rule.
       - name: pod-exceptions                             # The name of the alert rule group, which corresponds to the Group_Name field in the alert template.
         rules:
           - name: pod-oom                                # The name of the alert rule.
             type: event                                  # The type of the alert rule (Rule_Type). Valid values: event, metric-cms (CloudMonitor metric), and metric-prometheus (Prometheus metric).
             expression: sls.app.ack.pod.oom              # The alert rule expression. If the rule type is event, the value is the Rule_Expression_Id from the default alert rule template in this topic.
             enable: enable                               # The state of the alert rule. Valid values: enable and disable.
           - name: pod-failed
             type: event
             expression: sls.app.ack.pod.failed
             enable: enable
       # The following is a sample configuration for a cluster infrastructure resource alert rule.
       - name: res-exceptions                              # The name of the alert rule group, which corresponds to the Group_Name field in the alert template.
         rules:
           - name: node_cpu_util_high                      # The name of the alert rule.
             type: metric-cms                              # The type of the alert rule (Rule_Type). Valid values: event, metric-cms (CloudMonitor metric), and metric-prometheus (Prometheus metric).
             expression: cms.host.cpu.utilization          # The alert rule expression. If the rule type is metric-cms, the value is the Rule_Expression_Id from the default alert rule template in this topic.
             contactGroups:                                # The contact groups mapped to the alert rule. This configuration is generated by the ACK console. Contacts are consistent for a single account and can be reused across multiple clusters.
               - arms_contact_group_id_v2: '69xxx'
                 cms_contact_group_name: xxx Contact Group
                 id: '10xxx'
             enable: enable                                # The state of the alert rule. Valid values: enable and disable.
             thresholds:                                   # The threshold of the alert rule.
               - key: CMS_ESCALATIONS_CRITICAL_Threshold
                 unit: percent
                 value: '85'                                # CPU utilization threshold. Default value: 85%.
               - key: CMS_ESCALATIONS_CRITICAL_Times
                 value: '3'                                # An alert is triggered if the threshold is exceeded 3 consecutive times.
               - key: CMS_RULE_SILENCE_SEC                 # The silence period after the first alert is reported.
                 value: '900'    

   Use rules.thresholds to customize alert thresholds. For example, the preceding configuration triggers an alert when a node's CPU utilization exceeds 85% three consecutive times and more than 900 seconds have passed since the last alert.

   Parameter	Required	Description	Default
   CMS_ESCALATIONS_CRITICAL_Threshold	Yes	The threshold for the alert rule. If this parameter is omitted, rule synchronization fails and the rule is disabled. unit: The unit of the threshold. Valid values: percent, count, and qps. value: The threshold value.	Varies based on the default alert rule template.
   CMS_ESCALATIONS_CRITICAL_Times	Optional	The number of consecutive times the condition must be met before CloudMonitor triggers an alert. If this parameter is omitted, the default value is used.	3
   CMS_RULE_SILENCE_SEC	Optional	The silence period in seconds after an initial alert is reported for a continuously triggering CloudMonitor rule. This prevents alert fatigue. If this parameter is omitted, the default value is used.	900

CLI

1. Edit the alert rule YAML file:

   kubectl edit ackalertrules default -n kube-system

2. Modify the YAML file, then save and exit. See Default alert rule template for parameter details.

   Alert rule YAML configuration

   apiVersion: alert.alibabacloud.com/v1beta1
   kind: AckAlertRule
   metadata:
     name: default
   spec:
     groups:
       # The following is a sample configuration for a cluster event alert rule.
       - name: pod-exceptions                             # The name of the alert rule group, which corresponds to the Group_Name field in the alert template.
         rules:
           - name: pod-oom                                # The name of the alert rule.
             type: event                                  # The type of the alert rule (Rule_Type). Valid values: event, metric-cms (CloudMonitor metric), and metric-prometheus (Prometheus metric).
             expression: sls.app.ack.pod.oom              # The alert rule expression. If the rule type is event, the value is the Rule_Expression_Id from the default alert rule template in this topic.
             enable: enable                               # The state of the alert rule. Valid values: enable and disable.
           - name: pod-failed
             type: event
             expression: sls.app.ack.pod.failed
             enable: enable
       # The following is a sample configuration for a cluster infrastructure resource alert rule.
       - name: res-exceptions                              # The name of the alert rule group, which corresponds to the Group_Name field in the alert template.
         rules:
           - name: node_cpu_util_high                      # The name of the alert rule.
             type: metric-cms                              # The type of the alert rule (Rule_Type). Valid values: event, metric-cms (CloudMonitor metric), and metric-prometheus (Prometheus metric).
             expression: cms.host.cpu.utilization          # The alert rule expression. If the rule type is metric-cms, the value is the Rule_Expression_Id from the default alert rule template in this topic.
             contactGroups:                                # The contact groups mapped to the alert rule. This configuration is generated by the ACK console. Contacts are consistent for a single account and can be reused across multiple clusters.
               - arms_contact_group_id_v2: 'xxx'
                 cms_contact_group_name: xxx
                 id: 'xxx'
             enable: enable                                # The state of the alert rule. Valid values: enable and disable.
             thresholds:                                   # The threshold of the alert rule.
               - key: CMS_ESCALATIONS_CRITICAL_Threshold
                 unit: percent
                 value: '85'                                # CPU utilization threshold. Default value: 85%.
               - key: CMS_ESCALATIONS_CRITICAL_Times
                 value: '3'                                # An alert is triggered if the threshold is exceeded 3 consecutive times.
               - key: CMS_RULE_SILENCE_SEC                 # The silence period after the first alert is reported.
                 value: '900'    

   Use rules.thresholds to customize alert thresholds. For example, the preceding configuration triggers an alert when a node's CPU utilization exceeds 85% three consecutive times and more than 900 seconds have passed since the last alert.

   Parameter	Required	Description	Default
   CMS_ESCALATIONS_CRITICAL_Threshold	Yes	The threshold for the alert rule. If this parameter is omitted, rule synchronization fails and the rule is disabled. unit: The unit of the threshold. Valid values: percent, count, and qps. value: The threshold value.	Varies based on the default alert rule template.
   CMS_ESCALATIONS_CRITICAL_Times	Optional	The number of consecutive times the condition must be met before CloudMonitor triggers an alert. If this parameter is omitted, the default value is used.	3
   CMS_RULE_SILENCE_SEC	Optional	The silence period in seconds after an initial alert is reported for a continuously triggering CloudMonitor rule. This prevents alert fatigue. If this parameter is omitted, the default value is used.	900