Container Service for Kubernetes:Container Service alert management

Step 1: Grant permissions to the worker RAM role

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the cluster and click its name. In the left-side pane, click Cluster Information.

3. In the Cluster Resources section, copy the name of the Worker RAM Role and click the link to open the Resource Access Management (RAM) console.

4. Create a custom policy with the following permissions. For more information, see Create a custom policy on the JSON tab.

   This policy grants broad permissions for simplicity. In a production environment, follow the principle of least privilege and grant only the required permissions.

      {
          "Action": [
              "log:*",
              "arms:*",
              "cms:*",
              "cs:UpdateContactGroup"
          ],
          "Resource": [
              "*"
          ],
          "Effect": "Allow"
      }

5. Attach the custom policy to the worker RAM role. For more information, see Grant permissions to a RAM role.

Step 2: Verify permissions

1. In the left-side pane of the cluster management page, choose Workloads > Deployments.

2. Set Namespace to kube-system and click the name of the alicloud-monitor-controller application.

3. Click the Logs tab. The pod logs indicate that the authorization was successful.

Step 3: Enable default alert rules

1. In the left-side pane, choose Operations > Alerts.

2. On the Alerts page, configure alert rules and contacts. For details on each tab, see Manage alerts in the console.

Alert Rules

• Status: Toggle alert rule sets on or off.

• Modify Contacts: Assign a contact group for alert notifications.

Only contact groups can be selected as notification targets. To notify a single person, create a group containing only that contact. Create contacts and contact groups before you configure alert rules.

Alert History

View the most recent 100 alert records from the last 24 hours.

• Click the link in the Alert Rule column to view the detailed rule configuration in the corresponding monitoring system.

• Click Details to locate the resource where the anomaly occurred.

• Click Intelligent Analytics for AI-assisted root cause analysis and troubleshooting guidance.

Alert Contacts

Create, edit, or delete alert contacts. Supported notification methods:

For DingTalk robots, add the security keywords: Alerting, Dispatch.

Before you configure email and robot notifications, verify them in the CloudMonitor console under Alert Service > Alert Contacts.

Alert Contact Groups

Create, edit, or delete contact groups. Contact groups are the only selectable notification targets when you Modify Contacts for an alert rule set.

If no contact group exists, the console creates a default group based on your Alibaba Cloud account information.

Threshold parameters

Threshold parameters apply to metric-cms type rules:

Pod eviction triggered by disk pressure

Alert message:

(combined from similar events): Failed to garbage collect required amount of images.
Attempted to free XXXX bytes, but only found 0 bytes eligible to free

Symptoms: The pod status is Evicted. The node reports disk pressure (The node had condition: [DiskPressure].).

Cause: When node disk usage reaches the eviction threshold (default: 85%), the kubelet performs pressure-based eviction and garbage collection to reclaim unused image files. Run df -h on the target node to check disk usage.

Solution:

1. Log on to the target node (containerd runtime) and remove unused container images:

      crictl rmi --prune

2. Clean up logs or resize the node disk:

   • Create a snapshot backup of the data disk or system disk, then delete files or folders that are no longer needed. For more information, see Resolve full disk space issues on Linux instances.

   • Scale out the system disk or data disk of the target node to increase storage capacity.

3. Adjust thresholds:

   • Adjust the kubelet image garbage collection threshold to reduce pod evictions. See Customize kubelet configurations for a node pool.

   • Modify the alert threshold in the node_disk_util_high alert rule in the YAML configuration. See Customize alert rules.

Best practices:

• For nodes that frequently encounter this issue, assess the actual storage needs and plan resource requests and node disk capacity accordingly.

• Regularly monitor storage usage with the Node Storage Dashboard to identify and address potential issues early.

Pod OOMKilling

Alert message:

pod was OOM killed. node:xxx pod:xxx namespace:xxx uuid:xxx

Symptoms: The pod status is abnormal, and the event details contain PodOOMKilling.

Cause: OOM events can occur at two levels:

• Container cgroup-level OOM: The actual memory usage of a pod exceeds its memory limits. The Kubernetes cgroup forcibly terminates the pod.

• Node-level OOM: Too many pods without resource limits (requests/limits) are running on a node, or non-Kubernetes processes consume excessive memory.

Diagnosis: Log on to the target node and run:

dmesg -T | grep -i "memory"

If the output contains out_of_memory, an OOM event occurred. If the log also contains Memory cgroup, it is a container cgroup-level OOM. Otherwise, it is a node-level OOM.

Solution:

• Container cgroup-level OOM:

  • Increase the pod's memory limits. Keep actual usage below 80% of the specified limits. See Manage pods and Upgrade or downgrade node resources.

  • Enable resource profiling to get recommended configurations for container requests and limits.

• Node-level OOM:

  • Scale out the memory resources of the node or distribute workloads across more nodes. See Upgrade or downgrade node resources and Schedule applications to specific nodes.

  • Identify pods with high memory usage on the node and set reasonable memory limits.

For more information, see Causes and solutions for OOM Killer.

Pod CrashLoopBackOff

When a process in a pod exits unexpectedly, ACK tries to restart the pod. If the pod fails to reach the desired state after multiple restarts, its status changes to CrashLoopBackOff.

Diagnosis:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster and click its name. In the left-side pane, choose Workloads > Pods.

3. Find the abnormal pod and click View Details in the Actions column.

4. Check the Events section for abnormal event descriptions.

5. Click the Logs tab to view process-level errors.

   If the pod has been restarted, select Show the log of the last container exit to view the logs of the previous container.

   The console displays a maximum of 500 recent log entries. To view more historical logs, set up a log persistence solution for unified collection and storage.