All Products
Search

This Product

    Container Service for Kubernetes:Configure alert rules with Prometheus

    Document Center

    Container Service for Kubernetes:Configure alert rules with Prometheus

    Last Updated:Jun 15, 2026

    Write custom PromQL alert rules to monitor cluster nodes, hosts, container replicas, and workloads.

    Prerequisites

    Before you begin, ensure that you have:

    Configure alert rules with custom PromQL

    Both Alibaba Cloud Prometheus and open source Prometheus support custom PromQL-based alert rules that trigger notifications when conditions are met.

    Alibaba Cloud Prometheus

    To create a Prometheus alert rule using custom PromQL, see Create a Prometheus alert rule.

    Open source Prometheus

    1. Configure an alert notification policy. Open source Prometheus supports webhooks, DingTalk, and email. Set the notification method via the receiver parameter in ack-prometheus-operator. See Alerting configuration.

    2. Create an alert rule. Deploy a PrometheusRule CRD in the cluster to define alert rules (see Deploying Prometheus rules). The following example fires when node CPU usage exceeds 90% over a 2-minute window. The expr field specifies the PromQL expression and trigger condition.

      apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    # Labels must match ruleSelector.matchLabels in the Prometheus CRD.
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: example.rules
    rules:
    - alert: ExampleAlert
      # expr: PromQL query and trigger condition.
      # Refer to the PromQL configuration column in the alert rule tables below.
      expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 90

    3. Verify that the alert rule is active.

      1. Run the following command to forward the Prometheus service to port 9090 on your local machine: kubectl port-forward svc/ack-prometheus-operator-prometheus 9090:9090 -n monitoring

      2. Open localhost:9090 in your browser.

      3. Choose Status > Rules. If the alert rule appears on the Rules page, it is active.

    Alert rule reference

    ACK provides recommended alert rules based on O&M experience, covering cluster stability, node and host resource usage, container replicas, workloads, storage, and networking.

    Severity levels:

    • Critical: Affects cluster, application, or business. Requires immediate action.

    • Warning: Affects cluster, application, or business. Investigate promptly.

    • Normal: Indicates an important feature change.

    The Rule description column uses the Alert Rules tab on the Alerts page as entry point. To access: log on to the ACK console, click your cluster in the Clusters list, then choose Operations > Alerts > Alert Rules.

    Abnormal container replicas

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    Abnormal pod status Critical min_over_time(sum by (namespace, pod, phase) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"})[5m:1m]) > 0 > 0 5 min Fires when a pod stays in Pending, Unknown, or Failed status for 5 minutes. Configure via Alert Rule Set for Pod Exceptions > Pod anomaly. See Manage alerts in ACK. See Troubleshoot pod exceptions.
    Pod startup failed Critical sum_over_time(increase(kube_pod_container_status_restarts_total{}[1m])[5m:1m]) > 3 > 3 restarts 5 min Fires when a pod restarts more than 3 times in 5 minutes. Configure via Alert Rule Set for Pod Exceptions > Pod startup failures. See Manage alerts in ACK. See Troubleshoot pod exceptions.
    More than 1,000 pods failed to be scheduled for 10 continuous minutes Critical count((min_over_time(kube_pod_status_phase{phase="Pending"}[10m]) == 1) and (count_over_time(kube_pod_status_phase{phase="Pending"}[10m:15s]) >= (10 * 4 - 1))) > 1000 > 1,000 pods 10 min Fires when over 1,000 pods remain in the Pending state due to scheduling failures for more than 10 continuous minutes. May indicate excessive scheduling pressure. ACK Pro Edition provides enhanced scheduling and an SLA. See Overview of ACK managed cluster Pro Edition.
    Frequent container CPU throttling Warning rate(container_cpu_cfs_throttled_seconds_total[3m]) * 100 > 25 > 25% throttled time 3 min Fires when throttled CPU time exceeds 25% over 3 minutes. Throttling reduces time slices, increasing process runtime and slowing application logic. Check if the pod CPU resource limit is too low. Reduce throttling with the CPU Burst policy. On multi-core nodes, use CPU topology-aware scheduling to maximize fragmented CPU resources.
    Pod CPU usage > 85% of limit Warning (sum(irate(container_cpu_usage_seconds_total{pod=~"{{PodName}}.*",namespace=~"{{Namespace}}.*",container!="",container!="POD"}[1m])) by (namespace,pod) / sum(container_spec_cpu_quota{pod=~"{{PodName}}.*",namespace=~"{{Namespace}}.*",container!="",container!="POD"}/100000) by (namespace,pod) * 100 <= 100 or on() vector(0)) >= 85 >= 85% of pod limit 1 min Fires when pod CPU exceeds 85% of its limit. No effect without a configured limit. Default threshold: 85% — adjust as needed. To filter, replace pod=~"{{PodName}}.*",namespace=~"{{Namespace}}.*" with actual values; remove the filter to query all pods. High CPU usage causes throttling and reduces time slices. Check if the CPU resource limit is too low. See Enable the CPU Burst policy and Enable CPU topology-aware scheduling.
    Pod memory usage > 85% of limit Warning ((sum(container_memory_working_set_bytes{pod=~"{{PodName}}.*",namespace=~"{{Namespace}}.*",container!="",container!="POD"}) by (pod,namespace) / sum(container_spec_memory_limit_bytes{pod=~"{{PodName}}.*",namespace=~"{{Namespace}}.*",container!="",container!="POD"}) by (pod, namespace) * 100) <= 100 or on() vector(0)) >= 85 >= 85% of pod limit — Fires when pod memory exceeds 85% of its limit. No effect without a configured limit. Default threshold: 85% — adjust as needed. High memory usage can trigger OOM kills and pod restarts. Check if the memory resource limit is too low. Right-size limits with Resource profiling.

    Abnormal workloads

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    Deployment replica mismatch Critical kube_deployment_spec_replicas{} != kube_deployment_status_replicas_available{} Any mismatch — Fires when available Deployment replicas do not match the desired count. Configure via Alert Rule Set for Workload Exceptions > Deployment pod anomaly. See Manage alerts in ACK. See Troubleshoot pod exceptions.
    DaemonSet replica mismatch Critical ((100 - kube_daemonset_status_number_ready{} / kube_daemonset_status_desired_number_scheduled{} * 100) or (kube_daemonset_status_desired_number_scheduled{} - kube_daemonset_status_current_number_scheduled{})) > 0 > 0 — Fires when available DaemonSet replicas do not match the desired count. Configure via Alert Rule Set for Workload Exceptions > DaemonSet pod anomaly. See Manage alerts in ACK. See Troubleshoot pod exceptions.
    DaemonSet scheduling error Critical kube_daemonset_status_number_misscheduled{job} > 0 > 0 — Fires when a DaemonSet replica runs on a wrong node. Configure via Alert Rule Set for Workload Exceptions > DaemonSet pod scheduling errors. See Manage alerts in ACK. See Troubleshoot pod exceptions.
    Job failed Critical kube_job_status_failed{} > 0 > 0 — Fires when a Job fails. Configure via Alert Rule Set for Workload Exceptions > Job execution failures. See Manage alerts in ACK. Check failed pod logs for error details. See Troubleshoot pod exceptions.

    Storage exceptions

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    PersistentVolume (PV) status abnormal Critical kube_persistentvolume_status_phase{phase=~"Failed|Pending"} > 0 > 0 — Fires when a PV enters Failed or Pending state. Configure via Alert Rule Set for Storage Exceptions > PV anomaly. See Manage alerts in ACK. See the disk mounting section in FAQ about disk PVs.
    Host disk usage > 85% Critical (100 - node_filesystem_avail_bytes / node_filesystem_size_bytes * 100) >= 85 >= 85% — Fires when node disk free space is less than 15%. Configure via Alert Rule Set for Resource Exceptions > Node - Disk usage >= 85%. See Manage alerts in ACK. Scale out the node or expand its disk. See FAQ about disk PVs.

    Abnormal node status

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    Node NotReady for 3 minutes Critical (sum(max_over_time(kube_node_status_condition{condition="Ready",status="true"}[3m]) <= 0) by (node)) or (absent(kube_node_status_condition{condition="Ready",status="true"})) > 0 > 0 3 min Fires when a node stays NotReady for 3 minutes. Configure via Alert Rule Set for Node Exceptions > Node changes to the unschedulable state. See Manage alerts in ACK. Determine if NotReady is expected (for example, node replacement or maintenance). If unexpected, check if application pods are affected and evict if necessary. Check node conditions for causes such as memory pressure or full disk.

    Abnormal host resource usage

    Host resource metrics measure physical or virtual machine resources. Usage = all processes' resource usage / maximum host capacity.
    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    Host memory usage > 85% Warning (100 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) >= 85 >= 85% — Fires when host memory usage exceeds 85%. Configure via Alert Rule Set for Resource Exceptions > Node - Memory usage >= 85% . See Manage alerts in ACK . Default threshold: 85% — adjust as needed.
    Note

    ACK alert rules are provided by Cloud Monitor, with metrics consistent with Prometheus rules.

    		 Release resources: check pod resource allocation with cost analysis, and right-size memory requests with resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Host memory usage > 90% Critical (100 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) >= 90 >= 90% — Fires when host memory usage exceeds 90%. Release resources with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Host CPU usage > 85% Warning 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) >= 85 >= 85% 2 min Fires when host CPU usage exceeds 85%. Configure via Alert Rule Set for Resource Exceptions > Node - CPU usage >= 85% .
    Note

    ACK uses CloudMonitor ECS metrics, equivalent to this Prometheus rule. Default threshold: 85% — adjust as needed. See Manage alerts in ACK.

    		 Release resources with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Host CPU usage > 90% Critical 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) >= 90 >= 90% 2 min Fires when host CPU usage exceeds 90%. Release resources with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.

    Abnormal node resources

    Node resource metrics measure container resource consumption relative to the node's allocatable capacity, not the physical machine capacity.
    Consumed resources (numerator): Total resources of all containers on the node, including working set memory, page cache, and more.
    Allocatable resources (denominator): Resources available for containers after subtracting node reservations. See Node resource reservation policy.
    Pod scheduling is based on resource requests, not actual usage.
    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    Node CPU usage > 85% Warning sum(irate(container_cpu_usage_seconds_total{pod!=""}[1m])) by (node) / sum(kube_node_status_allocatable{resource="cpu"}) by (node) * 100 >= 85 >= 85% 1 min Fires when node CPU usage exceeds 85% of allocatable resources. Formula: Node resource usage / Total allocatable resources on the node. Release resources with cost analysis and resource profiling to distribute pods across nodes. Scale out nodes — see Scale nodes in an ACK cluster.
    Node CPU resource allocation rate > 85% Normal (sum(sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, node) * on (pod) group_left max(kube_pod_status_ready{condition="true"}) by (pod, node)) by (node)) / sum(kube_node_status_allocatable{resource="cpu"}) by (node) * 100 >= 85 >= 85% — Fires when the CPU resource allocation rate exceeds 85% of allocatable resources. Formula: Total resource requests of scheduled pods / Total allocatable resources on the node. Insufficient resources to schedule more pods. Check for resource waste (actual usage far below requests) with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Node CPU oversold rate > 300% Warning (sum(sum(kube_pod_container_resource_limits{resource="cpu"}) by (pod, node) * on (pod) group_left max(kube_pod_status_ready{condition="true"}) by (pod, node)) by (node)) / sum(kube_node_status_allocatable{resource="cpu"}) by (node) * 100 >= 300 >= 300% — Fires when the CPU oversold rate exceeds 300% of allocatable resources. Formula: Total resource limits of scheduled pods / Total allocatable resources on the node. The 300% threshold is the recommended default — adjust as needed. Total CPU limits far exceed allocatable resources. During traffic peaks, contention and throttling can slow responses. Right-size CPU requests and limits with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Node memory usage > 85% Warning sum(container_memory_working_set_bytes{pod!=""}) by (node) / sum(kube_node_status_allocatable{resource="memory"}) by (node) * 100 >= 85 >= 85% — Fires when node memory usage exceeds 85% of allocatable resources. Formula: Node resource usage / Total allocatable resources on the node. Release resources with cost analysis and resource profiling to distribute pods across nodes. Scale out nodes — see Scale nodes in an ACK cluster.
    Node memory resource allocation rate > 85% Normal (sum(sum(kube_pod_container_resource_requests{resource="memory"}) by (pod, node) * on (pod) group_left max(kube_pod_status_ready{condition="true"}) by (pod, node)) by (node)) / sum(kube_node_status_allocatable{resource="memory"}) by (node) * 100 >= 85 >= 85% — Fires when the memory resource allocation rate exceeds 85% of allocatable resources. Formula: Total resource requests of scheduled pods / Total allocatable resources on the node. Insufficient resources to schedule more pods. Check for resource waste (actual usage far below requests) with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.
    Node memory oversold rate > 300% Warning (sum(sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, node) * on (pod) group_left max(kube_pod_status_ready{condition="true"}) by (pod, node)) by (node)) / sum(kube_node_status_allocatable{resource="memory"}) by (node) * 100 >= 300 >= 300% — Fires when the memory oversold rate exceeds 300% of allocatable resources. Formula: Total resource limits of scheduled pods / Total allocatable resources on the node. The 300% threshold is the recommended default — adjust as needed. Total memory limits far exceed allocatable resources. During traffic peaks, memory can hit the node limit, triggering OOM kills that disrupt workloads. Right-size memory requests and limits with cost analysis and resource profiling. Scale out nodes — see Scale nodes in an ACK cluster.

    Network exceptions

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    CoreDNS request count drops to zero Critical (sum(rate(coredns_dns_request_count_total{}[1m]))by(server,zone)<=0) or (sum(rate(coredns_dns_requests_total{}[1m]))by(server,zone)<=0) <= 0 1 min Detectable only in ACK managed clusters (Pro and Basic editions). Check whether the CoreDNS pods in the cluster are running normally.
    CoreDNS panic exception Critical sum(rate(coredns_panic_count_total{}[3m])) > 0 > 0 3 min Detectable only in ACK managed clusters (Pro and Basic editions). Check whether the CoreDNS pods in the cluster are running normally.
    Ingress controller certificate expiring within 14 days Warning ((nginx_ingress_controller_ssl_expire_time_seconds - time()) / 24 / 3600) < 14 < 14 days — Requires the ACK Ingress controller component to be installed with the Ingress feature enabled. Reissue the Ingress controller certificate.

    Auto Scaling exceptions

    Description Severity PromQL Threshold Window Rule description Common troubleshooting
    HPA replica count at maximum Warning max(kube_horizontalpodautoscaler_spec_max_replicas) by (namespace, horizontalpodautoscaler) - max(kube_horizontalpodautoscaler_status_current_replicas) by (namespace, horizontalpodautoscaler) <= 0 <= 0 difference — Fires when HPA current replicas reach the configured maximum.
    Note

    Enable horizontalpodautoscaler metrics in Alibaba Cloud Prometheus first — disabled by default, free of charge.

    		 Check if the HPA policy meets expectations. If workload remains high, increase maxReplicas or optimize application performance.

    Next steps