ACS supports the scheduled inspection feature provided by CIS. Set scheduled rules to periodically inspect your clusters for risk alerts. This topic describes common risk alerts identified during cluster inspections and their solutions.
Cluster inspection items and alerts
-
To learn how to use the cluster inspection feature, see Use cluster diagnostics.
-
Inspection items may vary slightly by cluster configuration. The inspection report provides the definitive results.
|
Check Type |
Inspection Item |
Alert |
|
Resource Quotas ResourceQuotas |
||
|
SLB backend server quota check |
||
|
SLB listener quota check |
||
|
Resource Watermark ResourceLevel |
SLB bandwidth usage check |
|
|
SLB maximum connections check |
||
|
Versions & Certificates Versions&Certificates |
Cluster Kubernetes version check |
|
|
Cluster Risk ClusterRisk |
API server SLB instance exists |
|
|
API server SLB instance status |
||
|
API server SLB port 6443 listener configuration |
||
|
API server SLB access control configuration |
||
|
DNS Service ClusterIP |
||
|
DNS Service backend endpoints |
||
|
Multiple Services sharing an SLB port |
Insufficient SLB quota in VPC
Impact: The remaining Server Load Balancer (SLB) quota in the Virtual Private Cloud (VPC) is less than five. Each LoadBalancer Service creates one SLB instance. After the SLB quota is exhausted, newly created LoadBalancer Services cannot work.
Solution: By default, an account can retain 60 SLB instances. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For quota limits, see Quotas.
Insufficient SLB backend server quota
Impact: An SLB instance supports a limited number of Elastic Compute Service (ECS) instances as backend servers. When a LoadBalancer Service grows large, its pods are distributed across multiple ECS instances. If the number of ECS instances exceeds the backend server quota of the SLB instance, the excess ECS instances cannot be added to the SLB instance.
Solution: By default, an SLB instance can have 200 backend servers. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For quota limits, see Quotas.
Insufficient SLB listener quota
Impact: An SLB instance supports a limited number of listeners. For a LoadBalancer Service, each listening port maps to one SLB listener. When the number of ports reaches the listener limit of the SLB instance, the excess ports cannot serve traffic.
Solution: By default, an SLB instance can have 50 listeners. If you need a higher quota, log on to the Quota Center console and submit an application to increase the quota. For quota limits, see Quotas.
High SLB bandwidth usage
Impact: The peak outbound network bandwidth over the past three days exceeded 80% of the limit. After the bandwidth reaches the limit, the SLB instance may drop packets, which causes request jitter and latency.
Solution: Upgrade the SLB instance type to increase the bandwidth limit. For more information, see Expose an application by using an existing SLB instance.
High SLB maximum connections
Impact: The maximum number of connections over the past three days reached 80% of the limit. After the number of connections reaches the limit, new connections cannot be established and clients cannot connect.
Solution: Upgrade the SLB instance type to increase the maximum connection limit. For more information, see Expose an application by using an existing SLB instance.
High SLB new connection rate
Impact: The maximum new connection rate over the past three days reached 80% of the limit. After the new connection rate reaches the limit, new connections cannot be established for a short period and clients cannot connect.
Solution: Upgrade the SLB instance type to increase the maximum new connection rate. For more information, see Expose an application by using an existing SLB instance.
High SLB QPS
Impact: The maximum queries per second (QPS) over the past three days reached 80% of the limit. After the QPS reaches the limit, clients may fail to connect.
Solution: Upgrade the SLB instance type to increase the maximum QPS limit. For more information, see Expose an application by using an existing SLB instance.
Outdated cluster version
Impact: The cluster version is approaching or has passed the end of its support period.
Solution: Upgrade the cluster to a supported Kubernetes version as soon as possible.
API server SLB instance exists
Impact: Without a load balancer for the cluster API server, only one API server instance runs, which makes the API server a single point of failure (SPOF). If that instance fails, the entire cluster stops working.
Solution: Configure a load balancer for the cluster API server.
API server SLB instance status
Impact: All cluster operations, such as pod scheduling, Service deployment, and scaling, are interrupted or delayed. In-cluster Service discovery depends on the API server. If the SLB instance is abnormal, Service discovery may fail.
Solution: Confirm that the SLB instance is configured correctly, including the backend server configuration, listening ports, and health check mechanism.
API server SLB port 6443 listener configuration
Impact: If the listener on port 6443 of the cluster API server SLB instance is misconfigured, all requests that reach the API server through the SLB instance fail. This includes kubectl operations, Dashboard access, and API requests from other workloads. In-cluster Service discovery may also fail because name resolution depends on the API server.
Solution: Confirm that the SLB instance is configured correctly, including the backend server configuration, listening ports, and health check mechanism. Confirm that a listener is configured for port 6443 and that the listener uses HTTPS.
API server SLB access control configuration
Impact: If the access control configuration of the cluster API server SLB instance is misconfigured, cluster management operations such as node management, pod scheduling, and Service deployment are interrupted or restricted. Workloads in the cluster depend on the API server for communication and Service discovery, so these requests fail.
Solution: Check the access control configuration of the SLB instance, such as security groups and access control lists (ACLs). Confirm that legitimate IP addresses and ports, especially port 6443, can access the API server. Confirm that the TLS/SSL configuration between the SLB instance and the API server is correct and that the certificates are valid and not expired.
DNS Service ClusterIP
Impact: If a ClusterIP is not correctly assigned to the cluster DNS Service, the DNS Service becomes unavailable. This breaks cluster functions and affects all workloads that depend on DNS resolution.
Solution: Check the network plugins and cluster configurations for conflicts or errors. Redeploy the CoreDNS add-on so that the DNS Service is configured and assigned a ClusterIP correctly.
DNS Service backend endpoints
Impact: If no backend DNS servers are associated with the cluster DNS Service, the DNS Service is unavailable and all DNS resolution in the cluster fails.
Solution: Check whether the CoreDNS add-on configuration file (Corefile) is configured correctly. Make sure that the forward or proxy directive points to a valid set of backend DNS servers.
Multiple Services sharing an SLB port
Impact: If multiple Services share the same port on the same SLB instance, port conflicts occur and the affected Services do not function correctly.
Solution: Modify or delete the conflicting Services. Make sure that Services that share one SLB instance use different ports.