Container Service for Kubernetes:Recommendations for using large-scale clusters

Single cluster vs. multiple clusters

A single large-scale cluster reduces management overhead and improves resource utilization. In some business scenarios, however, splitting services across multiple clusters makes more sense.

Consider multiple clusters when:

To manage multiple clusters for tasks such as application deployment, traffic management, job distribution, and monitoring, enable fleet management.

Keep clusters on the latest version

Newer Kubernetes versions include stability, performance, and scalability improvements that directly benefit large-scale clusters. Notable examples:

• v1.31: kube-apiserver serves consistent reads for List requests from its cache, reducing direct etcd access and lowering etcd load. See Consistent reads from cache.

• v1.33: kube-apiserver uses streaming encoding (StreamingCollectionEncodingToJSON and StreamingCollectionEncodingToProtobuf) for List operations, reducing kube-apiserver memory usage at scale. See Streaming List responses.

ACK releases supported Kubernetes versions in sync with the community and discontinues support for expired versions—including stopping new feature releases, bug fixes, and security patches—while providing only limited technical support for expired versions. Monitor version release announcements through the console and upgrade promptly.

• Supported versions: Version guide

• Upgrade considerations and procedures: Upgrade clusters

• Manual upgrade: Manually upgrade an ACK cluster

• Automatic upgrade: Automatically upgrade a cluster

Monitor cluster resource limits

Stay within the following limits to maintain availability and performance in large-scale clusters.

Configure control plane component parameters

ACK managed cluster Pro lets you customize control plane component parameters for kube-apiserver, kube-controller-manager, and kube-scheduler. In large-scale clusters, tuning throttling-related parameters is critical.

Plan scaling rates

The control plane is typically under low pressure during stable operation, even in large clusters. The risk comes from rapid, large-scale changes—creating or deleting many resources at once, or scaling many nodes simultaneously.

For example, a 5,000-node cluster running stable workloads may show little control plane pressure. But a 1,000-node cluster that creates 10,000 short-lived jobs within a minute, or concurrently scales out 2,000 nodes, can push the control plane to its limits.

Node scaling:

For clusters with more than 2,000 nodes, when manually scaling through node pools:

• Single node pool, single operation: no more than 100 nodes

• Across multiple node pools simultaneously: no more than 300 nodes total

Pod scaling:

When a pod is associated with a service, each scaling event updates the Endpoints or EndpointSlice and pushes that update to all nodes—creating a cluster-wide data propagation event. In large clusters, this effect is amplified.

For clusters with more than 5,000 nodes:

• Pods not associated with a service endpoint: update QPS ≤ 300/s

• Pods associated with a service endpoint: update QPS ≤ 10/s

For deployments using a Rolling Update strategy, set smaller values for maxUnavailable and maxSurge to reduce the pod replacement rate.

Optimize client access patterns

As cluster resource counts grow, frequent API Server requests amplify control plane load and can cause cascading failures. Follow these guidelines when building controllers or tools that access the API Server.

Use informers for cached data access:

• Use client-go informers to read resources from a local cache instead of issuing direct List requests to the API Server.

• Informers maintain a single Watch connection and serve read requests locally, reducing API Server load significantly.

Optimize direct API Server requests:

• Set `resourceVersion=0` in List requests to read from the API Server's cache rather than going through to etcd. This reduces API Server–etcd round-trips and speeds up responses:

  k8sClient.CoreV1().Pods("").List(context.Background(), metav1.ListOptions{ResourceVersion: "0"})

• Use label selectors and field selectors to narrow List request scope and reduce response payload size. Note: etcd is a key-value store and cannot filter by label or field—the API Server handles that filtering from its cache. Always combine selectors with resourceVersion=0 to avoid hitting etcd directly.

• Use protobuf for non-CRD resources. protobuf uses less memory and bandwidth than JSON. Specify multiple content types in the Accept header to fall back to JSON when protobuf is unavailable:

  Accept: application/vnd.kubernetes.protobuf, application/json

  See Alternate representations of resources.

• Avoid repeated or broad List calls. List less often, and chunk large lists when informers are unavailable.

• Add exponential backoff and retry policies to prevent a burst of retries from overwhelming the API Server after a transient error.

Use a centralized controller design:

Avoid deploying independent controllers on every node that each watch the full cluster state. On startup, all such controllers issue simultaneous List requests to sync state, which can crash the control plane.

Instead, run one or a small group of centrally managed controller instances for the entire cluster. A centralized controller issues a single List request on startup and maintains the minimum number of Watch connections, dramatically reducing API Server pressure.

Plan large-scale workloads

Monitor control plane metrics

Use the control plane component monitoring dashboard to track core metrics and detect issues early. See Control plane component monitoring.

What's next

• Cluster quotas and limits: Quotas and limits

• Network planning: Plan CIDR blocks for an ACK managed cluster

• High-reliability workload configurations: Recommended workload configurations

• Cluster troubleshooting: Troubleshooting and FAQ about cluster management