Considerations and CCM resource update policies for configuring Service load balancing - Container Service for Kubernetes

When a Service is set to Type=LoadBalancer, ACK Cloud Controller Manager (CCM) creates or configures a load balancer (Classic Load Balancer or Network Load Balancer), including the instance, listeners, and backend server groups. This topic explains considerations for configuring Service load balancing and CCM resource update policies.

Considerations

Which SLB instances can be reused?

Only instances created in the Server Load Balancer (SLB) console can be reused. SLB instances automatically created by cloud-controller-manager or managed by ACK (such as the API Server instance) cannot be reused.
To reuse a private SLB instance, it must be in the same Virtual Private Cloud (VPC) as the cluster. Cross-VPC reuse applies only to NLB instances.
The address type of the reused SLB instance must match the access type of the Service. If the Service is set for Public Access (using the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"), the IP Version of the SLB instance must be Public. If the Service is set for internal access (using the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"), the IP Version of the SLB instance must be Private.
Multiple Services cannot use the same listener port of the same SLB instance.
When reusing an SLB instance across clusters, ensure the namespace and Service name combination is unique in each cluster.

Considerations for CCM-managed SLB instances

CCM configures load balancing only for Type=LoadBalancer Services.

Important

If a Type=LoadBalancer Service is changed to another type, CCM deletes the SLB configurations. The Service becomes inaccessible through the SLB instance.

CCM uses a declarative API and may refresh SLB configuration based on Service changes. Manual modifications in the SLB console may be overwritten.

Important

Do not modify configurations of ACK-managed SLB instances in the SLB console. Your changes may be lost, and the Service may become inaccessible.

Do not delete or modify the service.k8s.alibaba/resources or service.k8s.alibaba/nlb finalizer. This may prevent CLB or NLB resources from being properly reclaimed.
For Cloud Controller Manager v2.5.0 or later, specifying a CLB instance when creating a Service requires whitelisting. Submit a request on Quota Center.

Considerations for accessing the external IP of a LoadBalancer Service from within a cluster

When you access the SLB IP address of a LoadBalancer Service from within the cluster, the cluster network may intercept traffic and forward it directly to the backend Service Endpoint, depending on the network plug-in type, plug-in version, and cluster version.

This bypasses the external SLB instance, causing configurations that rely on it to fail. The main affected scenarios include:

externalTrafficPolicy is set to Local: Access may fail if traffic is forwarded to a node without backend pods.
Proxy Protocol is enabled: The backend service cannot obtain the Proxy Protocol header from the SLB instance, causing a handshake failure.
HTTP/HTTPS listener: SLB-dependent operations such as TLS termination or adding headers like X-Forwarded-For do not take effect.

Therefore, to access a LoadBalancer Service from within a cluster:

Use the Service's in-cluster address (ClusterIP or DNS name such as <service-name>.<namespace>.svc.cluster.local) for stable inter-service communication.
To access via the SLB entry point, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname annotation and use the configured domain name instead of the IP address. See Set a hostname for a Service.

Important
When using this feature, avoid scheduling the client pod and server pod on the same node. Asymmetric routing may cause access failures.

Considerations for managing large-scale load balancing in a single cluster with CCM

In large clusters, CCM may be slow to process Service events. The following scenarios can delay operations such as creating or deleting SLB instances and updating server group endpoints:

Many LoadBalancer Services are created or deleted at the same time.
Many Service Endpoints are updated at the same time.
Nodes are added or deleted in batches when many LoadBalancer Services exist.

Follow these practices:

Before large-scale changes, perform capacity assessment and stress testing. Reserve enough processing margin to prevent disruptions.
Use configurations such as readinessGates and preStop hooks to prevent traffic interruptions during changes. See Implement zero-downtime rolling deployments.
When using many LoadBalancer Services, monitor CLB and NLB quota limits closely.
Upgrade the cluster and CCM to the latest versions for large-scale scenario optimizations.

Replace the SLB instance for a Service

You cannot directly replace the SLB instance for an existing LoadBalancer Service. Delete and recreate the Service instead.

External access and network traffic path

When you use a LoadBalancer Service to expose applications externally, understanding the traffic path helps you correctly size bandwidth and locate monitoring data.

Traffic path

The request path is: Client → Internet → Load Balancer (CLB/NLB) → Alibaba Cloud internal network → Backend containers/nodes. Traffic between the load balancer and backend nodes travels through Alibaba Cloud's internal network, not through the public bandwidth of cluster nodes.

Common bandwidth misconception

Increasing the public bandwidth of ACK nodes does not resolve high-concurrency or peak traffic issues for externally exposed services, because business traffic does not pass through the node's public network egress. Public bandwidth consumption only occurs between the client and the load balancer. To address external access performance bottlenecks, adjust the load balancer bandwidth specification rather than the node's public bandwidth.

Monitoring upstream and downstream traffic

To view upstream and downstream traffic for a Service, log on to the CLB or NLB console and check the monitoring data on the monitoring page of the corresponding instance. Node-level monitoring does not reflect the actual throughput of externally exposed Service traffic.

Access methods and restrictions from outside the cluster

ClusterIP access restrictions

ECS instances or other resources outside the cluster in the same VPC cannot directly access a Service's ClusterIP. ClusterIP is a virtual IP address that is only reachable within the cluster network. Attempts to access ClusterIP from an external ECS instance will fail.

Recommended access method: Private LoadBalancer

To access cluster applications from other ECS instances in the same VPC, create a LoadBalancer-type Service with an intranet address type by adding the following annotation:

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet

This creates a private SLB instance accessible from within the VPC. Use the generated private SLB IP address to access the Service from external ECS instances.

Pod IP instability

Pod IP addresses are dynamically assigned and change when a Pod restarts, making them unsuitable as fixed access endpoints. Use a private LoadBalancer or NodePort Service to provide a stable and persistent access entry point for cluster applications.

Quota limits

VPC

Each cluster node uses one VPC route table entry. The default limit is 200 entries. If your cluster exceeds 200 nodes, log on to the Quota Center console and submit an application.
See VPC limits and quotas.

To query VPC quotas, see VPC quota management.

Server Load Balancer

CCM creates an SLB instance for each Type=LoadBalancer Service. The default quota is 60 CLB and 60 NLB instances. To increase this quota, log on to the Quota Center console and submit an application.
CCM attaches ECS instances or Elastic Network Interfaces (ENIs) to the SLB backend server group based on Service configuration, creating a separate server group per targetPort. Quota limits:
- Number of backend servers: A CLB instance supports up to 200 backend servers; an NLB instance supports up to 400 ECS, ENI, or IP-based servers. Quota formula: backend servers × targetPorts. To increase the quota, log on to the Quota Center console and submit an application in advance.
- Server group attachments per instance: An ECS instance or ENI can be attached to up to 50 CLB backend server groups and 200 NLB backend server groups. To increase this limit, log on to the Quota Center console and submit an application.
- Extra quota during rolling updates: New pods are created before old pods are destroyed, consuming extra quota that may exceed expectations. Reserve sufficient quota in advance.
CCM creates listeners based on Service ports. The default limit is 50 listeners per CLB or NLB instance. To add more, log on to the Quota Center console and submit an application.
See CLB limits and NLB limits.

Query SLB quotas in Server Load Balancer quota management.

SLB instance update policies

You can specify an existing SLB instance or let CCM create one. The update policies differ between these two methods.

Resource object	Specify an existing SLB instance	CCM-managed SLB instance
Server Load Balancer	Set the annotation: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id`. CCM uses this instance as the Service load balancer, configures it based on other annotations, and creates vServer groups. When the Service is deleted, CCM retains the specified SLB instance.	CCM creates and configures resources such as the SLB instance, listeners, and vServer groups based on Service configuration. When the Service is deleted, CCM deletes the automatically created SLB instance.
Listener	Set the annotation: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners`: If set to `"false"`, CCM does not manage any listener configurations for the SLB instance. If set to `"true"`, CCM manages listeners based on the Service configuration. If a listener already exists, CCM overwrites it.	CCM creates and configures the listener policy based on Service configuration.
Backend server group	When Service backend Endpoints or cluster nodes change, CCM updates the backend vServer groups. Terway: CCM attaches pod IPs to the SLB backend by default, instead of ECS nodes. Flannel: The backend server group update policy varies by Service mode. Cluster mode (`spec.externalTrafficPolicy = Cluster`): CCM attaches all nodes to the SLB backend by default, unless the backend is configured with BackendLabel. Important Each ECS instance has limits on how many SLB instances it can use. Cluster mode consumes this quota quickly. When exhausted, Services fail to reconcile. Use Local mode to resolve this. Local mode (`spec.externalTrafficPolicy = Local`): CCM adds only nodes running the Service's pods to the SLB backend. This reduces SLB quota consumption and preserves Layer 4 source IPs. Flannel: CCM never adds master nodes to the SLB backend. Flannel: CCM does not remove drained (`kubectl drain`) or unschedulable (`kubectl cordon`) nodes from the SLB backend by default. To remove them, set `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend` to `on`.

Enable deletion protection for a Service

Deletion protection prevents accidental removal of Services that involve critical business operations or sensitive data. Protected resources can only be deleted after you manually disable the protection. See Enable deletion protection for a Service.