Considerations for configuring Service load balancers and CCM resource update policies - Container Service for Kubernetes

When you set the type of a Service to load balancing (Type=LoadBalancer), the cloud-controller-manager (CCM) component of ACK creates or configures a Server Load Balancer (SLB) instance for the Service. The instance can be a Classic Load Balancer (CLB) or a Network Load Balancer (NLB). The configuration includes resources such as the instance, listeners, and backend server groups. This topic describes the considerations for configuring a Service load balancer and the resource update policies of CCM.

Considerations

Which SLB instances can be reused?

Only SLB instances created in the SLB console can be reused. SLB instances automatically created by the cloud-controller-manager cannot be reused.
If you want to reuse a private SLB instance in an ACK cluster, the instance must be in the same Virtual Private Cloud (VPC) as the cluster. Cross-VPC reuse is supported only for NLB instances.
The address type of the reused SLB instance must match the access type of the Service. If the Service is configured for Public Network Access (service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"), the Address Type of the SLB instance must be Public. If the Service is configured for Internal Access (service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"), the Address Type of the SLB instance must be Private.
Multiple Services cannot use the same listener port of the same SLB instance.
When you reuse an existing SLB instance across clusters, make sure that the combination of the namespace and Service name is unique in each cluster.

Considerations for managing SLB instances with CCM

CCM configures an SLB instance only for a Service of Type=LoadBalancer. CCM does not configure an SLB instance for a Service of another type.

Important

When a Type=LoadBalancer Service is changed to Type!=LoadBalancer, CCM deletes the configuration added for the load balancer, making the Service inaccessible through the load balancer.

CCM uses a declarative API and automatically refreshes the SLB configuration based on the Service configuration under certain conditions. If you set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: to "true", any configurations you modify in the SLB console are at risk of being overwritten.

Important

Do not manually modify any configurations of an SLB instance that is created and maintained by ACK in the SLB console. Otherwise, the configurations may be lost and the Service may become inaccessible.

Do not manually delete or modify the service.k8s.alibaba/resources or service.k8s.alibaba/nlb finalizer on the Service. Manually modifying the finalizer may prevent the CLB or NLB resources from being revoked properly.
When the Cloud Controller Manager version is v2.5.0 or later, the option to use a CLB instance when creating a Service in the console is a whitelisted feature. To use this feature, submit a request on the Quota Center platform.

Considerations for managing large-scale SLB instances in a single cluster with CCM

CCM has limits on its ability to process Service events. In large clusters with many nodes or many LoadBalancer-type Services, CCM may experience latency. This can occur when you create or delete many Services, change many Service Endpoints simultaneously, or add or remove nodes. The latency affects operations such as creating and deleting SLB instances and updating Endpoints in server groups.

To perform batch changes for your business, evaluate the capacity and perform stress testing in advance. This helps prevent business disruptions caused by CCM processing latency.

How do I replace the SLB instance for a Service?

You cannot reassign an SLB instance for an existing Service of the LoadBalancer type. To replace the SLB instance, delete and then recreate the Service.

Quota limits

VPC

Each node in a cluster corresponds to one route entry. By default, a VPC supports only 200 route entries. If your cluster has more than 200 nodes, log on to the Quota Center console and submit an application.
For more information about VPC limits, see Limits and quotas.
To query VPC quotas, see VPC quota management.

Server Load Balancer

CCM creates an SLB instance for each Service of Type=LoadBalancer. By default, a user can have up to 60 instances. If you need more than 60 instances, log on to the Quota Center console and submit an application.
CCM attaches Elastic Compute Service (ECS) instances to the backend server groups of the SLB instance based on the Service configuration.
- By default, an ECS instance can be attached to 50 backend server groups. If an ECS instance needs to be attached to more backend server groups, log on to the Quota Center console and submit an application.
- By default, an SLB instance can have 200 backend servers. To add more backend servers, log on to the Quota Center console and submit an application.
CCM creates listeners based on the ports defined in the Service. By default, an SLB instance can have 50 listeners. To add more listeners, log on to the Quota Center console and submit an application.
For more information about SLB limits, see CLB limits and NLB limits.
To query SLB quotas, see Server Load Balancer quota management.

SLB instance update policies

ACK lets you specify an existing SLB instance for a Service or have CCM automatically create a new one. The resource update policies for these two methods differ, as shown in the following table.

Resource object	Specify an existing SLB instance	Let CCM manage the SLB instance
Server Load Balancer	Set the annotation: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id`. CCM uses this instance as the load balancer for the Service. It configures the SLB instance based on other annotations and automatically creates multiple vServer groups. When the Service is deleted, CCM does not delete the existing SLB instance that you specified by its ID.	CCM automatically creates and configures resources such as the SLB instance, listeners, and vServer groups based on the Service configuration. All these resources are managed by CCM. When the Service is deleted, CCM deletes the automatically created SLB instance.
Listener	Set the annotation: `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners`: If set to `"false"`, CCM does not manage any listener configurations for the SLB instance. If set to `"true"`, CCM manages the listeners based on the Service configuration. If a listener already exists, CCM overwrites it.	CCM automatically creates and configures the listener policy based on the Service configuration.
Backend server group	When the backend Endpoints of the Service or the cluster nodes change, CCM automatically updates the backend vServer groups of the SLB instance. For the Terway network plug-in, CCM attaches pod IPs to the backend of the SLB instance by default, instead of attaching ECS nodes. For the Flannel network plug-in, the update policy for backend server groups varies based on the Service mode. Cluster mode (`spec.externalTrafficPolicy = Cluster`): CCM attaches all nodes to the backend of the SLB instance by default (unless you configure the backend using the BackendLabel tag). Important SLB has a quota limit on the number of SLB instances to which an ECS instance can be added. This method consumes the quota quickly. When the quota is exhausted, the Service reconciliation fails. To resolve this issue, use Local mode for the Service. Local mode (`spec.externalTrafficPolicy = Local`): CCM adds only the nodes that run the pods for the Service to the backend of the SLB instance by default. This slows the consumption of the SLB quota and supports Layer 4 source IP preservation. For the Flannel network plug-in, CCM never adds master nodes to the backend of the SLB instance. For the Flannel network plug-in, CCM does not remove drained (`kubectl drain`) or unschedulable (`kubectl cordon`) nodes from the backend of the SLB instance by default. To remove these nodes, set `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend` to `on`.

Enable deletion protection for a Service

You can enable deletion protection for Services that involve critical business or sensitive data to prevent them from being accidentally deleted. If you enable this feature, the corresponding resources can be deleted only after you manually disable deletion protection. For more information about how to enable deletion protection for a Service, see Enable deletion protection for a Service.