Container Service for Kubernetes:Service FAQ

What CLB instances does an ACK cluster create by default?

If the Nginx Ingress Controller add-on is installed during cluster creation, the cluster creates two CLB instances:

• API server CLB: The API server endpoint. All cluster requests pass through this instance on TCP port 6443. Backends are API server pods or master ECS instances.

• Nginx Ingress Controller CLB: Associated with the nginx-ingress-controller Service in kube-system. Dynamically binds to Ingress Controller pods for load balancing external requests. Listens on TCP ports 80 and 443.

How do I choose between the Local and Cluster external traffic policies?

Local and Cluster external traffic policies differ by network plugin. See External traffic policies: Local and Cluster.

Why was my CLB configuration modified?

CCM uses a declarative API and automatically updates CLB configuration based on the Service definition. Direct CLB console changes may be overwritten.

How are node weights calculated in Local mode?

The following example uses an application pod (app=nginx) deployed on three ECS instances and exposed through a Service.

v1.9.3.276-g372aa98-aliyun and later

CCM sets the node weight equal to the number of pods on that node. For example, if three ECS instances host 1, 2, and 3 pods respectively, their weights are 1, 2, and 3. Traffic is distributed at a 1:2:3 ratio, resulting in balanced load across pods.

Formula:

Between v1.9.3.164-g2105d2e-aliyun and v1.9.3.276-g372aa98-aliyun

CCM calculates weights based on pod count. For the same three ECS instances, the calculated weights are 16, 33, and 50, giving an approximate 1:2:3 traffic ratio.

Formula:

Earlier than v1.9.3.164-g2105d2e-aliyun

All backend servers have a weight of 100, which distributes traffic evenly across ECS instances regardless of pod count. This causes unbalanced load across pods -- nodes with fewer pods receive disproportionately more traffic per pod.

How do I enable connection draining for a LoadBalancer Service?

Add the following annotations to the Service. After a backend server is removed, the CLB instance continues processing existing connections for the duration of the drain timeout.

• service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

• service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

See Enable connection draining for a listener.

How do I list all CLB instances in a cluster?

1. Connect to the cluster using kubectl.

2. Retrieve the name, namespace, IP address, and address type of every LoadBalancer Service. Example output:

      kubectl get services -A -ojson | jq '.items[] | select(.spec.type == "LoadBalancer") | {name: .metadata.name, namespace: .metadata.namespace, ip: .status.loadBalancer.ingress[0].ip, lb_type: .metadata.annotations."service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type"}'

      {
        "name": "test",
        "namespace": "default",
        "ip": "192.168.*.*",
        "lb_type": "intranet"
      }
      {
        "name": "nginx-ingress-lb",
        "namespace": "kube-system",
        "ip": "47.97.*.*",
        "lb_type": "null"
      }

How do I enable CLB renaming for older CCM versions?

CCM v1.9.3.10 and later automatically tag CLB instances to support renaming. For CLB instances created by earlier CCM versions, add the tag manually.

This procedure applies only to CLB instances created by CCM v1.9.3.10 or earlier. The Service type must be LoadBalancer.

1. Connect to the cluster master node. See Connect to an ACK cluster using kubectl.

2. View the Service type and IP address. Replace ${namespace} and ${service} with the actual namespace and Service name.

      kubectl get svc -n ${namespace} ${service}

   

3. Generate the tag for the CLB instance:

      kubectl get svc -n ${namespace} ${service} -o jsonpath="{.metadata.uid}"|awk -F "-" '{print "kubernetes.do.not.delete: "substr("a"$1$2$3$4$5,1,32)}'

   

4. Log on to the CLB console. Using the IP address from step 2, search for the CLB instance in the corresponding region.

5. Add a tag to the CLB instance using the key and value from step 3 (items 1 and 2 in the preceding figure). See Create and manage a CLB instance.

Why is a CLB IP address inaccessible from within the cluster?

Two scenarios can cause this.

Scenario 1: Private CLB not created by a Service

When a pod accesses a private CLB instance not created by a Service, access fails if the pod and a backend server are on the same node. This is a Layer 4 limitation: an ECS instance cannot be both a backend server and a client for the same CLB instance.

To resolve this:

• Change the CLB IP address to a public IP address.

• Create the CLB instance through a Service and set the external traffic policy to Cluster. Kube-proxy then intercepts CLB-bound traffic from within the cluster, bypassing the limitation.

Scenario 2: externalTrafficPolicy: Local blocks internal access

When a Service uses externalTrafficPolicy: Local, internal access to the CLB IP fails from nodes without a backend pod. The CLB address is for external use. Internal traffic is intercepted by kube-proxy and routed locally (iptables or IPVS). If the node has no local backend pod, the connection fails.

See kube-proxy adds external-lb address to local node iptables rules.

Solutions (listed by recommendation):

1. Access Services internally by ClusterIP or Service name. For example, the Ingress Service name is nginx-ingress-lb.kube-system. This is the recommended approach.

2. Change externalTrafficPolicy to Cluster. This forwards traffic to all nodes but enables SNAT, so the backend cannot retrieve the real client IP.

      kubectl edit svc nginx-ingress-lb -n kube-system

3. Use elastic network interface (ENI) passthrough on Terway clusters. Set externalTrafficPolicy to Cluster and add the service.beta.kubernetes.io/backend-type: "eni" annotation. This preserves the source IP while allowing internal access. See Use annotations to configure a Classic Load Balancer (CLB) instance.

      apiVersion: v1
      kind: Service
      metadata:
        annotations:
          service.beta.kubernetes.io/backend-type: eni
        labels:
          app: nginx-ingress-lb
        name: nginx-ingress-lb
        namespace: kube-system
      spec:
        externalTrafficPolicy: Cluster

When are CLB instances deleted?

Deletion behavior depends on whether CCM created the CLB instance and how it was associated with the Service.

If the Service reuses an existing CLB instance (annotated with service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: {your-slb-id}), deleting the Service does not delete the CLB instance. Otherwise, deleting the Service also deletes the CLB instance.

What do I do if I accidentally delete a CLB instance?

Scenario 1: Ingress CLB deleted

The following steps use Nginx Ingress as an example.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the target cluster and click its name. In the left navigation pane, choose Network > Services.

3. Select kube-system from the Namespace drop-down list.

   • If nginx-ingress-lb appears in the list, click Edit YAML in the Actions column. Remove the status.loadBalancer field and its content, then click OK. CCM rebuilds the CLB instance.

   • If nginx-ingress-lb does not appear, click Create from YAML and use the following template: ``yaml apiVersion: v1 kind: Service metadata: labels: app: nginx-ingress-lb name: nginx-ingress-lb namespace: kube-system spec: externalTrafficPolicy: Local ports: selector: app: ingress-nginx type: LoadBalancer ``

     • name: http port: 80 protocol: TCP targetPort: 80

     • name: https port: 443 protocol: TCP targetPort: 443

Scenario 2: Application-specific CLB deleted

• If the Service is no longer needed, delete it.

• If the Service is still in use:

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, find the target cluster and click its name. In the left navigation pane, choose Network > Services.

  3. From the Namespace drop-down list, click All Namespaces, then find the target Service.

  4. In the Actions column, click Edit YAML. Delete the status.loadBalancer content and click OK. CCM rebuilds the CLB instance.

Why does reusing an existing CLB instance fail?

Check the following in order:

1. CCM version: Versions earlier than v1.9.3.105-gfd4e547-aliyun do not support reusing CLB instances. For upgrade instructions, see Upgrade the CCM component.

2. Instance origin: CLB instances created by the cluster cannot be reused.

3. API server CLB: The API server CLB instance cannot be reused.

4. VPC mismatch: A private-facing CLB instance must be in the same virtual private cloud (VPC) as the cluster. Cross-VPC reuse is not supported.

Why are listeners not created when I reuse an existing CLB instance?

Verify that the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners annotation is "true". Without it, listeners are not created.

CCM does not overwrite listeners on existing CLB instances by default:

Overwriting active listeners can cause service interruptions.

CCM supports limited backend configurations. For complex setups, create listeners in the CLB console. Only force-override listeners when their ports are no longer in use.

What do I do if a CLB instance stays in the Pending state?

1. Run kubectl -n {your-namespace} describe svc {your-svc-name} to check event messages.

2. Resolve the errors reported in the events. See Service error messages and solutions for guidance.

If no events appear, see Why are no events displayed for Service and LoadBalancer synchronization?

What do I do if a vServer group is not updated?

1. Run kubectl -n {your-namespace} describe svc {your-svc-name} to check event messages.

2. Resolve the errors reported in the events. See Service error messages and solutions for guidance.

If no events appear, see Why are no events displayed for Service and LoadBalancer synchronization?

What do I do if a Service annotation does not take effect?

1. Run kubectl -n {your-namespace} describe svc {your-svc-name} to check for error events. If errors exist, see Service error messages and solutions.

2. If no errors appear, check the following:

   • CCM version: Verify the CCM version supports the annotation. See Use annotations to configure a Classic Load Balancer (CLB) instance.

   • Annotation presence: Log on to the ACK console. On the Services page, click the Service name and confirm that the annotation exists. If missing, add it. See Use annotations to configure a Classic Load Balancer (CLB) instance.

   • Annotation syntax: Verify the annotation key and value are correct.

To view the Service list, see Manage services.

Why are no events displayed for Service and LoadBalancer synchronization?

If kubectl -n {your-namespace} describe svc {your-svc-name} shows no events, check the CCM version:

• Earlier than v1.9.3.276-g372aa98-aliyun: Events are not supported. Upgrade CCM. For upgrade instructions, see Upgrade the CCM component.

• v1.9.3.276-g372aa98-aliyun and later: Submit a ticket for further investigation.

Quota and resource errors

Configuration errors

Network errors

CLB reuse errors

Billing errors

How do I access a NodePort Service?

• From within the cluster (on a cluster node): Use ClusterIP + port, or node IP + Service NodePort. The default NodePort is greater than 30000.

• From outside the cluster (within the same VPC): Use the node IP address and the Service NodePort.

• From outside the VPC (from another VPC or the internet): Expose the Service as a LoadBalancer type and access it through its external endpoint.

If the external traffic policy is Local, verify the node you access hosts a backend pod of the Service. See External traffic policies: Local and Cluster.

How do I configure a listener for a NodePort Service?

CCM supports listener configuration only for LoadBalancer Services. Change the Service type from NodePort to LoadBalancer.

How do I configure the NodePort range?

The API server --service-node-port-range parameter (ServiceNodePortRange) controls the port range for NodePort and LoadBalancer Services. The default range is 30000 to 32767. In ACK Pro clusters, customize control plane parameters to adjust this range. See Customize the parameters of the control plane for an ACK Pro cluster.

Keep the following in mind:

• Avoid port range conflicts. The NodePort range must not overlap with the net.ipv4.ip_local_port_range kernel parameter, which controls local port numbers available to applications on the node. The default ip_local_port_range is 32768 to 60999.

• Default ranges are safe. With default ACK settings, ServiceNodePortRange (30000-32767) and ip_local_port_range (32768-60999) do not overlap. If either range has been extended and they now overlap, sporadic network exceptions may occur. In severe cases, health checks fail and nodes go offline. Restore the default values or adjust both ranges to eliminate overlap.

• Reconfigure existing Services after changes. After adjusting the port range, some NodePort or LoadBalancer Services may still use ports within the ip_local_port_range range. Run kubectl edit <service-name> and change the spec.ports.nodePort field to an unused NodePort.

What do I do if a CCM upgrade fails?

See Cloud Controller Manager (CCM) upgrade check fails.

How do I add permissions required for a CCM upgrade in an ACK dedicated cluster?

Later CCM versions introduce Alibaba Cloud APIs that require additional Resource Access Management (RAM) permissions. For example, v2.11.2 adds route management for Flannel networks, and v2.12.1 adds batch management for Network Load Balancer (NLB).

Before upgrading to v2.11.2 or later in an ACK dedicated cluster, grant the required permissions to the cluster RAM role:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, find the target cluster and click its name. In the left navigation pane, click Cluster Information.

3. Click the Basic Information tab, then click the role name for Master RAM Role.

4. In the RAM console, click Permissions > Policies in the left navigation pane. Find the custom policy that starts with k8sMasterRolePolicy-Ccm- and click the policy name.

   For older clusters, this policy may not exist. Select a custom policy whose name starts with k8sMasterRolePolicy- instead.

5. Click Edit Policy Document and add the nlb:ListAsynJobs permission to the NLB permissions. For Flannel clusters, also add the vpc:CreateRouteEntries and vpc:DeleteRouteEntries permissions to the VPC permissions.

   

   

6. Submit the changes and proceed with the CCM upgrade.