Container Service for Kubernetes:Troubleshoot service issues

Uneven load distribution across CLB backends

Cause: The CLB scheduling algorithm is not suited to the traffic pattern.

Symptom: Request load is unevenly distributed across backend servers.

Solution:

• For services with externalTrafficPolicy: Local, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler:"wrr" annotation to use weighted round-robin scheduling.

• For services using persistent connections, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler:"wlc" annotation to use weighted least connections scheduling. This prevents a single long-lived connection from receiving a disproportionate share of traffic.

503 error during application updates

Cause: Connection draining is not configured on the CLB listener, or graceful termination is not configured on the pod. During a rolling update, CLB may route traffic to pods that are already shutting down.

Symptom: A 503 error is returned when accessing the CLB instance during an application update.

Solution:

1. Configure connection draining on the CLB listener by adding the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain annotation. For the full list of listener annotations, see Common operations to manage listeners.

2. Configure readinessProbe and preStop on the pod: Example pod configuration:

   • `readinessProbe`: A pod is added to the service Endpoint only after it passes the readiness probe. After ACK detects the Endpoint change, it attaches the node to the CLB backend. Set the probe frequency, delay, and failure threshold to match your application's startup time. If the timeout is too short, pods may restart repeatedly.

   • `preStop` and `terminationGracePeriodSeconds`: Set preStop to the time your application needs to drain in-flight requests. Set terminationGracePeriodSeconds to at least 30 seconds longer than preStop.

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx
     namespace: default
   spec:
     containers:
     - name: nginx
       image: nginx
       # Liveness probe
       livenessProbe:
         failureThreshold: 3
         initialDelaySeconds: 30
         periodSeconds: 30
         successThreshold: 1
         tcpSocket:
           port: 5084
         timeoutSeconds: 1
       # Readiness probe
       readinessProbe:
         failureThreshold: 3
         initialDelaySeconds: 30
         periodSeconds: 30
         successThreshold: 1
         tcpSocket:
           port: 5084
         timeoutSeconds: 1
       # Graceful termination
       lifecycle:
         preStop:
           exec:
             command:
             - sleep
             - 30
     terminationGracePeriodSeconds: 60

CLB inaccessible from within the cluster

Cause: externalTrafficPolicy: Local is set on the service. With this setting, kube-proxy only forwards traffic to pods running on the same node as the request origin. If the node receiving the request has no backend pod for the service, the connection fails. This affects in-cluster traffic routed to the CLB address. For details, see kube-proxy adds external-lb address to node-local iptables rule.

Symptom: The CLB instance is accessible from outside the cluster but connection fails from within the cluster.

Solution: Use one of the following approaches:

• Access via ClusterIP or service name (recommended for in-cluster access): From within the cluster, use the service's ClusterIP or DNS name instead of the CLB address. For the Ingress service, the service name is nginx-ingress-lb.kube-system.

• Switch to `externalTrafficPolicy: Cluster`: This lets in-cluster traffic reach the service regardless of pod placement, but the client's original source IP is not preserved. To modify the Ingress service:

  If you use an Ingress CLB instance, pods can only access services exposed through the Ingress or CLB from the node where the Ingress pod is running.

  kubectl edit svc nginx-ingress-lb -n kube-system

• Use `externalTrafficPolicy: Cluster` with ENI pass-through (Terway only): If your cluster uses Terway with ENIs or multiple IPs per ENI, set externalTrafficPolicy: Cluster and add the service.beta.kubernetes.io/backend-type: "eni" annotation. This preserves the source IP and allows in-cluster access without issue. For more information, see Use annotations to configure a Classic Load Balancer (CLB) instance.

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

CLB inaccessible from outside the cluster

Cause: An access control list (ACL) is blocking the client IP, the CLB vServer group has no backend servers, or the CLB health check is failing.

Symptom: The CLB instance cannot be reached from outside the cluster.

Solution:

1. Check for service error events and resolve them. See Service error events and solutions.

   kubectl -n {your-namespace} describe svc {your-svc-name}

2. Check whether an ACL is configured on the CLB instance. If one is configured, verify that it allows inbound traffic from the client IP address. For ACL configuration details, see Resource Access Management.

3. Check whether the CLB vServer group is empty. If it is empty, verify that an application pod is associated with the service and running as expected. If the pod is unhealthy, resolve the pod issue first. See Troubleshoot pod issues.

4. Check whether the CLB listener health check is passing. If it is failing, verify that the application pod is responding correctly. For health check troubleshooting, see CLB health check FAQ.

Cannot connect to the backend HTTPS service

Cause: When a certificate is configured on the CLB listener, CLB terminates TLS and forwards decrypted HTTP traffic to the backend pod. If targetPort is set to an HTTPS port (for example, 443), the pod receives an HTTP request on an HTTPS port and rejects it with "The plain HTTP request was sent to HTTPS port."

Symptom: Connections to the backend service fail after configuring HTTPS on the CLB listener.

Solution: Set targetPort for the HTTPS listener port to the HTTP port that the backend pod listens on. For example, if Nginx listens on HTTPS port 443 internally, set targetPort to 80.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer