Container Compute Service:Troubleshoot service issues

Step 1: Verify pods and endpoints

Run the following command to check whether pods are running and endpoints are populated:

kubectl -n {your-namespace} get pods -o wide
kubectl -n {your-namespace} get endpoints {your-svc-name}

If no endpoints exist, check whether the pods are running and whether the pod labels match the Service selector.

Step 2: Find the Service associated with the SLB instance

Run the following command to find the Service by the SLB IP address:

kubectl get svc -A |grep -i LoadBalancer|grep ${XXX.XXX.XXX.XXX}  #XXX.XXX.XXX.XXX is the IP address of the SLB instance.

Step 3: Check for Service error events

Run the following command to check for Service error events:

kubectl -n {your-namespace} describe svc {your-svc-name}

• If error events are present, see Service errors and solutions.

• If no error events are present, see Troubleshooting.

Step 4: Escalate if needed

If the issue persists after troubleshooting, join the ACS DingTalk group for technical support.

SLB creation errors

SLB reuse errors

Network errors

Billing and quota errors

Configuration errors

Issues when accessing an SLB instance

Issues related to SLB configurations

Issues related to SLB deletion

The SLB instance does not evenly distribute traffic

Cause

The scheduling algorithm specified for the SLB instance is improper.

Issue

Traffic is not evenly distributed to the backend servers of an SLB instance.

Solution

If long-lived connections are established to your Service, set the scheduling algorithm to Weighted Least Connections (WLC) by adding the following annotation:

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler:"wlc"

503 error during application updates

Cause

Connection draining is not configured for the SLB listener, or graceful shutdown is not configured for the pod.

Issue

The 503 error occurs when accessing the SLB instance during application updates.

Solution

1. Add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain annotation to configure connection draining for the SLB listener. For more information about the annotation, see Common operations to manage listeners.

2. Set the preStop and readinessProbe parameters for the pod based on the network mode of the pod. Pod configuration example:

   • readinessProbe checks whether the container is ready to accept network traffic. The pod is added to the endpoint only after it passes the readiness probing. The pod is then attached to the SLB instance after ACS detects the endpoint update. Set a proper probing interval, delay period, and unhealthy threshold for readinessProbe. Some applications may take longer to start. If you specify a short time period, the application pods may repeatedly restart.

   • Set the value of preStop to a time period that the application pods require to handle the remaining requests. Set the value of terminationGracePeriodSeconds to a time period that is 30 seconds longer than preStop.

      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx
        namespace: default
      spec:
        containers:
        - name: nginx
          image: nginx
          # Liveness probing
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            tcpSocket:
              port: 5084
            timeoutSeconds: 1
          # Readiness probing
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            tcpSocket:
              port: 5084
            timeoutSeconds: 1
          # Graceful shutdown
          lifecycle:
            preStop:
              exec:
                command:
                - sleep
                - "30"
        terminationGracePeriodSeconds: 60

The SLB instance cannot be accessed from outside the cluster

Cause

Access control list (ACL) rules are configured for the SLB instance, or the SLB instance is not running as expected.

Issue

The SLB instance cannot be accessed from outside the cluster.

Solution

1. Run the following command to query Service events and troubleshoot errors. For more information, see Service errors and solutions.

      kubectl -n {your-namespace} describe svc {your-svc-name}

2. Check whether ACL rules are configured for the SLB instance. If ACL rules are configured, check whether the client IP address is allowed to access the SLB instance. For more information about how to configure ACL rules, see Access control.

3. Check whether the SLB instance is associated with a vServer group. If no vServer group is associated, check whether the application pods are associated with the Service and whether they are running normally. If the pods are not running as expected, identify the causes and troubleshoot the errors. For more information, see Pod troubleshooting.

4. Check whether unhealthy backend servers are detected by the SLB listeners. If unhealthy backend servers are detected, check whether the application pods are running normally. For more information about health checks for SLB, see Execute a health check script.

5. If the issue persists, join the ACS DingTalk group for technical support.

Backend HTTPS services cannot be accessed

Cause

After you specify the certificate information in the SLB instance, the SLB instance decrypts HTTPS requests and sends HTTP requests to the backend pods.

Issue

Backend HTTPS services cannot be accessed.

Solution

Set targetPort to an HTTP port in the Service. For example, if the HTTPS port is 443 in the following NGINX Service, change the value of targetPort to 80.

Example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer