This topic provides answers to some frequently asked questions about Services in Container Service for Kubernetes (ACK) clusters.

FAQ about CCM updates

How do I troubleshoot failures to update the CCM?

Others

How is session persistence implemented in Kubernetes Services?

Why is no event generated for the synchronization between a Service and an SLB instance?

If no event is generated after you run the kubectl -n {your-namespace} describe svc {your-svc-name} command, check the version of the CCM.
  • If the CMM version is earlier than v1.9.3.276-g372aa98-aliyun, no event is generated for the synchronization between a Service and an SLB instance. We recommend that you update the CCM version. For more information about how to view and update the CCM version, see Manually update the CCM.
  • If the CMM version is v1.9.3.276-g372aa98-aliyun or later, Submit a ticket.

How do I handle an SLB instance that remains in the Pending state?

  1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view the events.
  2. Troubleshoot the errors that are reported in the events. For more information about how to troubleshoot errors that are reported in the events, see Errors and solutions.

What do I do if the vServer groups of an SLB instance are not updated?

  1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view the events.
  2. Troubleshoot the errors that are reported in the events. For more information about how to troubleshoot errors that are reported in the events, see Errors and solutions.

What do I do if the annotations of a Service do not take effect?

  1. Perform the following steps to view the errors:
    1. Run the kubectl -n {your-namespace} describe svc {your-svc-name} command to view the events.
    2. Troubleshoot the errors that are reported in the events. For more information about how to troubleshoot errors that are reported in the events, see Errors and solutions.
  2. If no errors are reported, you can resolve the issue based on the following scenarios:
    • Make sure that the CCM version meets the requirements of the annotations. For more information about the correlation between annotations and CCM versions, see Common annotations.
    • On the Services page, find the Service that you want to manage and click View in YAML in the Actions column. In the panel that appears, check whether annotations are configured for the Service. If annotations are not configured for the Service, you must configure annotations for the Service.

      For more information about how to configure annotations, see Use annotations to configure load balancing.

      For more information about how to view the list of Services, see Manage Services.

    • Verify that the annotations are valid.

Why is the configuration of an SLB instance modified?

When specific conditions are met, the CCM calls a declarative API to update the configuration of an SLB instance based on the Service configuration. If you modify the configurations of an SLB instance in the SLB console, the CCM may overwrite the changes. We recommend that you use annotations to configure an SLB instance. For more information about how to configure annotations for an SLB instance, see Use annotations to configure load balancing.
Notice If the SLB instance is created and managed by the CCM, we recommend that you do not modify the configuration of the SLB instance in the SLB console. Otherwise, the CCM may overwrite the configuration and the Service may be unavailable.

Why does the system fail to use an existing SLB instance for more than one Services?

  • Check the version of the CCM. If the version is earlier than v1.9.3.105-gfd4e547-aliyun, the CCM cannot use existing SLB instances for more than one Services. For more information about how to view and update the CCM version, see Manually update the CCM.
  • Check whether the reused SLB instance is created by the cluster. The SLB instance cannot be reused if it is created by the cluster.
  • Check whether the SLB instance is used by the API server. The SLB instance cannot be reused if it is used by the API server.
  • If the SLB instance is an internal-facing SLB instance, check whether the SLB instance and the cluster are deployed in the same virtual private cloud (VPC). The SLB instance cannot be reused if they are deployed in different VPCs.

Why is no listener created when I reuse an existing SLB instance?

Make sure that you set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners to true in the annotation settings. If you do not set the value to true, no listener is automatically created.
Note The following list explains why the CCM does not overwrite the listeners of an existing SLB instance:
  • If the listeners of the SLB instance are associated with applications, service interruptions may occur after the configurations of the listeners are overwritten.
  • The CCM supports limited backend configurations and cannot handle complex configurations. If you require complex backend configurations, you can manually configure listeners in the SLB console without overwriting the existing listeners.
In both cases, we recommend that you do not overwrite the listeners of existing SLB instances. However, you can overwrite an existing listener if the port of the listener is no longer in use.

How do I troubleshoot failures to update the CCM?

For more information about how to resolve CCM update failures, see CCM update failures.

Why does the cluster fail to access the IP address of the SLB instance?

For more information about why does a cluster fail to access the IP address of the SLB instance, see Kubernetes clusters cannot access the IP address of the SLB instance.

When is the SLB instance automatically deleted?

The system automatically deletes the SLB instance of the LoadBalancer Service under specific conditions if the SLB instance is created by the CCM. The following table describes the conditions under which the SLB instance is automatically deleted.
Condition When an SLB instance created by the CMM is used When an existing SLB instance is used
Delete the LoadBalancer Service Delete the SLB instance Retain the SLB instance
Change the Service type of the LoadBalancer Service Delete the SLB instance Retain the SLB instance

Is the SLB instance associated with the LoadBalancer Service automatically deleted after I delete the Service?

If the SLB instance is reused, it is not deleted together with the Service. If the SLB instance is not reused, it is deleted. If a Service contains the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: {your-slb-id} annotation, the corresponding SLB instance is reused.

If you change the type of the Service, for example, from LoadBalancer to NodePort, and the SLB instance is created by the CMM, the SLB instance is automatically deleted.

What do I do if I accidentally delete an SLB instance?

  • Scenario 1: What do I do if I accidentally delete the SLB instance of the API server?

    The deleted SLB instance cannot be restored. You must create a new SLB instance. For more information about how to attach the policy to the RAM role, see Create an ACK Pro cluster.

  • Scenario 2: What do I do if I delete the SLB instance of an Ingress?

    Perform the following steps to recreate the SLB instance:

    1. Log on to the the ACK console.
    2. In the left-side navigation pane, click Clusters.
    3. On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
    4. Choose Network > Services.
    5. On the top of the Services page, select kube-system from the Namespace drop-down list. Then, find nginx-ingress-lb in the Services list and click View in YAML in the Actions column.
      If you cannot find nginx-ingress-lb in the Services list, use the following template to create a Service named nginx-ingress-lb:
      apiVersion: v1
      kind: Service
      metadata:
        labels:
          app: nginx-ingress-lb
        name: nginx-ingress-lb
        namespace: kube-system
      spec:
        externalTrafficPolicy: Local
        ports:
        - name: http
          port: 80
          protocol: TCP
          targetPort: 80
        - name: https
          port: 443
          protocol: TCP
          targetPort: 443
        selector:
          app: ingress-nginx
        type: LoadBalancer
    6. In the Edit YAML dialog box, delete the content in the status field. Then, click Update. This way, the CCM creates a new SLB instance.
  • Scenario 3: What do I do if I delete an SLB instance that is configured to handle workloads?
    • If you no longer need the Service that is associated with the SLB instance, delete the Service.
    • If you want to keep the Service, perform the following steps:
      1. Log on to the the ACK console.
      2. In the left-side navigation pane, click Clusters.
      3. On the Clusters page, find the cluster that you want to manage and click its name or click Details in the Actions column.
      4. In the left-side navigation pane of the cluster details page, choose Network > Services.
      5. On the top of the Services page, select All Namespaces from the Namespace drop-down list. Then, find the Service in the Services list and click View in YAML in the Actions column.
      6. In the Edit YAML dialog box, delete the content in the status field. Then, click Update. This way, CCM creates a new SLB instance.

How do I rename an SLB instance when the CCM version is V1.9.3.10 or earlier?

For CCM versions later than V1.9.3.10, a tag is automatically added to the SLB instances in the cluster. You need only to change the value if you want to rename an SLB instance. For CCM V1.9.3.10 and earlier, you must manually add a specific tag to an SLB instance if you want to rename the SLB instance.
Note
  • You can rename an SLB instance by adding a tag to the instance only if the CCM version is V1.9.3.10 or earlier.
  • The Service type is LoadBalancer.
  1. Log on to a master node in an ACK cluster. For more information, see Connect to ACK clusters by using kubectl.
  2. Run the kubectl get svc -n ${namespace} ${service} command to view the Service type and IP address of the Service. Service type
    Note Replace namespace with the cluster namespace and service with the Service name.
  3. Run the following command to create the tag that you want to add to the SLB instance:
    kubectl get svc -n ${namespace} ${service} -o jsonpath="{.metadata.uid}"awk -F "-" '{print "kubernetes.do.not.delete: "substr("a"$1$2$3$4$5,1,32)}'
    tag
  4. Log on to the SLB console, select the region where the SLB instance is deployed, and then find the specified SLB instance based on the IP address that is returned in Step 2.
  5. Add the tag that is generated in Step 3 to the SLB instance. Callout 1 in the preceding figure is the tag key, and callout 2 is the tag value. For more information, see Create and manage a CLB instance.

How does the CCM calculate node weights in Local mode?

In this example, pods with the app=nginx label are deployed on three ECS instances. In the following figure, when externalTrafficPolicy is set to Local, the pods provide services for external users by using Service A. The following sections describe how node weights are calculated.
CCM2
  • For CCM versions earlier than V1.9.3.164-g2105d2e-aliyun

    For CCM versions that are earlier than V1.9.3.164-g2105d2e-aliyun, the following figure shows that the weight of each ECS instance in Local mode is 100. This indicates that traffic loads are evenly distributed to the ECS instances. However, the load amounts of the pods are different because the pods are unevenly deployed on the ECS instances. For example, the pod on ECS 1 takes the heaviest load and the pods on ECS 3 take the lightest load.

    CCM3
  • For CCM versions that are later than V1.9.3.164-g2105d2e-aliyun but earlier than V1.9.3.276-g372aa98-aliyun

    For CCM versions that are later than V1.9.3.164-g2105d2e-aliyun but earlier than V1.9.3.276-g372aa98-aliyun, the node weights are calculated based on the number of pods deployed on each node, as shown in the following figure. The weights of the ECS instances are 16, 33, and 50 based on this calculation. Therefore, traffic loads are distributed to the ECS instances at the ratio of approximately 1:2:3.

    The node weight is calculated based on the following formula.Formula
    ccm4
  • For CCM V1.9.3.276-g372aa98-aliyun and later

    The weights of pods are slightly imbalanced due to the precision of the calculation formula. For CCM V1.9.3.276-g372aa98-aliyun and later, the weight of each node equals the number of pods deployed on the node. In the following figure, the weights of the ECS instances are 1, 2, and 3. Traffic loads are distributed to the ECS instances at the ratio of 1:2:3. This way, the pods have a more balanced load than the pods in the preceding figure.

    The node weight is calculated based on the following formula.Node weight
    node

Service errors and solutions

The following table describes how to fix the errors that occur in Services.
Error message Description and solution
The backend server number has reached to the quota limit of this load balancers

The quota on the backend servers of the SLB instance is insufficient.

Solution: You can use the following methods to reduce the number of vServer groups that are required.
  • By default, an SLB instance supports 200 backend servers. For more information about how to query and increase the quota, see Quota Management page in the SLB console.
  • We recommend that you set externalTrafficPolicy of the SLB instance to Local (externalTrafficPolicy: Local). The system may create a large number of backend servers in Cluster mode. If you want to use the Cluster mode, we recommend that you use the label service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label to specify the backend servers that you want to use. This reduces the number of backend servers that are required. For more information about how to associate backend servers with an SLB instance, see Use annotations to configure load balancing.
  • If multiple Services share an SLB instance, all backend servers that are used by the Services are counted. We recommend that you create an SLB instance for each Service.
The loadbalancer does not support backend servers of eni type Shared-resource SLB instances do not support elastic network interfaces (ENIs).
Solution: If you want to specify an ENI as a backend server, create a high-performance SLB instance. Add the annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small" annotation to the Service.
Notice Make sure that the annotations that you add meet the requirements of the CCM version. For more information about the correlation between annotations and CCM versions, see Common annotations.
There are no available nodes for LoadBalancer No backend server is associated with the SLB instance. Check whether pods are associated with the Service and whether the pods run as normal.
Solution:
  • If no pod is associated with the Service, associate application pods with the Service.
  • If the pods that are associated with the Service do not run as normal, identify the cause and troubleshoot the error. For more information, see Troubleshoot application errors in ACK.
  • If no backend server is associated with the SLB instance, but the pods run as normal, check whether the pods are deployed on master nodes. If the pods are deployed on master nodes, evict the pods to worker nodes. If the pods are not deployed on master nodes, Submit a ticket.
  • alicloud: not able to find loadbalancer named [%s] in openapi, but it's defined in service.loaderbalancer.ingress. this may happen when you removed loadbalancerid annotation
  • alicloud: can not find loadbalancer, but it's defined in service

The system fails to associate a Service with the SLB instance.

Solution: Log on to the SLB console, select the region where the SLB instance is deployed, and then find the SLB instance based on the EXTERNAL-IP of the Service.
  1. If the SLB instance does not exist and the Service is no longer used, delete the Service.
  2. If the SLB instance exists, perform the following steps:
    1. If the SLB is created in the SLB console, add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id annotation in the Service. For more information, see Use annotations to configure load balancing.
    2. If the SLB instance is automatically created by the CCM, check whether the kubernetes.do.not.delete label is added to the SLB instance. If the label is not added to the SLB instance, add the label to the SLB instance. For more information, see How do I rename an SLB instance when the CCM version is V1.9.3.10 or earlier?.
ORDER.ARREARAGE Message: The account is arrearage. Your account has overdue payments.
PAY.INSUFFICIENT_BALANCE Message: Your account does not have enough balance. Your account balance is insufficient. Top up your account balance.
Status Code: 400 Code: Throttlingxxx API throttling is triggered for SLB.
Solution:
  1. Go to the Quota Management page in the SLB console and check whether the SLB resource quotas are sufficient.
  2. Run the following command to check whether errors occur in the Service. If errors occur in the Service, refer to the information provided in this table to troubleshoot the errors.
    kubectl -n {your-namespace} describe svc {your-svc-name}
Status Code: 400 Code: RspoolVipExist Message: there are vips associating with this vServer group. The listener that is associated with the vServer group cannot be deleted.
Solution:
  1. Check whether the annotation of the Service contains the ID of the SLB instance. Example: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: {your-slb-id}.

    If the annotation of the Service contains the ID of the SLB instance, the SLB instance is reused.

  2. Log on to the SLB console and delete the listener that uses the Service port. For more information about how to delete listeners for an SLB instance, see Manage forwarding rules for a listener.
Status Code: 400 Code: NetworkConflict The reused internal-facing SLB instance and the cluster are not deployed in the same virtual private cloud (VPC).

Solution: Make sure that your SLB instance and the cluster are deployed in the same VPC.

Status Code: 400 Code: VSwitchAvailableIpNotExist Message: The specified VSwitch has no available ip. The idle IP addresses in the vSwitch are insufficient.

Solution: Specify another vSwitch in the same VPC by using the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}" annotation.

The specified Port must be between 1 and 65535. The targetPort field does not support STRING type values in ENI mode.

Solution: Set the targetPort field in the Service YAML file to a value of the INTEGER type or update the CCM. For more information about how to update the CCM, see Manually update the CCM.

Status Code: 400 Code: ShareSlbHaltSales Message: The share instance has been discontinued. By default, earlier versions of CCM automatically create shared-resource SLB instances, which are no longer available for purchase.

Solution: Manually update the CCM.

can not change ResourceGroupId once created You cannot modify the resource group of an SLB instance after the resource group is created.

Solution: Delete the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id:"rg-xxxx" annotation from the Service.

can not find eniid for ip x.x.x.x in vpc vpc-xxxx The specified IP address of the ENI cannot be found in the VPC.

Solution: Check whether the service.beta.kubernetes.io/backend-type: eni annotation is added to the Service. If the annotation is added to the Service, check whether Flannel is used as the network plug-in of the cluster. If Flannel is used, delete the annotation from the Service. Flannel does not support ENI mode.

External traffic policies supported by Services

Services support the following external traffic policies: