All Products
Search
Document Center

How can I troubleshoot the precheck failure that occurs before I upgrade the cloud controller manager (CCM)?

Last Updated: Nov 18, 2020

Problem description

A precheck failure may occur before you upgrade the cloud controller manager (CCM) of a Container Service for Kubernetes cluster.

Solutions

Alibaba Cloud reminds you that:

  • Before you perform operations that may cause risks, such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
  • You can modify the configurations and data of instances including but not limited to Elastic Compute Service (ECS) and Relational Database Service (RDS) instances. Before the modification, we recommend that you create snapshots or enable RDS log backup.
  • If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify such information in a timely manner.

Precautions

  • Before you upgrade the CCM, we recommend that you learn about the new features of the CCM. For more information, see Cloud Controller Manager.
  • If you modify the Service object of the CCM during the upgrade precheck, the precheck results may be invalid. In this case, perform the precheck again after the modification.
  • For CCM version v1.9.3.164-g2105d2e-aliyun and later, if you set externalTrafficPolicy to Local, the Service object automatically specifies a weight for a node based on the number of pods that are assigned to the node. For more information about the weight calculation rules, see How does CCM calculate node weights in Local mode? If your CCM version is earlier than v1.9.3.164-g2105d2e-aliyun, after you upgrade the CCM, the weights of the nodes that serve as the backend services of the Server Load Balancer (SLB) instance are automatically changed. This ensures load balancing among the pods. You can check the load distribution among the pods based on the traffic changes.
  • If you want to find the instance that fails a precheck, you can click the Instance Information tab. On this tab, the instance ID is the ID of the SLB instance, and the instance name consists of the namespace and service.

Error messages and solutions for the CCM upgrade precheck

Error Cause Solution
create a new load balancer An SLB instance is created when a Service object is created or synchronized from an existing Service object during the upgrade. You can use the console or run the following command to check whether the Service object is in the pending state. If the Service object is in the running state and a public IP address is used, perform the precheck again. If the Service object is in the pending state, submit a ticket.
kubectl get svc [$Svc_Name] -n [$Namespace]
Note:
  • [$Svc_Name] specifies the name of the Service object.
  • [$Namespace] specifies the namespace of the Service object.
modify the slb instance spec The specifications of the SLB instance are different from those specified for the Service object. In the annotation of the Service object, set the parameter service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec to the specifications of the SLB instance. For more information, see Use SLB to access services.
delete the load balancer The Service object does not exist, but the SLB instance with which the Service object is associated still exists. If you no longer require the SLB instance, delete the SLB instance in the SLB console. If you require the SLB instance, submit a ticket.
modify the slb internet spec The bandwidth or billing method of the SLB instance is different from that specified for the Service object.
  • If the specified bandwidth is invalid, in the annotation of the Service object, set the parameter service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth to the bandwidth of the SLB instance. For more information, see Use SLB to access services.
  • If the specified billing method is invalid, in the annotation of the Service object, set the parameter service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type to the billing method of the SLB instance. For more information, see Use SLB to access services.
remove backend servers The endpoint of the Service object is different from that specified for the backend server of the SLB instance.
  • If you have not modified the backend server configurations in the SLB console,
    do not change the endpoint of the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the backend server configurations in the SLB console,
    restore the VServer group configurations of the SLB instance and perform the precheck again.
add backend servers
stop listener The port of the Service object is different from that specified for the SLB listener.
  • If you have not modified the listener configurations in the SLB console,
    do not modify the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the listener configurations in the SLB console,
    restore the listener configurations of the SLB instance and perform the precheck again.
start listener
delete listener The port of the Service object is different from that specified for the SLB listener.
  • If you have not modified the listener configurations in the SLB console,
    do not modify the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the listener configurations in the SLB console,
    delete the added listeners and perform the precheck again.
create listener The port of the Service object is different from that specified for the SLB listener.
  • If you have not modified the listener configurations in the SLB console,
    do not modify the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the listener configurations in the SLB console,
    add the deleted listeners and perform the precheck again.
update listener The port of the Service object is different from that specified for the SLB listener.
  • If you have not modified the listener configurations in the SLB console,
    do not modify the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the listener configurations in the SLB console,
    restore the listener configurations of the SLB instance, such as the certificate, access control, health check, and cookie settings. Then, perform the precheck again. For more information, see Use SLB to access services.

create VServerGroup

The endpoint of the Service object is different from that specified for the VServer group of the SLB instance.
  • If you have not modified the VServer group configurations in the SLB console,
    do not change the endpoint of the Service object during the upgrade precheck and perform the precheck again.
  • If you have manually modified the backend server configurations in the SLB console,
    restore the VServer group configurations of the SLB instance and perform the precheck again.
delete VServerGroup
add VServerGroup backends
remove VServerGroup backends

Application scope

  • Alibaba Cloud Container Service for Kubernetes