Container Service for Kubernetes:Update the Kubernetes version of an ACK cluster

Precautions

Usage notes

• ACK guarantees the stability of the three latest Kubernetes minor versions. You can update ACK clusters of the previous two minor versions to the latest minor version. For example, you can update ACK clusters from Kubernetes V1.20 or Kubernetes V1.22 to Kubernetes V1.24. The stability of ACK clusters that run other earlier Kubernetes minor versions is not guaranteed and you may fail to update these clusters to the latest minor version. However, these issues do not adversely affect your applications deployed in these clusters. We recommend that you update your ACK clusters at the earliest opportunity. For more information about Kubernetes versions supported by ACK, see Support for Kubernetes versions.
• Applications that run in the cluster are not interrupted during the update. Applications that are strongly reliant on the API server may be temporarily interrupted. We recommend that you perform the update during off-peak hours.
• The system runs a precheck before updating an ACK cluster. The cluster can be updated only after it passes the precheck.
• You can update the Kubernetes version of an ACK cluster only to the next version. For example, the Kubernetes version of an ACK cluster is 1.18. If you want to update the Kubernetes version to 1.24, you must manually update the version to 1.20, 1.22, and then 1.24 in sequence.
• The cluster update policy defines how the ACK cluster is updated. The default policy is batch update. Batch update is performed during the worker node update phase to update worker nodes in batches. The policy works in the following manner:
  • Multiple node pools are updated one after one.
  • The nodes in a node pool are updated in batches. The first batch includes one node. The number of nodes increases based on the powers of two in subsequent batches. The batch update policy still applies after you resume a paused update process.
  • The system updates no more than 10 nodes in each batch.

Considerations

• To update a cluster, you need to use Yum to download the required software packages. If you have modified the network configuration of the nodes or used a custom OS image, make sure that Yum can function as expected on the nodes. You can run the yum makecache command to perform a check.
• Do not add nodes, remove nodes, or perform other operations on the cluster during the update process.
• If you specify a custom operating system image, we cannot guarantee a successful node update because custom images are not strictly validated by ACK.
• If you have modified the configurations of the ACK cluster, such as swap partition changes or kubelet configuration changes by using the CLI, the update may fail or your custom configurations may be lost.
• After the update is completed, we recommend that you update kubectl on your on-premises machine. For more information about how to install kubectl, see Install kubectl. Otherwise, the kubectl version may be incompatible with the API server version. As a result, an error similar to invalid object doesn't have additional properties may occur.
• For ACK clusters that run Kubernetes 1.20 or later, the system checks whether deprecated APIs are used in the clusters. For more information, see Deprecated APIs.
• If the resource usage of the cluster is excessively high, the system may fail to schedule the evicted pods promptly when the node update fails. We recommend that you reserve resources for the cluster nodes. Do not use more than 50% of CPU resources or more than 70% of memory resources.

Kubernetes version descriptions

• Kubernetes 1.24 no longer supports the Docker runtime. Before you update a cluster to Kubernetes 1.24, you need to use the node pool feature to update the container runtime of the nodes from Docker to containerd in batches. For more information, see Change the container runtime from Docker to containerd. If you use an ACK Pro cluster, the master nodes automatically update from Docker to containerd during the cluster update. All containers on the master nodes will be recreated. Back up your custom containers on the master nodes before you update the cluster.
• ACK clusters that run Kubernetes 1.20 do not support the selfLink field. If FlexVolume is used and alicloud-nas-controller is installed in your cluster, you must update the image version of alicloud-nas-controller to v1.14.8.17-7b898e5-aliyun or later before you can update an earlier Kubernetes version to 1.20.
• Before you update an ACK cluster from Kubernetes 1.16 to Kubernetes 1.18, read Usage notes for updating CSI block volumes if the application in the cluster uses a disk volume whose type is Block Volume.
• Before you update an ACK cluster to Kubernetes 1.14, make sure that the existing configurations allow access to the IP addresses of the Server Load Balancer (SLB) instances that are used by LoadBalancer Services. For more information, see What Can I Do if the Cluster Cannot Access the IP Address of the SLB Instance Exposed by the LoadBalancer Service.
• Object Storage Service (OSS) volumes that are mounted to the Kubernetes cluster by using FlexVolume 1.11.2.5 or earlier will be remounted during the update. You must recreate the pods that use OSS volumes after the update is completed.
• After you update an ACK cluster to Kubernetes 1.18, ACK automatically configures resource reservation by default. Workloads on cluster nodes may be evicted when the resource usage of the nodes is high and resource reservation is not configured. For more information about how to configure resource reservation, see Resource reservation policy.

How ACK clusters are updated

The cluster update process consists of the precheck, control plane update, and node update. The following sections describe the cluster update process in detail.

Control plane update

Node update

Step 1: Perform a precheck

Before you update a cluster, you must check the health status of the cluster and make sure that the cluster meets the update requirements.

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, find the cluster on which you want to perform a check and choose More > Cluster Check in the Actions column.
3. In the left-side navigation pane of the Container Intelligence Service page, choose Cluster Check > Upgrade Check.
4. On the Upgrade Check page, click Start.
5. In the Upgrade Check panel, select the check box under Warning and click Start.
   After the precheck is completed, click Details.

   • If the result is normal in the report, the cluster passes the precheck and you can update the cluster.
   • If the result is abnormal in the report, follow the suggestions displayed on the page to fix the issues.
     Note

     • The precheck is performed only before clusters are updated. A cluster can still run as normal and the cluster status does not change if the cluster fails to pass the precheck.
     • If your cluster runs Kubernetes 1.20 or later, the precheck checks whether deprecated APIs are used in your cluster. The precheck result is for reference only and does not determine whether the cluster is updatable. For more information, see Deprecated APIs.

Step 2: Update the cluster

1. Log on to the ACK console and click Clusters in the left-side navigation pane.
2. On the Clusters page, find the cluster that you want to upgrade and choose More > Upgrade Cluster in the Actions column.
3. Click Upgrade.
   Note If you click Precheck, the system performs a precheck on the cluster as described in Step 1: Perform a precheck.
4. In the message that appears, click Confirm.
   You can view the progress of the update.

   Note

   • If you want to pause the update, click Pause. After you pause the update, you can click Continue to resume the update process. Do not perform operations after you pause the update of a cluster. In addition, we recommend that you resume and complete the update at your earliest convenience. If the update is paused for more than 7 days, the system automatically terminates the update process. The events and log data that are generated during the update process are also deleted.
   • After the update is paused, you can click Cancel to cancel the update. After you cancel the update, the system will complete updating the nodes in the current batch and skip the nodes that have not been updated. After the update is canceled, you cannot roll back the nodes that have been updated. However, you can continue to update the nodes that have not been updated.

Deprecated APIs

If your cluster runs Kubernetes 1.20 or later, the precheck checks whether deprecated APIs are used in your cluster. After the precheck is complete, the deprecated APIs that are used by your cluster are displayed on the precheck result page.

Troubleshoot cluster update failures

What do I do if the master node update of an ACK dedicated cluster times out?

The self-signed server certificate of the admission webhook component does not contain the Subject Alternative Name field. As a result, the master component fails to start up.

Run the following commands to check whether the self-signed server certificate of the admission webhooks contains the Subject Alternative Name field. You need to run the following commands on nodes that have kubectl configured.

1. Run the following command to query the admission webhooks in the cluster:

   kubectl get mutatingwebhookconfigurations

   Expected output:

   NAME                                      WEBHOOKS   AGE
   ack-node-local-dns-admission-controller   1          27h

2. Run the following command to query the Service configured for an admission webhook:

   kubectl get mutatingwebhookconfigurations ack-node-local-dns-admission-controller -oyaml | grep service -A 5
       service:
         name: ack-node-local-dns-admission-controller
         namespace: kube-system
         path: /inject
         port: 443
     failurePolicy: Ignore

3. Run the following command to query the cluster IP of the Service:

   kubectl -n kube-system get service ack-node-local-dns-admission-controller

   Expected output:

   NAME                                      TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)   AGE
   ack-node-local-dns-admission-controller   ClusterIP   192.168.XX.XX   <none>        443/TCP   27h

4. Run the following command to use the cluster IP to access the admission webhook, obtain its certificate, and check whether the Subject Alternative Name field exists:

   openssl s_client -connect 192.168.XX.XX:443 -showcerts </dev/null 2>/dev/null|openssl x509 -noout -text

   

What do I do if the etcd update of an ACK dedicated cluster fails?

Cloud Assistant is unavailable.

Cloud Assistant becomes unavailable. As a result, the update command fails to be issued. Update Cloud Assistant and try again. For more information about how to update Cloud Assistant, see Upgrade or disable upgrades for the Cloud Assistant client.

What do I do if the PLEG module of nodes is unhealthy?

You need to restart the nodes and initiate the update again when the containers or container runtime does not respond.