When you update the Kubernetes version of a cluster, update the control plane first, and then update the node pools during off-peak hours. A node pool update includes updating the kubelet and the container runtime. Before starting the update, ACK performs a precheck to identify and report potential risks for a smooth process.
Notes
Node scaling
If node scaling is enabled for the cluster, the cluster-autoscaler component is automatically updated to the latest version after the cluster update. This ensures that the auto scaling feature works correctly. After the cluster update, verify that the cluster-autoscaler component is running the correct version. For more information, see Enable node auto scaling.
During a cluster update, nodes whose Scaling Mode is set to Swift may fail to update because they are shut down. If a node in Swift is not updated after the cluster update is complete, we recommend that you manually remove the node.
After you update a cluster to Kubernetes 1.18, ACK configures node resource reservation by default. If resource reservation is not configured and node resource usage is high, pods may fail to reschedule promptly after being evicted. We recommend that you reserve resources for your nodes. For optimal performance, CPU utilization should not exceed 50% and memory utilization should not exceed 70%. For more information, see Node resource reservation policy.
In clusters that run Kubernetes 1.24 or earlier, if a pod in a workload is configured only with a startup probe, the pod may enter the NotReady state for a short period after the kubelet restarts. We recommend that you use a multi-replica deployment strategy to distribute workloads across multiple nodes. This ensures that enough pods remain available during a node restart.
If a pod accesses another pod on the same node by using the IP address of the SLB instance exposed by a
LoadBalancerservice, and theexternalTrafficPolicyof the service is set toLocal, the two pods may no longer reside on the same node after a node is replaced. This can cause network interruptions.Custom operating system (OS) images are not rigorously validated by ACK. ACK cannot guarantee a successful update for clusters that use a custom OS image.
During a cluster update, yum is used to download the required software packages. If you have modified the network configurations of your nodes or used a custom OS image, ensure that yum functions correctly on the nodes. You can run the
yum makecachecommand to verify its status.If you have made custom configuration changes to the cluster, such as enabling the SWAP partition or modifying kubelet or container runtime configurations from the command line, the update process may fail or your custom configurations may be overwritten.
When you update a node by replacing its system disk, ACK performs node draining. Pods are evicted from the node to other available nodes according to the configured PodDisruptionBudget (PDB). To ensure high service availability, we recommend that you use a multi-replica deployment strategy to distribute workloads across multiple nodes. You should also configure a PDB for critical services to control the number of pods that can be simultaneously disrupted.
The default timeout for node draining is 30 minutes. If pod migration does not complete within this time, ACK terminates the update to ensure service stability.
When you update a node by replacing its system disk, ACK reinitializes the node based on the current configuration of the node pool, such as the node logon method, labels, taints, OS image, and container runtime version. Typically, you update a node pool configuration by editing the node pool. If you have modified a node by using other methods, these changes will be overwritten during the update.
If a pod on a node uses a HostPath volume that points to the system disk, the data in that directory is lost after an update by replacing the system disk.
During a node pool update, only scale-out operations are supported. Scale-in operations are not supported.
If a node is an unmanaged node (a worker node that is not managed by any node pool), you must first migrate it to a node pool. For more information, see Migrate unmanaged nodes to a node pool.
When you update an ACK cluster, you cannot update Lingjun node pools.
When you upgrade a node pool in a cluster of version 1.31 or earlier, the process also upgrades the NVIDIA Device Plugin and resets any of its non-standard configurations.
Features
A node pool update includes updating the kubelet and the container runtime.
Kubelet update: Upgrades the kubelet on the nodes in a node pool to the same version as the control plane. By default, ACK performs an in-place update.
Container runtime update: When a new container runtime version is released, you can upgrade the container runtime on your nodes to the latest version.
When you migrate the container runtime from Docker to containerd, the nodes in the node pool are updated by replacing their system disks. This erases all system disk content. Before you start the update, make sure to back up all important data from the system disk. For more information, see Migrate the node container runtime from Docker to containerd.
Except for ContainerOS nodes, ACK performs an in-place update by default when you upgrade from one version of containerd to a newer one. The
/etc/containerd/config.tomlfile on the node is replaced with a new version provided by ACK.ImportantContainerOS nodes do not support in-place updates for containerd. You can only update them by replacing the system disk. For the upgrade procedure, see Upgrade a ContainerOS version earlier than 3.4 to the latest version.
During a container runtime update, pod probers and lifecycle hooks may fail. Pods might also restart in place.
In clusters that run Kubernetes 1.24 or earlier, when you update Docker to a newer version, ACK updates the nodes in the node pool by replacing their system disks by default. This erases all system disk content. Before you start the update, make sure to back up all important data from the system disk.
Procedure
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
On the Node Pools page, find the node pool that you want to update and choose
> Kubelet Update in the Actions column. Configure the parameters as described in the following table.Parameter
Description
Kubelet Update Information
View the current kubelet version and select the target version.
Runtime Update Information
View the current runtime version and select the target runtime version.
When you migrate the node container runtime from Docker to containerd, the nodes in the node pool are updated by replacing their system disks. This process erases all content on the system disks.
If your cluster runs Kubernetes 1.22 and the installed containerd version is 1.6.34 (a relatively new version), updates are not supported.
Update Nodes
Specify the nodes to update: all nodes or specific nodes.
Update Method
Select an update method. In-place Upgrade and Upgrade by Replacing System Disk are supported. For information about the update logic and process, see Reference information: In-place updates and updates by replacing system disks.
In-place Upgrade: Updates the required components on the existing nodes. This method does not replace the system disk or reinitialize the node, and does not affect data on the node.
Upgrade by Replacing System Disk: Reinitializes the node by replacing its system disk. The node's instance properties, such as its name, ID, and IP address, remain unchanged, but all data on the system disk is erased. Data on separately mounted data disks is not affected.
Ignore Warnings
Specifies whether to proceed with the update if the precheck reports warnings. An example of a warning-level item is that a pod in the node pool uses a
hostPaththat points to the system disk.Batch Update Policy
Maximum Number of Nodes per Batch
ACK updates nodes in batches based on the maximum number of concurrent nodes that you specify. For more information about the update process, see Reference information: In-place updates and updates by replacing system disks.
Automatic Pause Policy
The pause policy for the node update process.
Interval Between Batches
If you set Automatic Pause Policy to Do not set, you can specify a time interval between update batches. The value ranges from 5 to 120 minutes.
Auto Snapshot
If your node's system disk contains important business data, you can choose to create a snapshot for the node before updating the node pool. This lets you back up and restore node data. Creating snapshots incurs fees. For more information, see Snapshot pricing. The creation progress is updated in real time. If the snapshot is no longer needed after the update, delete it promptly.
NoteIf you select Upgrade By Replacing System Disk as the update method, we recommend that you enable Auto Snapshot. Creating snapshots incurs fees. For more information, see Snapshot pricing.
After you complete the configurations, click Precheck. After the precheck passes, follow the on-screen instructions to start the update.
NoteIf the precheck fails or returns warnings, see Solutions for failed check items or view the Pre-check Report as prompted to troubleshoot the issues.
During the update, you can perform the following operations as prompted:
Pause: Pausing a cluster update is an intermediate state. We recommend that you do not perform other operations on the cluster during this time and complete the update process as soon as possible. If an update remains paused for more than seven days, ACK automatically terminates the process and clears all related events and logs.
After you pause the update, you cannot roll back the kubelet and container runtime versions on already-updated nodes.
Cancel: Cancels the update. After you click Cancel, you cannot roll back the kubelet and container runtime versions on already-updated nodes.
After the update, you can go to the Nodes page, click a node name, and then click the Basic Information tab to verify that the kubelet and container runtime versions are correct.
In-place updates and system disk replacements
In-place and replacement update process
The following section describes the processes for in-place updates and updates by replacing system disks. ACK updates nodes in a node pool in batches based on the specified maximum number of concurrent nodes. The number of nodes in each batch increases exponentially (1, 2, 4, 8, and so on) until it reaches the specified maximum. After that, each subsequent batch proceeds with the maximum number of nodes. For example, if you set the maximum number of concurrent nodes to 4, one node is updated in the first batch, two nodes in the second, and four nodes in the third and all subsequent batches.
The figure below shows the batch update process where the maximum number of concurrent nodes is N. The number of nodes updated in each batch is 1, 2, 4, 8, and so on, up to N.
In-place update logic
ACK performs an update precheck. If a critical container issue is detected, such as a failure to process ttrpc requests or a container process not responding to signals, the update stops.
The current states of containers and pods are saved to a temporary tmp directory.
The system upgrades containerd, crictl, and related configuration files to the new versions provided by ACK, and then restarts containerd. This action does not affect running containers. If you previously modified the
/etc/containerd/config.tomlfile on the node, the update overwrites your changes.The system ensures that the kubelet is running properly and the node is ready.
System disk replacement logic
The node is drained. If the node is schedulable, the system sets it to unschedulable.
The ECS instance is shut down.
The system disk is replaced. The system disk ID changes, but the cloud disk type, instance IP address, and elastic network interface MAC address remain unchanged.
The node is reinitialized.
The node is restarted and becomes ready. At the same time, the node is set to schedulable.
If you set a node to unschedulable before node draining began, it is not automatically set back to schedulable after the update is complete.
FAQ
Can I roll back a node pool after an update?
You cannot roll back the kubelet or container runtime after an update. You can only roll back the OS. When you roll back the OS, you must ensure that the node pool still supports the original image.
Does an update affect my applications?
In-place update: Pods are not restarted, and your applications are not affected.
Update by replacing system disk: Nodes are drained during this type of update. If your pods implement graceful shutdown logic and are deployed with multiple replicas across different nodes, your applications are not affected. To prevent multiple replicas of the same application from being updated in the same batch, you can manually set the maximum number of concurrent nodes to a value less than the number of pod replicas.
How long does each update batch take?
In-place update: Less than 5 minutes.
Update by replacing system disk: Typically less than 8 minutes if no snapshot is created. If you choose to create a snapshot, the update starts after the snapshot is created, and the total time depends on the snapshot creation time. The node pool update process allows 40 minutes for snapshot creation. If the snapshot does not complete within 40 minutes, the node update times out and fails. At this point, the update operation on the node has not yet started. If you do not store business data on the system disk, you can skip snapshot creation to reduce the update time.
Is node data lost during an update?
When you update the container runtime by replacing the system disk, do not store important data on the system disk, or make sure to back it up beforehand. Data on data disks is not affected during the update.
Impact on node IP after system disk replacement
When the system disk is replaced, its ID changes, but the cloud disk type, instance IP address, and elastic network interface MAC address remain unchanged. For more information, see Replace the system disk (operating system).
How to update unmanaged nodes?
Clusters created before the node pool feature was available may have unmanaged nodes that do not belong to any node pool. You can migrate these unmanaged nodes to a node pool and then update the node pool. For more information about how to migrate unmanaged nodes, see Migrate unmanaged nodes to a node pool.
Docker directory cleanup after containerd switch
In addition to files managed by the Kubernetes cluster, such as containers, images, and logs, the Docker directory also contains file paths that you created. If you no longer need the contents of the Docker directory, you can manually delete it from the data disk after switching the runtime.
How do I restore data from a snapshot?
When you update a node pool, you can create a snapshot for each node. The snapshot is retained for seven days by default, but you can manually delete it earlier. In rare cases, such as data loss after an update, you can use one of the following methods to restore your data:
If you performed an in-place update (for example, only the kubelet version was updated), you can roll back the disk by using the snapshot. For more information, see Roll back a cloud disk by using a snapshot.
If you performed an update by replacing the system disk (for example, the OS or container runtime was updated), you can create a new cloud disk from the snapshot to restore the data. For more information, see Create a data disk from a snapshot.
Related topics
You can also enable automatic cluster upgrades to reduce O&M workloads. For more information, see Automatically upgrade a cluster.
For the change history of containerd, see containerd release notes.
ACK managed node pools provide automatic patching for OS Common Vulnerabilities and Exposures (CVEs). For more information, see Automatic CVE patching (Recommended).
Kubernetes 1.24 no longer supports Docker as a built-in container runtime. You must migrate the node container runtime from Docker to containerd. For more information, see Migrate the node container runtime from Docker to containerd.
The command-line tools for Docker and containerd are different. For a comparison of common commands, see Comparison of common commands for Docker and containerd.
We recommend that you update your OS image to the latest version promptly. For more information, see Change the operating system.