Hot migrate an ACK dedicated cluster to an ACK Pro cluster - Container Service for Kubernetes

You can hot migrate an ACK dedicated cluster to an ACK managed Pro cluster. This process migrates the cluster without interrupting your workloads.

Container Service for Kubernetes (ACK) stopped offering ACK dedicated clusters on August 21, 2024. For production environments, we recommend that you use ACK managed Pro clusters. They provide higher reliability, security, and scheduling efficiency. ACK managed Pro clusters feature a managed control plane, high availability, and other advanced capabilities.

Prerequisites

You have an ACK dedicated cluster (the cluster to be migrated) of Kubernetes version 1.18 or later. If you need to upgrade the cluster, see Manually upgrade a cluster.
The Kubernetes version of the cluster remains unchanged after the migration. If you need to both migrate and upgrade the cluster, we recommend migrating the cluster first and then upgrading the cluster version.
Before the migration, you must set the time zone for the cluster on its Basic Information page. This ensures that the control plane of the resulting ACK managed Pro cluster uses the same time zone as the original cluster, to prevent issues such as unexpected changes to CronJob execution times.
You have created an Object Storage Service (OSS) bucket of the Standard storage class in the same region as the cluster to be migrated. Ensure that hotlink protection is disabled for the bucket, as it can cause the migration to fail. For more information, see Create a bucket and Hotlink protection.

Considerations

Item	Description
Billing	After a successful migration, the billable items change. You are no longer charged for the removed master nodes. Instead, you are charged cluster management fees by ACK. Billing for other cloud resources remains unchanged. Creating an OSS bucket is free of charge. You are charged only for the storage of objects uploaded to the bucket and the traffic generated from accessing these objects. For more information, see Billing overview.
Public access	If the API server is exposed to the public by using an EIP, the cluster access address remains unchanged after the migration. Some earlier ACK dedicated clusters use a public-facing Server Load Balancer (SLB) instance to access the API server. After such clusters are migrated to ACK managed Pro clusters, you must manually switch to EIP mode by binding an EIP to the internal-facing Classic Load Balancer (CLB) instance of the API server. For more information, see Control public access to the API server of a cluster.
Custom Pod configuration	If custom Pod configurations are enabled for an ACK dedicated cluster, you cannot directly migrate it to an ACK managed Pro cluster. You must stop terway-controlplane before the migration and re-enable it afterward. For more information, see Stop terway-controlplane when you migrate an ACK dedicated cluster. To learn how to customize Pod configurations, see Configure a static IP address, a separate vSwitch, and a separate security group for a Pod.
Master nodes	In some earlier clusters, the Cloud Assistant Agent is not installed on the master nodes by default. You need to manually install it. For more information, see Install the Cloud Assistant Agent. After the migration, the status of the master nodes changes to Not Ready.
ECS instance release	When you remove the original master nodes after migration, ACK automatically releases only pay-as-you-go ECS instances and their data disks. You must manually release subscription ECS instances. For more information, see Release an instance.

Step 1: Hot migrate an ACK dedicated cluster to an ACK managed Pro cluster

After you complete the prerequisites and review the considerations, you can start the migration. After the hot migration, you cannot roll back the ACK managed Pro cluster to an ACK dedicated cluster.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to migrate and choose More>Migrate to Pro in the Actions column.
In the Migrate to Pro dialog box, complete the precheck and RAM authorization, select the OSS bucket that you prepared for the hot migration, read the notes carefully, and then click Confirm Migration.
Precheck
Click Precheck to open the Container Intelligence Service console, and then run the migration check. In the panel, review the check items, select I understand and agree, and then click Check.
If the precheck fails, follow the on-screen instructions to fix the issues.
RAM authorization
1. Click Go to RAM console to grant the permissions required for the hot migration. Also, obtain the OSS bucket name. You will need it to modify the policy in the next step.
2. Click a policy whose name starts with k8sMasterRolePolicy. On the details page, click the Policy Content tab, and then click Modify Policy Document. In the JSON Editor, add the following content to the Statement field, and then click OK.
  In the following code, replace <YOUR_BUCKET_NAME> with the name of the bucket you prepared. This name must match the one specified in the Migrate to Pro dialog box. Do not include the angle brackets (<>).
```
,
        {
            "Action": [
                "oss:PutObject",
                "oss:GetObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "acs:oss:*:*:<YOUR_BUCKET_NAME>/*"  
            ]
        }
```
After the migration is complete, a message appears in the Migrate to Pro dialog box. You can then check the cluster type and Migrate to Pro status.
- Cluster type: Return to the Clusters page. In the Cluster Type column, check that the type has changed from ACK Dedicated Cluster to ACK Managed Cluster, and that the Cluster Specification column shows Pro.
- Master node status: On the Clusters page, click Details in the Actions column for the target cluster. In the left-side navigation pane, choose Nodes>Nodes. On the Nodes page, check the Role/Status column. The status of the original master nodes changes to Unknown. This status indicates that the master nodes are disconnected from the cluster and no longer in use. You can then remove them. For more information, see Step 2: Remove the original master nodes.

Step 2: Remove the master nodes from the ACK dedicated cluster after hot migration

After the hot migration is complete, you can use the console or kubectl commands to manually remove the master node from the ACK dedicated cluster.

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Nodes.
On the Nodes page, find the master node that you want to remove and choose More>Remove in the Actions column. To remove multiple master nodes, select them and click Batch Remove at the bottom of the page. In the dialog box that appears, configure the settings, read the notes carefully, and then click OK.

kubectl

Before running the commands, connect to the cluster by using kubectl. For more information, see Obtain the kubeconfig of a cluster and use kubectl to connect to the cluster.

Get and record the names of the master nodes to be removed.
```
kubectl get node | grep control-plane
```
Remove the target master node. Replace <MASTER_NAME> with the master node name from the previous step.
```
kubectl delete node <MASTER_NAME>
```
To delete multiple master nodes at once, replace the <MASTER_NAME> placeholders with the names of the master nodes. For example, to delete the master nodes cn-hangzhou.192.xx.xx.65 and cn-hangzhou.192.xx.xx.66, run the following command:
```
kubectl delete node cn-hangzhou.192.xx.xx.65 cn-hangzhou.192.xx.xx.66
```

(Optional) Step 3: Manage add-ons

Check whether the ALB Ingress controller or ACK Virtual Node component is installed in the original ACK dedicated cluster. If so, you need to reinstall or migrate the component after the cluster migration is complete.

On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose Operations > Add-ons.
On the Add-ons page, check for the ALB Ingress controller or ACK Virtual Node component. If either is installed, handle it as described in the following sections.
Reinstall the ALB Ingress controller
If the ALB Ingress controller is installed in the ACK dedicated cluster, you must reinstall it after the migration. For more information about how to install the ALB Ingress controller, see Manage add-ons.
After installation, run the following command to delete the original application. Make sure that you have connected to the cluster by using kubectl. For more information, see Obtain the kubeconfig of a cluster and use kubectl to connect to the cluster.
```
kubectl delete deployment alb-ingress-controller -n kube-system
```
Migrate ACK Virtual Node to the managed version
If the ACK Virtual Node component is installed in the ACK dedicated cluster, you must migrate it to the managed version after the cluster migration to ensure a seamless transition for your workloads.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons .
3. On the Add-ons page, install the ACK Virtual Node component.
4. After the ACK Virtual Node component is installed, run the following commands in sequence to remove the obsolete components and configurations.
```
# Delete the old vk-webhook Service, ack-virtual-node-controller Deployment, virtual-kubelet ClusterRoleBinding, and virtual-kubelet ServiceAccount in sequence.
kubectl -n kube-system delete service vk-webhook
kubectl -n kube-system delete deployment ack-virtual-node-controller
kubectl delete clusterrolebinding virtual-kubelet
kubectl -n kube-system delete serviceaccount virtual-kubelet
```
5. After the migration is complete, you can create a new Pod to test whether the cluster is running as expected.

What to do next

After you migrate to an ACK managed Pro cluster, you must manually restrict the permissions of the worker RAM role to improve node security. For more information, see Manually restrict the permissions of the worker RAM role for an ACK managed cluster.
If cGPU Basic Edition is installed in the ACK dedicated cluster, you must upgrade it to cGPU Professional Edition after you migrate to an ACK managed Pro cluster. For more information, see Upgrade from cGPU Basic Edition to cGPU Professional Edition in an ACK Pro cluster.
In ultra-large-scale clusters or scenarios with high-concurrency bursts, pre-allocate and dedicate control plane resources to ensure predictable control plane performance. For more information, see ACK Pro preset control plane. The ACK Pro control plane is available in three predefined tiers: Pro XL, Pro 2XL, and Pro 4XL. The capacity of the control plane is defined by metrics such as API request concurrency (seats), pod scheduling rate (pods/second), and etcd database size (GB).

FAQ

Will services on an ACK dedicated cluster be affected during migration?

The control plane components of the ACK dedicated cluster enter sleep mode during the migration, but running workloads are unaffected.

Migration duration

The migration process takes 10 to 15 minutes and consists of three stages: control plane sleep, etcd data backup, and managed component startup. The API server is expected to be unavailable for 5 to 10 minutes during this period.

Does the access path change?

No. The IP address of the CLB instance for the API server remains unchanged. The cluster address does not change when accessing the cluster by using its kubeconfig.

ACK Virtual Node precheck failure

If the ACK Virtual Node component is installed in the ACK dedicated cluster, you must manually configure the internal endpoint of kube-apiserver before the migration. To do this, perform the following steps:

On the Cluster Information page, obtain the internal endpoint of kube-apiserver.
On the Deployments page, select the kube-system namespace, find the ack-virtual-node-controller Deployment, and then add the following environment variables to its spec.template.spec.containers[0].env field:
- KUBERNETES_APISERVER_HOST: The private IP address of kube-apiserver.
- KUBERNETES_APISERVER_PORT: The private port of kube-apiserver, which is usually 6443.

How do I recover after accidentally deleting master nodes of an ACK dedicated cluster?

The recovery method depends on the cluster version:

Cluster version 1.20 or later: Use the console to scale out new master nodes to restore the cluster.
Cluster version earlier than 1.20: Follow Step 1 in this topic to hot migrate the dedicated cluster to an ACK Pro cluster. The system automatically runs a precheck before the migration to ensure safety.