Transform Cross-Account Cluster Backup and Recovery to Single-Account Operations via Registered Clusters - Container Service for Kubernetes

To back up and restore applications when the backup cluster and restore cluster belong to different accounts, you can create a registered cluster with the account of the restore cluster. Then, connect the backup cluster to the registered cluster and configure permissions to access Alibaba Cloud services, such as Object Storage Service (OSS). This allows you to back up and restore applications within the same account when the backup and restore clusters belong to different accounts.

How it works

Precautions

For cross-region restoration, create a registered cluster in the same region as the restore cluster under your restore account. The backup cluster will connect to the registered cluster via public network, using Internet for cloud backup, while using internal networks for data restoration to the restore cluster.
The backup cluster and restore cluster must run Kubernetes v1.16 or later. To ensure apiVersion compatibility, we recommend that you do not use the backup center to migrate applications from a cluster that runs a later Kubernetes version to a cluster that runs an earlier Kubernetes version.
The backup center does not back up resources that are being deleted.
To restore backups to File Storage NAS (NAS) volumes managed by CNFS (by setting StorageClass to alibabacloud-cnfs-nas), you need to create a StorageClass first. For more information, see Use CNFS to manage NAS file systems (recommended).
The backup center preferably restores applications to the API version suggested for the restore cluster. If no API version is supported by the old and new Kubernetes versions of a resource, you need to manually deploy the resource. Example:
- Deployments in a cluster that runs Kubernetes 1.16 support extensions/v1beta1, apps/v1beta1, apps/v1beta2, and apps/v1. In this scenario, the API versions of Deployments in a cluster that runs Kubernetes 1.28 are restored to apps/v1.
- Ingresses in a cluster that runs Kubernetes 1.16 support extensions/v1beta1 and networking.k8s.io/v1beta1. In this scenario, you cannot restore Ingresses in a cluster that runs Kubernetes 1.22 or later.
For more information about API updates for different Kubernetes versions, see Release notes for Kubernetes versions supported by ACK and Deprecated API Migration Guide.
Important
In a cluster that runs Kubernetes 1.16, groups such as apps and rbac.authorization.k8s.io already support API version v1. After you migrate applications to a cluster that runs Kubernetes 1.28, you need to manually restore the Ingress and CronJob resources.
The backup center installation templates for registered clusters and other Container Service for Kubernetes (ACK) clusters are different. When you connect the backup cluster to (or disconnect the backup cluster from) the registered cluster, if you want to continue to use the backup center, you need to reinstall the backup center component in the registered cluster (or backup cluster).
Disk snapshots created by the Elastic Compute Service (ECS) snapshot feature in the registered cluster belong to the account of the backup cluster. You cannot directly use the account of the restore cluster to restore disks from these snapshots. If your application uses disk volumes, you can use the snapshot sharing feature of ECS to manually restore disks from the snapshots. You can also modify the global configuration to use Cloud Backup to back up the data in disks. For more information, see Step 4: Create a restore task in the restore cluster.

Prerequisites

Cloud Backup is activated. To back up volumes that use NAS file systems, OSS buckets, and local disks or back up volumes in hybrid cloud scenarios, you must configure the backup center to use Cloud Backup to create backups. For more information, see Cloud backup.
A cluster where the volume is restored is created. To ensure that you can use snapshots of Elastic Compute Service (ECS) instances to restore disk data, we recommend that you update the Kubernetes version of the cluster to 1.18 or later. For more information, see Create an ACK managed cluster, Create an ACK dedicated cluster, or Create a cluster registration proxy and register a Kubernetes cluster that is deployed in a data center.
Important
- The restore cluster must use the Container Storage Interface (CSI) plug-in. Application restoration is not supported in clusters that use FlexVolume or use csi-compatible-controller and FlexVolume.
- The backup center is used to back up and restore applications. Before you run a restore task, you must install and configure system components in the restore cluster. Example:
  - aliyun-acr-credential-helper: You need to grant permissions to the restore cluster and configure acr-configuration.
  - alb-ingress-controller: You need to configure an ALBConfig.
migrate-controller is installed and permissions are granted. For more information, see Install migrate-controller and grant permissions.
To create disk snapshots to back up volumes, you must install CSI 1.1.0 or later. For more information about how to install the CSI plug-in, see Manage the CSI plug-in.
A kubectl client is connected to the cluster.

Billing

The backup center feature is free of charge. However, you may be charged for the following related services when you use the feature:

OSS: An OSS bucket is used to store backup files of the cluster, such as the related YAML files. For more information about the billing of OSS, see Billing overview.
Disk snapshot: Used to back up volumes that use Alibaba Cloud disks. For more information about the billing, see Snapshots.
Note
- Starting 11:00 (UTC+8) on October 12, 2023, you are no longer charged storage fees and feature usage fees for the instant access feature. For more information, see Use the instant access feature.
- If you use PL0 enhanced SSD (ESSD), PL1 ESSD, PL2 ESSD, PL3 ESSD, or ESSD AutoPL disks, snapshots created during backups have the instant access feature enabled by default.
Cloud Backup: Used to back up data from volume types other than disk volumes. You are charged for storage usage of container backup vaults created at the Cloud Backup side. For more information, see Price Details.

Sample applications

The following sample applications are used to demonstrate how to back up and restore application data.

The backup cluster contains the following sample applications.

Application	Whether volumes are mounted	Application characteristic	Whether data backup is required
sts-disk	A disk volume is mounted to the sts-disk application. The name of the persistent volume claim (PVC) is pvc-disk.	The application requires data consistency. Write operations must be suspended before you back up the application.	Yes. Cloud Backup and ECS snapshots are used to back up data.
sts-nas	A File Storage NAS (NAS) volume is mounted to the sts-nas application. The name of the PVC is pvc-nas.	The application does not require data consistency and can tolerate loss of data that is written during the backup process. The application cannot be backed up or restored across accounts through virtual private clouds (VPCs). The restore cluster cannot read the original data stored in NAS.	Yes. Cloud Backup is used to back up data.
sts-oss	An OSS volume is mounted to the sts-oss application. The name of the PVC is pvc-oss.	The application needs to store large amounts of data. The AccessKey pair of the account to which the backup cluster belongs is used to access the original OSS bucket.	No. The volume is restored based on the original PVC and persistent volume (PV) and mapped to the original OSS bucket.

For more information about how to create applications that use different types of volumes, see Storage - CSI.

(Optional) Step 1: Uninstall the backup center component in the backup cluster

If you have already used the features of the backup center in the backup cluster, you must first uninstall the backup center component in the backup cluster. This is because the backup center installation templates for registered clusters and other ACK clusters are different. Skip this step if the backup center component is not installed in the backup cluster.

Log on to the ACK console with the account of the backup cluster. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Operations > Add-ons.
On the Add-ons page, click the Manage Applications tab.
Find the migrate-controller card and click Uninstall in the lower-right corner.
In the Uninstall message, click OK.

Step 2: Install the backup center component in the registered cluster

Use the account of the backup cluster to create a registered cluster and connect the backup cluster to the registered cluster.
For more information, see Create a registered cluster.
Install the backup center component in the registered cluster and grant permissions.
For more information, see Install migrate-controller and grant permissions.
Note
The registered cluster is deployed in the same region as the backup cluster. Therefore, you do not need to add a route that points to the internal network of the region where your OSS bucket resides.

Step 3: Create a backup task in the registered cluster

Important

You must perform this step in a registered cluster within the account of the backup cluster. Do not perform this step in the registered cluster within the account of the backup cluster.

Compared with Cloud Backup, the ECS snapshot feature is more efficient and can ensure data consistency. Snapshots of disk volumes created by using the ECS snapshot feature belong to the account of the backup cluster. You cannot directly use the snapshots to restore data in a cluster that belongs to another account.

You can use the following solutions to restore the sts-disk application that uses disk volumes across accounts.

Backup solution	Cloud Backup solution	ECS snapshot solution
Implementation	Before you back up data, configure configmap/csdr-config in the csdr namespace to use Cloud Backup to back up the data stored in the disk volume. When you restore the volume, set the StorageClass of the target volume to disk.	Use the default ECS snapshot feature to back up the data in the disk volume and share the snapshot from the account of the backup cluster to the account of the restore cluster. After you use the backup center to restore the volume, you need to manually map the volumesnapshotcontent resource to the snapshot ID to ensure that the disk volume can be mounted as normal.
Advantages	No additional configuration is needed.	The ECS snapshot feature ensures the consistency of data stored in the same disk and allows you to restore data much faster.
Disadvantages	Cloud Backup requires a long period of time to back up and restore data and cannot ensure data consistency. If your application requires data consistency, you must suspend write operations for a long period of time during the backup process.	If large numbers of disk volumes are involved, the manual configuration can be complex.

Note

If your application does not use disk volumes, choose the Cloud Backup solution.

Cloud Backup solution

If your application uses disk volumes and you want to use Cloud Backup, choose this solution.

Disable the ECS snapshot feature.
1. Run the following command to modify csdr-config:
```
kubectl -ncsdr edit cm csdr-config 
```
2. Make sure that enable_ecs_snapshot is set to false. Otherwise, modify the parameter, then save, and exit.
3. Run the following command to restart the component for the modification to take effect:
```
kubectl -ncsdr delete pod -l control-plane=csdr-controller
```
If the account of the restore cluster does not have a backup vault, create one and associate it with the OSS bucket named cnfs-oss-* within the account of the restore cluster. For more information, see Create a backup vault.
Back up the sts-oss application. For more information, see Create a backup plan or back up instantly.
- The sts-oss application needs to use the original OSS bucket. Therefore, you do not need to back up the data in the OSS volume. Create a backup task for the application and do not Select Enable for Volume Backup.
- Assume that the sts-oss application belongs to the OSS namespace. When you create the backup task, select the OSS namespace. If the namespace contains other applications, add labels to specify the sts-oss application.
- On the Backup Records tab of the Application Backup page, confirm that the status of the backup task changes from InProgress to Completed. Click the name of the backup record and confirm that resources related to the sts-oss application, such as the StatefulSet, pod, PVC, PV, and Secret that stores the AccessKey pair, are backed up.
For more information about how to create a backup task for the sts-nas and sts-disk applications, see Create a backup plan or back up instantly.
- When you create the backup task, select Enable for Volume Backup.
- Data in disk volumes is also backed up by Cloud Backup because enable_ecs_snapshot is set to false. You can log on to the Cloud Backup console, click Container Backup in the left-side navigation pane, and view the backup progress and result of each PVC on the Backup Jobs tab.

ECS snapshot solution

If your application uses disk volumes and you want to use the ECS snapshot solution to back up cloud disks, choose this solution.

Enable the ECS snapshot feature.
1. Run the following command to modify csdr-config:
```
kubectl -ncsdr edit cm csdr-config 
```
2. Make sure that enable_ecs_snapshot is set to true. Otherwise, modify the parameter, then save, and exit.
3. Run the following command to restart the component for the modification to take effect:
```
kubectl -ncsdr delete pod -l control-plane=csdr-controller
```
If the account of the restore cluster does not have a backup vault, create one and associate it with the OSS bucket named cnfs-oss-* within the account of the restore cluster. For more information, see Create a backup vault.
Back up the sts-oss application. For more information, see Create a backup plan or back up instantly.
- The sts-oss application needs to use the original OSS bucket. Therefore, you do not need to back up the data in the OSS volume. Create a backup task for the application and do not Select Enable for Volume Backup.
- Assume that the sts-oss application belongs to the OSS namespace. When you create the backup task, select the OSS namespace. If the namespace contains other applications, add labels to specify the sts-oss application.
- On the Backup Records tab of the Application Backup page, confirm that the status of the backup task changes from InProgress to Completed. Click the name of the backup record and confirm that resources related to the sts-oss application, such as the StatefulSet, pod, PVC, PV, and Secret that stores the AccessKey pair, are backed up.
Create a backup task named backup-nas-disk for the sts-nas and sts-disk applications. For more information, see Create a backup plan or back up instantly.
- When you create the backup task, select Enable for Volume Backup.
- Data in disk volumes is backed up by using the ECS snapshot feature by default. You can log on to the ECS console, click Snapshots in the left-side navigation pane, and view the snapshot of each disk on the Disk Snapshots tab.
- Data in NAS volumes is backed up by using Cloud Backup. You can log on to the Cloud Backup console, click Container Backup in the left-side navigation pane, and view the backup progress and result of each PVC on the Backup Jobs tab.

Step 4: Create a restore task in the restore cluster

Important

Before you start, you need to install and configure the required components in the restore cluster. Example:
- Container Registry password-free image pulling component: You need to grant permissions to and configure acr-configuration in the restore cluster.
- ALB Ingress component: You need to configure an ALBConfig.
During the restoration process, if a resource with the same name already exists in the cluster, the restore cluster skips the resource.

Note

If your application does not use disk volumes, choose the Cloud Backup solution.
Services are restored based on Service types:
- NodePort Services: The ports of NodePort Services are retained by default during cross-cluster restoration.
- LoadBalancer Services: When ExternalTrafficPolicy is set to Local, HealthCheckNodePort uses a random port by default. To retain the port, specify spec.preserveNodePorts: true when you create the restore task.
- If a Service in the backup cluster uses an existing Server Load Balancer (SLB) instance, the Service restored in the restore cluster still uses the original SLB instance but has all listeners disabled by default. You need to configure the listeners in the SLB console.
- LoadBalancer Services in the backup cluster are managed by the cloud controller manager (CCM). When the system restores these Services, the CCM will create SLB instances. For more information, see Considerations for configuring a LoadBalancer Service.

Cloud Backup solution

Use the account of the restore cluster to perform the following steps in the restore cluster.

Create a restore task to restore the sts-oss application. For more information, see Restore applications and volumes.
1. On the Application Backup page, click Back up Now, select and initialize a backup vault, and wait for the system to synchronize the backup file to the restore cluster.
2. Select the backup-oss backup file and create a restore task.
3. Check whether the sts-oss application runs as normal in the restore cluster. Run the kubectl exec command to log on to the container and check whether the data is consistent.
Create a restore task to restore the sts-nas and sts-disk applications, and use StorageClass conversion to restore data to a volume of the desired type. For more information, see Restore applications and volumes.
1. On the Application Backup page, click Back up Now and select backup-nas-disk. Select the alicloud-disk StorageClass for pvc-disk and the alibabacloud-cnfs-nas StorageClass for pvc-nas.
  By default, alicloud-disk refers to the alicloud-disk-topology-alltype StorageClass provided by the CSI plug-in. You can also create a custom StorageClass in the cluster for StorageClass conversion.
2. Confirm that the application and data in the restore cluster are normal. You can log on to the ECS console and confirm whether a new disk is created within the account of the restore cluster and mounted to the corresponding node.

ECS snapshot solution

Use the account of the restore cluster to create a restore task in the restore cluster to restore the sts-oss application. For more information, see Restore applications and volumes.
1. On the Application Backup page, click Back up Now, select and initialize a backup vault, and wait for the system to synchronize the backup file to the restore cluster.
2. Select the backup-oss backup file and create a restore task.
3. Check whether the sts-oss application runs as normal in the restore cluster. Run the kubectl exec command to log on to the container and check whether the data is consistent.
Use the account of the backup cluster to log on to the ECS console and share the snapshot to the account of the restore cluster. For more information, see Share a snapshot.
Snapshots created by the backup center start with snapshot-. You can find the snapshot based on the creation time of the corresponding backup record.
Use the account of the restore cluster to log on to the Resource Management console and accept the shared snapshot. For more information, see A sharee uses a shared snapshot.
The name of the shared snapshot within the account of the restore cluster remains unchanged but its ID is changed. You can map the new snapshot ID to the original one based on the snapshot name.
Create a restore task in the restore cluster to restore the sts-nas and sts-disk applications. For more information, see Restore applications and volumes.
1. On the Application Backup page, click Back up Now and select backup-nas-disk. If you do not select StorageClass conversion, the volume is restored based on the original StorageClass by default. Make sure that the StorageClass already exists in the restore cluster.
2. If pvc-disk in the restore cluster is in the Pending state, this is because the snapshot ID is changed. You need to manually specify the new snapshot ID. Event content:
```
  Warning  ProvisioningFailed     2m27s (x9 over 4m35s)  diskplugin.csi.alibabacloud.com_csi-provisioner-7ff546495c-glmj7_e333fc6a-8bd6-4192-92ee-a48155e711b9  failed to provision volume with StorageClass "alicloud-disk-topology-alltype": error getting handle for DataSource Type VolumeSnapshot by Name pvc-disk-0-volumesnapshot2024010817xxxx: snapshot pvc-disk-0-volumesnapshot2024010817xxxx is not Ready
```
  pvc-disk-0-volumesnapshot2024010817xxxx indicates the name of the volumesnapshot resource.
  1. Run the following command to confirm the snapshotcontent of the volumesnapshot resource.
    Note
    The name of the volumesnapshot resource in the command is pvc-disk-0-volumesnapshot2024010817xxxx. Replace it with the actual resource name displayed in the event.
    kubectl -n<disk-ns> get volumesnapshot pvc-disk-0-volumesnapshot2024010817xxxx
    Expected output: The SOURCESNAPSHOTCONTENT column displays the snapshotcontent of the volumesnapshot resource.
    NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE pvc-disk-0-volumesnapshot2024010817xxxx true snapcontent-7c9ef8fa-e83c-4900-8df8-40268662d10e-2024010820xxxx 20Gi snapcontent-7c9ef8fa-e83c-4900-8df8-40268662d10e-2024010820xxxx 51m 44m
  2. Run the following command to change the snapshothandle in volumesnapshotcontent to the ID of the ECS snapshot within the account of the restore cluster, save the change, and exit.
    Note
    Replace the value of volumesnapshotconent in the command with the actual value returned in the preceding step.
    kubectl edit volumesnapshotconent snapcontent-7c9ef8fa-e83c-4900-8df8-40268662d10e-2024010820xxxx
3. Modify volumesnapshotcontent for each disk PVC in the Pending state and wait until the status of the PVC changes from Pending to Bound.
4. Confirm that the application and data in the restore cluster are normal. You can log on to the ECS console and confirm whether a new disk is created within the account of the restore cluster and mounted to the corresponding node.

(Optional) Step 5: Disconnect the backup cluster from the registered cluster and redeploy the backup center component

If you no longer need to back up or restore applications across accounts, you can disconnect the backup cluster from the registered cluster.
If you want to continue to use the backup center in the backup cluster, uninstall the backup center component and reinstall it.

References

For more information about how to migrate applications across clusters that use different volume plug-ins or run different Kubernetes versions, see Use the backup center to migrate applications in an ACK cluster that runs an old Kubernetes version.
For more information about how to migrate applications cross clusters in the same region, see Migrate applications across clusters in the same region.
For more information about how to migrate applications across regions, see Migrate applications across clusters in different regions.
For more information about how to use kubectl to migrate applications, see Use kubectl to back up and restore applications.