Container Service for Kubernetes:Cross-version application migration with the backup center

Notes

• The backup cluster and the restore cluster must be in the same region. The backup cluster must run Kubernetes 1.16 or later. To avoid API version compatibility issues, we recommend that you do not use the backup center to migrate applications from a newer to an older Kubernetes version.

• The backup center does not back up resources that are being deleted.

• To restore data to a Network Attached Storage (NAS) volume managed by Container Network File System (CNFS), you must create a StorageClass if you plan to set the StorageClass parameter to alibabacloud-cnfs-nas during restoration. For more information, see Manage NAS file systems by using CNFS.

• When an application is restored, resources are preferentially restored using the recommended apiVersion for the Kubernetes version of the restore cluster. If a resource does not have an apiVersion that is supported by both cluster versions, you must manually deploy that resource. For example:

  • A Deployment in a Kubernetes 1.16 cluster supports the extensions/v1beta1, apps/v1beta1, apps/v1beta2, and apps/v1 API versions. When restored to a Kubernetes 1.28 cluster, it is restored by using apps/v1.

  • An Ingress in a Kubernetes 1.16 cluster supports the extensions/v1beta1 and networking.k8s.io/v1beta1 API versions. You cannot directly restore it to a cluster that runs Kubernetes 1.22 or later.

  For more information about API changes across Kubernetes versions, see ACK release notes and Deprecated API Migration Guide.

  Important

  In a Kubernetes 1.16 cluster, API groups such as apps and rbac.authorization.k8s.io already support v1. When you migrate applications to a Kubernetes 1.28 cluster, you must manually restore resources such as Ingress and CronJob.

Use cases

• Application migration across storage plug-ins

  ACK clusters that run Kubernetes 1.20 or later no longer support the FlexVolume storage plug-in. You can use the backup center to migrate stateful applications from a FlexVolume cluster to a CSI cluster.

  Note

  You can migrate applications from clusters that use either the FlexVolume or CSI storage plug-in, but the restore cluster must use the CSI storage plug-in.

• Switching between clusters with large Kubernetes version gaps

  In some scenarios, you may need to migrate services from an older Kubernetes cluster (1.16 or later) to a new cluster. For example, you might switch the network plug-in from Flannel to Terway. The backup center supports application migration across large version gaps and automatically adjusts basic configurations, such as the apiVersion in application templates, to match the new Kubernetes version.

Prerequisites

• Activate Cloud Backup. When backing up NAS, OSS, or local disk persistent volumes, and in hybrid cloud scenarios, the backup center needs to use Cloud Backup for File Backup.

• A cluster where the volume is restored is created. To ensure that you can use snapshots of Elastic Compute Service (ECS) instances to restore disk data, we recommend that you update the Kubernetes version of the cluster to 1.18 or later. For more information, see Create an ACK managed cluster, Create an ACK dedicated cluster (discontinued), or Create an ACK One registered cluster.

  Important

  • The restore cluster must use the Container Storage Interface (CSI) plug-in. Application restoration is not supported in clusters that use FlexVolume or use csi-compatible-controller and FlexVolume.

  • The backup center is used to back up and restore applications. Before you run a restore task, you must install and configure system components in the restore cluster. Example:

    • aliyun-acr-credential-helper: You need to grant permissions to the restore cluster and configure acr-configuration.

    • alb-ingress-controller: You need to configure an ALBConfig.

• The migrate-controller backup service component is installed and its permissions are configured. For more information, see Install the migrate-controller backup service component and configure permissions.

• To back up volumes by using cloud disk snapshots, you must install CSI plug-in v1.1.0 or later. For more information, see Install and upgrade CSI components.

• Connect to the cluster by using kubectl.

Migration workflow

The migration workflow varies based on the storage plug-in that the backup cluster uses. The following figures show the details.

Backup cluster with no storage applications

Backup cluster using FlexVolume

Backup cluster using CSI

Procedure

This section uses an example of migrating applications, configurations, and volume data from an ACK cluster that uses FlexVolume and runs Kubernetes 1.16 to an ACK cluster that uses CSI and runs Kubernetes 1.28. The migration uses either a data source change or an unchanged data source. If you want to migrate applications that do not use storage or if your backup cluster uses the CSI storage plug-in, you can skip the steps that are marked as Optional.

kubectl patch pv/<pv-name> --type='json' -p '[{"op":"replace","path":"/spec/persistentVolumeReclaimPolicy","value":"Retain"}]'

Prepare the environment

Step 1: Deploy a test application

1. Run the following command to deploy a dynamically provisioned disk volume.

   Replace alicloud-disk-topology with the name of the default disk StorageClass for the FlexVolume storage plug-in in your cluster.

   cat << EOF | kubectl apply -f -
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: disk-essd
   spec:
     accessModes:
       - ReadWriteOnce
     storageClassName: alicloud-disk-topology
     resources:
       requests:
         storage: 20Gi
   EOF

2. Run the following command to deploy a statically provisioned NAS volume.

   Replace server with the mount target of the NAS file system in your account.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: pv-nas
   spec:
     capacity:
       storage: 5Gi
     storageClassName: nas
     accessModes:
       - ReadWriteMany
     flexVolume:
       driver: "alicloud/nas"
       options:
         server: "1758axxxxx-xxxxx.cn-beijing.nas.aliyuncs.com"
         vers: "3"
         options: "nolock,tcp,noresvport"
   ---
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: pvc-nas
   spec:
     accessModes:
       - ReadWriteMany
     storageClassName: nas
     resources:
       requests:
         storage: 5Gi
   EOF

3. Run the following command to deploy the application. This application mounts both the disk and NAS volumes from the previous steps.

   The apiVersion in the following code uses extensions/v1beta1. This apiVersion has been deprecated in version 1.28 clusters.

   cat << EOF | kubectl apply -f -
   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     name: nginx
     labels:
       app: nginx
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx
           ports:
           - containerPort: 80
           volumeMounts:
             - name: nas
               mountPath: /cold
             - name: disk
               mountPath: /hot
         volumes:
           - name: nas
             persistentVolumeClaim:
               claimName: pvc-nas
           - name: disk
             persistentVolumeClaim:
               claimName: disk-essd   
   EOF

4. Run the following command to verify that the deployed application has started.

   kubectl get pod -l app=nginx

   Expected output:

   NAME                    READY   STATUS    RESTARTS   AGE
   nginx-5ffbc895b-xxxxx   1/1     Running   0          2m28s

Step 2: Install the backup center

1. In the backup cluster, install the migrate-controller backup service component.

   Note

   • For clusters that run Kubernetes 1.16 or later, you can directly install the backup service component of V1.7.6 or later from the component marketplace.

   • If your backup cluster is an ACK Dedicated Cluster or a registered cluster, or uses a storage plugin other than CSI (such as FlexVolume), you need to configure additional permissions. For more information, see Registered cluster.

2. (Optional) If your cluster is a FlexVolume cluster, run the following command to confirm that the required permissions are configured.

   kubectl -n csdr get secret alibaba-addon-secret

3. (Optional) If your cluster is a FlexVolume cluster, run the following command to add the USE_FLEXVOLUME environment variable for the migrate-controller in the kube-system namespace.

   Important

   In a FlexVolume cluster, after the migrate-controller backup service component is installed, the migrate-controller pod exits unexpectedly, and opening the Application Backup page of the cluster returns a 404 error. In this case, you must edit the component's YAML file to add the USE_FLEXVOLUME environment variable.

   kubectl -n kube-system patch deployment migrate-controller --type json -p '[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"USE_FLEXVOLUME","value":"true"}}]'

4. Run the following commands to confirm that the backup service component is running properly.

   kubectl -n kube-system get pod -l app=migrate-controller
   kubectl -n csdr get pod 

   Expected output:

   NAME                                  READY   STATUS    RESTARTS   AGE
   migrate-controller-6c8b9c6cbf-967x7   1/1     Running   0          3m55s
   NAME                               READY   STATUS    RESTARTS   AGE
   csdr-controller-69787f6dc8-f886h   1/1     Running   0          3m39s
   csdr-velero-58494f6bf4-52mv6       1/1     Running   0          3m37s

Step 3: Create a backup

1. In the same region as the backup cluster, create an OSS Bucket named in the cnfs-oss-* format to store backups. For more information, see Create a bucket.

   Note

   ACK managed clusters have permissions for OSS Buckets whose names start with cnfs-oss-* by default. If your bucket naming format does not meet the requirements, you also need to configure additional permissions. For more information, see Install components and configure permissions in the console.

2. Create a backup vault.

3. Run the following command to create an immediate backup task.

   For information about how to configure backup settings in the ACK console, see Back up and restore applications in a cluster. This step provides recommended configuration for the example scenario. Adjust the settings for your specific scenario.

   cat << EOF | kubectl apply -f -
   apiVersion: csdr.alibabacloud.com/v1beta1
   kind: ApplicationBackup
   metadata:
     annotations:
       csdr.alibabacloud.com/backuplocations: '{"name":"<your-backup-vault-name>","region":"<your-region-id>","bucket":"<your-oss-bucket-name>","provider":"alibabacloud"}'
     labels:
       csdr/schedule-name: fake-name
     name: <your-backup-name>
     namespace: csdr
   spec:
     excludedNamespaces:
     - csdr
     - kube-system
     - kube-public
     - kube-node-lease
     excludedResources:
     - storageclasses
     - clusterroles
     - clusterrolebindings
     - events
     - persistentvolumeclaims
     - persistentvolumes
     includeClusterResources: true
     pvBackup:
       defaultPvBackup: true
     storageLocation: <your-backup-vault-name>
     ttl: 720h0m0s
   EOF

   Parameter	Description
   excludedNamespaces	The namespaces to exclude from the backup. We recommend that you exclude the following namespaces: csdr: The working namespace for the backup center. The backup center has inter-cluster synchronization logic. Do not manually back up tasks, such as backup or restore, in the csdr namespace. This action may cause unexpected behavior. kube-system, kube-public, and kube-node-lease are namespaces that exist by default in ACK clusters. They cannot be easily restored between clusters due to differences in cluster parameters and configurations.
   excludedResources	The resources to exclude. You can configure this parameter based on your business requirements.
   includeClusterResources	Specifies whether to back up cluster-level resources, such as StorageClasses, CRDs, and webhooks. true: Backs up all Cluster-level resources. false: Backs up only Cluster-level resources that are referenced by Namespace-level resources in the selected namespace. For example, when you back up a Pod, if the referenced ServiceAccount is authorized by a ClusterRole, the ClusterRole is automatically backed up. When you back up a CR, the CRD is automatically backed up. Note By default, IncludeClusterResources is set to false for backup tasks created in the ACK console.
   defaultPvBackup	Specifies whether to back up volume data. true: Backs up the application and the data in volumes used by running Pods. false: Backs up the application only. Important For clusters that run both Kubernetes and CSI versions 1.18 or later, the backup center uses ECS snapshots to back up disk data by default. For other storage types or for disk data in clusters that run Kubernetes versions from 1.16 up to, but not including, 1.18, Cloud Backup is used. For volumes that are not used by running pods, you can only use the unchanged data source method. This requires manually creating a static PV and PVC in the new cluster and specifying the original data source, such as a disk ID or an OSS bucket. If your application requires strong data consistency, pause data writes during the backup period. Alternatively, you can choose the unchanged data source method and back up only the application.

4. Run the following command to query the status of the backup task.

   kubectl -ncsdr describe applicationbackup <your-backup-name>

   In the expected output, the Phase parameter of Status changes to Completed, which indicates that the backup task is created successfully.

5. Run the following commands to confirm the resource list for this backup.

   kubectl -ncsdr get pod | grep csdr-velero
   kubectl -ncsdr exec -it <csdr-velero-pod-name> -- /velero describe backup <your-backup-name> --details

   You can check the list for resources that were not backed up and adjust the backup configuration to run the backup again.

   Resource List:
     apiextensions.k8s.io/v1/CustomResourceDefinition:
       - volumesnapshots.snapshot.storage.k8s.io
     v1/Endpoints:
       - default/kubernetes
     v1/Namespace:
       - default
     v1/PersistentVolume:
       - d-2ze88915lz1il01v1yeq
       - pv-nas
     v1/PersistentVolumeClaim:
       - default/disk-essd
       - default/pvc-nas
     v1/Secret:
       - default/default-token-n7jss
       - default/oss-secret
       - default/osssecret
     v1/Service:
       - default/kubernetes
     v1/ServiceAccount:
       - default/default
     ...

Step 4: Install the backup center

1. Install the backup center in the restore cluster. For more information, see Step 2: Install the backup center.

2. Associate the backup vault that you created with the restore cluster.

   1. Log on to the ACK console.

   2. On the Clusters page, click the name of the target cluster. In the left-side navigation pane, choose Operations > Application Backup.

   3. On the Application Backup page, click Restore.

   4. Select the Backup Vault that was used for the backup, click Initialize Backup Vault, and wait for the backup to synchronize to this cluster.

(Optional) Step 5: Manually create PVCs and PVs

In most scenarios, you can simply follow Step 6 to create a restore task directly in the restore cluster. The backup center component then automatically generates storage claims and volumes based on the backup.

When the backup center executes a restore task, it skips the restoration of PVCs and PVs with the same names to protect existing data. This means it neither rebuilds them nor overwrites the data within the volumes. Therefore, in the following scenarios, you can pre-create the PVCs and PVs before the restore task for more flexible recovery:

• You backed up volumes, but some volumes contain data that does not need to be migrated, such as logs. You can pre-create empty volumes.

• You backed up volumes, but volumes in the backup cluster that are not used by running pods also need to be migrated to the restore cluster.

• You did not back up volumes, and the excludedResources list includes persistentvolumeclaims and persistentvolumes, or the migration involves moving an application from a FlexVolume cluster to a CSI cluster.

The following are the specific steps:

1. (Optional) If your backup cluster is a FlexVolume cluster, you can use a command-line tool to batch convert YAML files because the YAML for PVs and PVCs is different between FlexVolume and CSI. For more information, see Use the FlexVolume2CSI command-line tool to batch convert YAML files.

2. Run the following command to deploy the CSI YAML file obtained from FlexVolume2CSI.

   where outputfile.txt is the output of the command-line tool's YAML conversion.

   Expand to view the outputfile.txt file

   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     labels:
       alicloud-pvname: d-2ze88915lz1il0**** # The dynamically created disk.
     name: d-2ze88915lz1il0****
   spec:
     accessModes:
     - ReadWriteOnce
     capacity:
       storage: 20Gi
     csi:
       driver: diskplugin.csi.alibabacloud.com
       fsType: ext4
       volumeHandle: d-2ze88915lz1il0****
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: topology.diskplugin.csi.alibabacloud.com/zone
             operator: In
             values:
             - cn-beijing-i
     persistentVolumeReclaimPolicy: Delete
     storageClassName: alicloud-disk-essd
     volumeMode: Filesystem
   
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: disk-essd
     namespace: default
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
     selector:
       matchLabels:
         alicloud-pvname: d-2ze88915lz1il0****
     storageClassName: alicloud-disk-essd
     volumeMode: Filesystem
   
   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     labels:
       alicloud-pvname: pv-nas
     name: pv-nas
   spec:
     accessModes:
     - ReadWriteMany
     capacity:
       storage: 5Gi
     csi:
       driver: nasplugin.csi.alibabacloud.com
       volumeAttributes:
         server: 1758axxxxx-xxxxx.cn-beijing.nas.aliyuncs.com
       volumeHandle: pv-nas
     mountOptions:
     - vers=3
     - nolock,tcp,noresvport
     persistentVolumeReclaimPolicy: Retain
     storageClassName: nas
     volumeMode: Filesystem
   
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pvc-nas
     namespace: default
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 5Gi
     selector:
       matchLabels:
         alicloud-pvname: pv-nas
     storageClassName: nas
     volumeMode: Filesystem
   

   kubectl apply -f outputfile.txt

3. Run the following command to confirm that the PVC in the restore cluster is in the Bound state.

   kubectl get pvc 

   Expected output:

   NAME        STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
   disk-essd   Bound    d-2ze88915lz1il0xxxxxx   20Gi       RWO            alicloud-disk-essd   29m
   pvc-nas     Bound    pv-nas                   5Gi        RWX            nas                  29m

Step 6: Create a restore task

If you backed up volumes when creating the backup, you are using the data source change method for backup and restore. You can change the storage type by using StorageClass conversion (convertedarg). For example, you can convert NAS storage to disk storage. You can select the target StorageClass based on your requirements.

• In this example, because the backup cluster is a v1.16 FlexVolume cluster and disk volume backups use Cloud Backup, you can select alicloud-disk as the target StorageClass for the disk-essd storage claim (which is converted to a CSI disk class and defaults to alicloud-disk-topology-alltype). If your backup cluster is a v1.18 or later CSI cluster, you do not need to perform related configurations for disk volumes.

• This example also converts the FlexVolume NAS volume to a CNFS-managed isolated NAS volume by selecting the target StorageClass alibabacloud-cnfs-nas for the pvc-nas PVC. If your cluster does not have the alibabacloud-cnfs-nas StorageClass, see Manage NAS file systems by using CNFS.

The following are the specific steps:

1. Run the following command to create a restore task.

   For information about how to configure a restore task in the ACK console, see Restore applications and data volumes. This step provides recommended configuration for the example scenario. Adjust the settings for your specific scenario.

   cat << EOF | kubectl apply -f -
   apiVersion: csdr.alibabacloud.com/v1beta1
   kind: ApplicationRestore
   metadata:
     annotations:
       csdr.alibabacloud.com/backuplocations: >-
         '{"name":"<your-backup-vault-name>","region":"<your-region-id>","bucket":"<your-oss-bucket-name>","provider":"alibabacloud"}'
     name: <your-restore-name>
     namespace: csdr
   spec:
     backupName: <your-backup-name>
     excludedNamespaces:
     - arms-prom
     excludedResources:
     - secrets
     appRestoreOnly: false
     convertedarg:
     - convertToStorageClassType: alicloud-disk-topology-alltype
       namespace: default
       persistentVolumeClaim: alicloud-disk
     - convertToStorageClassType: alibabacloud-cnfs-nas
       namespace: default
       persistentVolumeClaim: pvc-nas
     namespaceMapping:
       <backupNamespace>: <restoreNamespace>
   EOF

   Parameter	Description
   excludedNamespaces	The namespaces to exclude. You can exclude unwanted namespaces from the backup resource list.
   excludedResources	The resources to exclude. You can exclude unwanted resource types from the backup resource list.
   appRestoreOnly	For a backup that includes volumes, this parameter specifies whether to restore the volumes. true: Creates dynamic volumes and storage claims that point to a new data source during restoration. Backup tasks created in the console default to true. false: A static volume is not created. You must manually deploy a static volume in advance. Note Typically, set this to true for a data source change and to false for an unchanged data source.
   convertedarg	The StorageClass conversion list. For volumes of the FileSystem type, such as OSS, NAS, CPFS, and local volumes, you can configure this parameter to convert the StorageClasses of their PVCs to the specified StorageClass during the restoration process. For example, you can convert NAS volumes to disk volumes. convertToStorageClassType: the desired StorageClass. Make sure that the StorageClass exists in the current cluster. You can specify only the disk or NAS StorageClass. namespace: the namespace of the PVC. persistentVolumeClaim: the name of the PVC. The above are the required parameters for the StorageClass conversion feature.

2. Run the following command to query the status of the restore task.

   kubectl -ncsdr describe applicationrestore <your-restore-name>

   In the expected output, the Phase of Status changes to Completed, which indicates that the task is restored successfully.

3. Run the following commands to check for any resources that failed to restore and to find the cause of the failure.

   kubectl -ncsdr get pod | grep csdr-velero
   kubectl -ncsdr exec -it <csdr-velero-pod-name> -- /velero describe restore <your-restore-name> --details

   Expected output:

   Warnings:
     Velero:     <none>
     Cluster:  could not restore, ClusterRoleBinding "kubernetes-proxy" already exists. Warning: the in-cluster version is different than the backed-up version.
     Namespaces:
       demo-ns:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version.
                  could not restore, Endpoints "kubernetes" already exists. Warning: the in-cluster version is different than the backed-up version.
                  could not restore, Service "kubernetes" already exists. Warning: the in-cluster version is different than the backed-up version.
   
   Errors:
     Velero:     <none>
     Cluster:    <none>
     Namespaces:
       demo-ns:  error restoring endpoints/xxxxxx/kubernetes: Endpoints "kubernetes" is invalid: subsets[0].addresses[0].ip: Invalid value: "169.254.128.9": may not be in the link-local range (169.xxx.0.0/16, fe80::/10)
                  error restoring endpointslices.discovery.k8s.io/demo-ns/kubernetes: EndpointSlice.discovery.k8s.io "kubernetes" is invalid: endpoints[0].addresses[0]: Invalid value: "169.xxx.128.9": may not be in the link-local range (169.xxx.0.0/16, fe80::/10)
                  error restoring services/xxxxxx/kubernetes-extranet: Service "kubernetes-extranet" is invalid: spec.ports[0].nodePort: Invalid value: 31882: provided port is already allocated

   From the preceding output, you can see if any resources in the restore cluster were not restored. For example, the Warnings indicate that a resource already exists and was skipped. The Errors indicate a NodePort conflict, as the original port is retained during a cross-cluster restore.

4. Confirm that the restored application is running properly.

   1. After the application is restored, check whether any resources are in an abnormal state due to application constraints, container runtime exceptions, or other reasons. If so, fix them manually.

   2. After the recovery is verified, the apiVersion of the Nginx application is adjusted by default to apps/v1, which is recommended for clusters of version 1.28.