Container Service for Kubernetes:Modify resource configurations during application restoration

How it works

Three mechanisms control resource modifications during restoration:

Prerequisites

Before you begin, ensure that you have:

• migrate-controller 1.8.1 or later installed in both the backup cluster and the restore cluster. For more information, see Install migrate-controller and grant permissions.

• A backup vault created in the backup cluster. For more information, see Create a backup vault.

• (For resource modifiers) Kubernetes 1.20 or later and the CSI plug-in enabled in the cluster.

If the backup cluster, restore cluster, and backup vault reside in different regions, enable public network access on both clusters.

Example scenario

This example backs up an NGINX StatefulSet and restores it with four modifications applied simultaneously:

• Namespace remapped from default to default1

• Image registry switched from the OpenAnolis community registry to Container Registry

• Volume mount path changed from /usr/share/nginx/html/ to /data

• Node affinity rules removed

The StatefulSet runs in the default namespace of the backup cluster. It uses the image anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6 and mounts a 20 GiB persistent volume claim (PVC) using the alicloud-disk-topology-alltype StorageClass.

<details> <summary>Show StatefulSet YAML</summary>

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
            weight: 1
      containers:
      - image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/nginx/html/
          name: www
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        app: nginx
      name: www
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: alicloud-disk-topology-alltype
      volumeMode: Filesystem

</details>

Run the following command to verify that the application is running and the volume is mounted:

kubectl get pod && kubectl get pvc

Expected output:

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          39s
web-1   1/1     Running   0          30s
NAME        STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
www-web-0   Bound    d-2zefofsg0uzoiaxxxxxx   20Gi       RWO            alicloud-disk-topology-alltype   <unset>                 3m18s
www-web-1   Bound    d-2zedgdlghw2oudxxxxxx   20Gi       RWO            alicloud-disk-topology-alltype   <unset>                 30s

Step 1: Back up the application

Back up the StatefulSet in the backup cluster using kubectl or the ACK console.

(Optional) Step 2: View the backup JSON file in OSS

The backup task generates a JSON file listing all backed-up resources, including their names, types, sizes, and creation dates. Check this file to confirm the resources you want to restore are included.

The backup vault used in this example has the following configuration:

1. Log on to the OSS console.

2. In the left-side navigation pane, click Buckets, then click the cnfs-oss-test bucket.

3. Choose Object Management > Objects.

4. Navigate to /subpath/backups/<backupName>/, download the <backupName>.tar.gz package, and decompress it to view the resource list.

Step 3: Configure resource modification rules in the restore cluster

A resource modifier ConfigMap has two sections: conditions (which resources to target) and patches (what changes to apply). The ConfigMap must reside in the csdr namespace and follow the naming convention <backupName>-resource-modifier.

The following ConfigMap removes node affinity rules and changes the volume mount path to /data for StatefulSets and pods with the label app: nginx:

apiVersion: v1
kind: ConfigMap
metadata:
  name: <backupName>-resource-modifier
  namespace: csdr
data:
  modifier: |
    version: v1
    resourceModifierRules:
    - conditions:
        groupResource: statefulsets.apps
        namespaces:
        - default
        labelSelector:
          matchLabels:
            app: nginx
      patches:
      - operation: remove
        path: "/spec/template/spec/affinity/nodeAffinity"
      - operation: replace
        path: "/spec/template/spec/containers/0/volumeMounts/0/mountPath"
        value: "/data"
    - conditions:
        groupResource: pods
        resourceNameRegex: "^web.*$"
        namespaces:
        - default
        labelSelector:
          matchLabels:
            app: nginx
      patches:
      - operation: remove
        path: "/spec/affinity"
      - operation: replace
        path: "/spec/containers/0/volumeMounts/0/mountPath"
        value: "/data"

(Optional) Step 4: Initialize the backup vault in the restore cluster

Skip this step if the backup vault is already initialized in the restore cluster.

Apply the following backuplocation.yaml to register the backup vault:

apiVersion: csdr.alibabacloud.com/v1beta1
kind: BackupLocation
metadata:
  name: <backup-vault-name>
  namespace: csdr
spec:
  backupSyncPeriod: 0s
  config:
    network: internal
    region: <oss-bucket-region>    # Must match the backup vault's initial configuration
  objectStorage:
    bucket: <oss-bucket-name>
    prefix: <oss-bucket-subpath>
  provider: alibabacloud

Step 5: Restore the application with modifications

Create a restore task in the restore cluster. The task below references the ConfigMap from Step 3 and uses namespaceMapping and imageRegistryMapping to remap the namespace and switch the image registry.

Sync the image to the new registry before changing the image address. The restore task references registry.cn-beijing.aliyuncs.com/<acrRegistry> — make sure the image exists there before applying.

1. Create a file named applicationrestore.yaml with the following content:

   apiVersion: csdr.alibabacloud.com/v1beta1
   kind: ApplicationRestore
   metadata:
     name: <restoreName>
     namespace: csdr
   spec:
     backupName: <backupName>
     namespaceMapping:
       default: default1                      # Restore resources into the default1 namespace
     imageRegistryMapping:
       anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis: registry.cn-beijing.aliyuncs.com/<acrRegistry>
     resourceModifier:
       kind: ConfigMap
       name: <backupName>-resource-modifier   # Must match the ConfigMap created in Step 3

2. Apply the manifest to start the restore task:

   kubectl apply -f applicationrestore.yaml

3. Watch the restore task until it reaches the Completed phase:

   kubectl -n csdr get applicationrestore <restoreName> --watch

   Expected output:

   NAME              PHASE       AGE
   <restoreName>     Completed   18h

   To view the restore task in the ACK console instead, see Step 3: Restore applications and volumes.

Step 6: Verify the restored application

Query the restored StatefulSet to confirm all modifications were applied:

kubectl -n default1 get sts web -o yaml

<details> <summary>Show expected output</summary>

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"web","namespace":"default"},"spec":{"replicas":2,"selector":{"matchLabels":{"app":"nginx"}},"serviceName":"nginx","template":{"metadata":{"labels":{"app":"nginx"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-role.kubernetes.io/master","operator":"Exists"}]},"weight":1}]}},"containers":[{"image":"anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.2","imagePullPolicy":"IfNotPresent","name":"nginx","ports":[{"containerPort":80,"name":"web","protocol":"TCP"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/usr/share/nginx/html/","name":"www"}]}]}},"volumeClaimTemplates":[{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"labels":{"app":"nginx"},"name":"www"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"20Gi"}},"storageClassName":"alicloud-disk-topology-alltype","volumeMode":"Filesystem"}}]}}
  creationTimestamp: "2024-10-09T08:32:50Z"
  generation: 1
  labels:
    app: nginx
    velero.io/backup-name: <backupName>
    velero.io/restore-name: <restoreName>
  name: web
  namespace: default1
  resourceVersion: "119622"
  uid: d23878ea-0b9f-40ba-b61b-1ff6bb77eb43
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      affinity: {}
      containers:
      - image: registry.cn-beijing.aliyuncs.com/<acrRegistry>/nginx:1.14.1-8.6
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: www
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
      name: www
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: alicloud-disk-topology-alltype
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 0
  collisionCount: 0
  currentRevision: web-7b454646b4
  observedGeneration: 1
  replicas: 2
  updateRevision: web-7b454646b4

</details>

The output confirms all four modifications were applied:

• Namespace: default → default1

• Image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6 → registry.cn-beijing.aliyuncs.com/<acrRegistry>/nginx:1.14.1-8.6

• Volume mount path: /usr/share/nginx/html/ → /data

• Node affinity: removed (affinity: {})

Some fields in the output (such as resourceVersion, uid, and creationTimestamp) are automatically added by Kubernetes and were not present in the original backup.

What's next

• Create a backup plan or back up instantly

• Restore applications and volumes

• Install migrate-controller and grant permissions