Container Service for Kubernetes:Migrate stateful applications with cloud disks across zones

Use cases

NAS and OSS support cross-zone and multi-mount usage. Disks are zone-bound — they cannot float across zones or reuse existing persistent volume claims (PVCs) and persistent volumes (PVs). When your StatefulSet uses disk volumes, you must create new disks in the target zone from snapshots.

Key constraints

Before you start, note the following constraints:

• Business interruption required: Cross-zone migration scales the StatefulSet to 0 replicas before migration, then restores all replicas at once after disk migration completes — not as a rolling update. Plan for downtime. The duration depends on replica count, container startup time, and disk capacity used.

• ESSD disks required: All storage used by the StatefulSet must be ESSD disks. The migration feature uses instant access snapshots to minimize snapshot creation time, which only support ESSD disks. For more information, see Snapshot instant access.

• Target zone requirements: The target zone must support ESSD disks, and the cluster must have nodes in that zone available for scheduling.

If your application uses non-ESSD disks, do one of the following before migrating:

• Change the disk type to ESSD.

• Manually create a snapshot for a single disk volume and rebuild the disk across zones.

How it works

Cross-zone migration relies on disk snapshots and uses instant access snapshots to minimize snapshot creation time. For more information, see Introduction to snapshots and Snapshot billing.

The storage-operator component runs the following steps:

1. Precheck: Verifies that the application is running properly and identifies disks to migrate. Migration stops if the precheck fails.

2. Scale to zero: Scales the StatefulSet to 0 replicas, pausing the application.

3. Create snapshots: Creates instant access snapshots for all mounted disks. Snapshots are zone-agnostic and can be used in any zone.

4. Provision new disks: After confirming snapshots are available, creates new disks in the target zone with the same data.

5. Rebuild PVCs and PVs: Rebuilds PVCs with the same names and their corresponding PVs, bound to the new disks.

6. Restore replicas: Restores the StatefulSet to its original replica count. Replicas automatically bind to the rebuilt PVCs and mount the new disks.

7. (Optional) Delete original resources: After confirming the application is healthy, delete the original PVs and disks. For disk billing details, see Block storage billing.

Prerequisites

Before you begin, ensure that you have:

• A cluster running Kubernetes 1.20 or later with the Container Storage Interface (CSI) driver installed

• storage-operator v1.26.2-1de13b6-aliyun or later installed in the cluster. For installation instructions, see Manage the storage-operator component

• The csi-plugin and csi-provisioner components installed, with csi-provisioner using the non-managed version

  If the managed version of csi-provisioner is currently installed, uninstall it and install the non-managed version. After switching, restart the storage controller: kubectl delete pod -n kube-system <storage-controller-pod-name>

• (ACK dedicated clusters only) The worker RAM role and master RAM role of your cluster must have permission to call the ModifyDiskSpec operation of the Elastic Compute Service (ECS) API. For instructions, see Create a custom policy. <details> <summary>View the required RAM policy</summary>

  ACK managed clusters do not require ModifyDiskSpec permission.

  {
      "Version": "1",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "ecs:CreateSnapshot",
                  "ecs:DescribeSnapshot",
                  "ecs:DeleteSnapshot",
                  "ecs:ModifyDiskSpec",
                  "ecs:DescribeTaskAttribute"
              ],
              "Resource": "*"
          }
      ]
  }

  </details>

Migrate a StatefulSet across zones

kubectl patch configmap/storage-operator \
  -n kube-system \
  --type merge \
  -p '{"data":{"storage-controller":"{\"imageRep\":\"acs/storage-controller\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-controller/install.yaml\",\"type\":\"deployment\"}"}}'

cat <<EOF | kubectl apply -f -
apiVersion: storage.alibabacloud.com/v1beta1
kind: ContainerStorageOperator
metadata:
  name: default
spec:
  operationType: APPMIGRATE
  operationParams:
    stsName: web
    stsNamespace: default
    stsType: kube
    targetZone: cn-beijing-h,cn-beijing-j
    checkWaitingMinutes: "1"
    healthDurationMinutes: "1"
    snapshotRetentionDays: "2"
    retainSourcePV: "true"
EOF

Examples

The following examples use an ACK Pro cluster with nodes in three zones:

• Zone B: cn-shanghai.192.168.5.245

• Zone G: cn-shanghai.192.168.2.214

• Zone M: cn-shanghai.192.168.3.236, cn-shanghai.192.168.3.237

FAQ

If a migration task returns FAILED, run the following command to get the error message:

kubectl describe cso <ContainerStorageOperator-name> | grep Message -A 1

Example output:

  Message:
    Consume: failed to get target pvc, err: no pvc mounted in statefulset or no pvc need to migrated web

This error means the component could not find a PVC to migrate. Common causes:

• The StatefulSet has no mounted storage.

• All disks are already in the target zone — no migration is needed.

• The component could not retrieve PVC information.

Resolve the issue based on the error message, then reapply the migration task.