Auto-Expand Cloud Disk Volumes on Usage Threshold - Container Service for Kubernetes

ACK can automatically expand a cloud disk volume when usage exceeds a threshold you define. This lets you prevent disk-full outages without manual intervention. You define the threshold and growth rules in a StorageAutoScalerPolicy Custom Resource Definition (CRD), and the storage-auto-expander module in storage-operator handles the rest.

How it works

When storage-auto-expander detects that a PVC's disk usage exceeds the configured threshold, it triggers a volume expansion in this sequence:

The module continuously monitors disk usage on PVCs that match your policy's label selector.
When usage exceeds the condition threshold, the module selects the first matching action and submits a resize request to the CSI (Container Storage Interface) driver.
The CSI driver resizes the underlying cloud disk and signals Kubernetes to resize the file system on the node.
The PVC capacity reflects the new size once the file system resize completes.

Timing constraints: The maximum interval between expansion trigger checks is 2 minutes, and disk expansion takes about 1 minute. Do not fill the disk within 3 minutes of a previous expansion.

Limitations

Limitation	Detail
Kubernetes version	The cluster must run Kubernetes 1.16 or later. Upgrade the cluster if needed.
Disk type	Basic disks cannot be expanded automatically. Only disks that support resizing are eligible — see ResizeDisk for supported types.
StorageClass	The PVC must be associated with a StorageClass that has `allowVolumeExpansion: true`. ACK-provided StorageClasses have this enabled by default. For custom StorageClasses, set the parameter at creation time — you cannot modify an existing StorageClass.
Pod state	The application pod using the disk must be in Running state.

Enable the storage-auto-expander module

The storage-auto-expander module in storage-operator handles automatic expansion. In the cluster console, go to Operations > Add-ons and check the Volumes tab to confirm storage-operator is installed and to find its version.

v1.33.1 and later: The module is enabled by default. No action needed.

Earlier than v1.33.1: Run the following command to enable the module:

kubectl patch configmap/storage-operator \
  -n kube-system \
  --type merge \
  -p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'

To upgrade storage-operator, see Upgrade add-ons.

Configure an automatic expansion policy

Step 1: Verify that volume expansion is enabled on your StorageClass

kubectl get sc

The ALLOWVOLUMEEXPANSION column must be true. Example output:

NAME                                    PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-efficiency                diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-essd                      diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-ssd                       diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-topology-alltype          diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   26h

Use alicloud-disk-topology-alltype when possible. It automatically selects the appropriate disk type based on your instance type and zone availability, reducing the risk of provisioning failures caused by instance limitations or insufficient disk inventory. For details, see Use dynamically provisioned volumes.

Step 2: Create a StorageAutoScalerPolicy

Create a file named StorageAutoScalerPolicy.yaml:

apiVersion: storage.alibabacloud.com/v1alpha1
kind: StorageAutoScalerPolicy
metadata:
  name: hybrid-expand-policy
spec:
  pvcSelector:
    matchLabels:
      app: nginx
  namespaces:
    - default
    - nginx
  conditions:
    - name: condition1
      key: volume-capacity-used-percentage
      operator: Gt
      values:
        - "80"
  actions:
    - name: action1
      type: volume-expand
      params:
        scale: 50Gi
        limits: 100Gi
    - name: action2
      type: volume-expand
      params:
        scale: 50%
        limits: 300Gi

Key fields:

Field	Description
`pvcSelector`	Selects target PVCs by label. This example targets PVCs with `app: nginx`.
`namespaces`	Namespaces where the target PVCs reside. Multiple namespaces use OR logic. Defaults to `default` if omitted.
`conditions`	Conditions that trigger expansion. Multiple conditions use AND logic. `key` specifies the metric (`volume-capacity-used-percentage` = disk usage as a percentage). `operator` is the comparison: `Gt` (greater than), `Lt` (less than), `Eq` (equal), or `Ne` (not equal) — case insensitive. `values` is the threshold as a string.
`actions`	Actions to run when conditions are met. The system runs the first matching action and skips the rest. `type` must be `volume-expand`. `params.scale` is the expansion amount: absolute (e.g., `50Gi`) or a percentage of current size (e.g., `50%`). `params.limits` is the maximum PVC size allowed under this action.

In this example, the policy triggers when a PVC's usage exceeds 80%:

action1 adds 50 GiB, up to a maximum of 100 GiB. It applies while the PVC is under 100 GiB.
action2 adds 50% of the current size, up to a maximum of 300 GiB. It applies once the PVC reaches 100 GiB.

Step 3: Apply the policy

kubectl create -f StorageAutoScalerPolicy.yaml

Verify automatic expansion

This section walks through five expansion events using a test StatefulSet. The policy from the previous section is used throughout.

Step 1: Create a test StatefulSet

Create a file named StatefulSet.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: pvc-disk
          mountPath: /data
      volumes:
        - name: pvc-disk
          persistentVolumeClaim:
            claimName: disk-pvc
  volumeClaimTemplates:
    - metadata:
        name: pvc-disk
        labels:
          app: nginx
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "alicloud-disk-topology-alltype"
        resources:
          requests:
            storage: 25Gi

Apply it and confirm the pod is running:

kubectl create -f StatefulSet.yaml
kubectl get pod -l app=nginx

Expected output:

NAME      READY   STATUS    RESTARTS   AGE
nginx-0   1/1     Running   0          99s

Check the initial disk capacity:

kubectl exec -it nginx-0 -- df -h /data

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb         25G   24K   25G   1% /data

Step 2: Trigger the first expansion (25 GiB to 75 GiB)

Write 22 GB of data to push usage above 80%:

kubectl exec -it nginx-0 -- fallocate -l 22G /data/test1

Check expansion events:

kubectl get events

Because the disk is 25 GiB and usage exceeds 80%, action1 fires and adds 50 GiB:

2m1s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 25Gi to 75Gi, usedCapacityPercentage:90%, freeSize:2498MB.
2m1s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m1s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
118s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
116s   Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 25Gi to 75Gi, this action action2 will skip.

Verify the PVC capacity:

kubectl get pvc

Expected output:

NAME               STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
pvc-disk-nginx-0   Bound    d-uf66kkzltnq6xgi9****   75Gi       RWO            alicloud-disk-topology-alltype   <unset>                 26m

Step 3: Trigger the second expansion (75 GiB to 100 GiB)

Write 40 GB to push usage above 80% again:

kubectl exec -it nginx-0 -- fallocate -l 40G /data/test2

action1 fires again, but its 100 GiB limit caps the expansion at 100 GiB instead of 125 GiB:

kubectl get events

7m4s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0      Start to expand of pvc pvc-disk-nginx-0 from 75Gi to 100Gi, usedCapacityPercentage:84%, freeSize:11927MB.
7m4s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0      waiting for an external controller to expand this PVC
7m4s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0      External resizer is resizing volume d-uf66kkzltnq6xgi9****
7m1s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0      Require file system resize of volume on node
5m59s  Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0      Pvc pvc-disk-nginx-0 is expanding status from 75Gi to 100Gi, this action action2 will skip.

Step 4: Trigger the third expansion (100 GiB to 150 GiB)

At 100 GiB, action1's limit is reached. The next trigger switches to action2, which adds 50% of the current size:

kubectl exec -it nginx-0 -- fallocate -l 20G /data/test3
kubectl get events

2m40s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 100Gi to 150Gi, usedCapacityPercentage:83%, freeSize:16637MB.
2m40s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m40s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m37s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
109s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 100Gi to 150Gi, this action action2 will skip.

Step 5: Trigger the fourth expansion (150 GiB to 225 GiB)

action2 fires again, adding 50% of 150 GiB = 75 GiB:

kubectl exec -it nginx-0 -- fallocate -l 50G /data/test4
kubectl get events

2m42s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 150Gi to 225Gi, usedCapacityPercentage:87%, freeSize:19621MB.
2m42s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m42s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m38s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
114s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 150Gi to 225Gi, this action action2 will skip.

Step 6: Trigger the fifth expansion (225 GiB to 300 GiB)

action2 fires one final time, capped at its 300 GiB limit:

kubectl exec -it nginx-0 -- fallocate -l 50G /data/test5
kubectl get events

17m     Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 225Gi to 300Gi, usedCapacityPercentage:82%, freeSize:40351MB.
17m     Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
17m     Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
17m     Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node

Both actions have now reached their limits. If disk usage exceeds 80% again, no further expansion is triggered.

What's next

If you encounter issues with cloud disk volumes, see Cloud disk volume FAQ.