Automatically expand a disk volume when its usage exceeds a threshold - Container Service for Kubernetes

ACK supports automatic expansion for disk volumes. You can use a Custom Resource Definition (CRD) to define a policy that automatically expands a disk volume when its usage exceeds a specified threshold. This topic describes how to configure this policy and verify the automatic expansion.

Scope

In this topic, a CRD is used to create an automatic expansion policy (StorageAutoScalerPolicy) to automatically expand a disk volume. The following section describes the scenarios in which you can use this method:

The PVC bound to the persistent volume (PV) of the disk is associated with a StorageClass, and the allowVolumeExpansion: true parameter is specified in the StorageClass.
Note
By default, the allowVolumeExpansion: true parameter is specified in StorageClass provided by ACK. You must specify this parameter for the StorageClass that you create. You cannot modify the properties of a StorageClass. You can only create a StorageClass.
The application pod to which the disk is attached must be in the Running state.
If the disk is a basic disk, you cannot expand the disk volumes by using this method.
If the Kubernetes version of the cluster is earlier than 1.16, you cannot use this method to expand the disk volumes. For more information about how to upgrade the Kubernetes version of a cluster, see Upgrade clusters.
You can expand only the disk volume of disks that can be resized. For more information, see ResizeDisk.

Precautions

A scale-out is triggered at a maximum interval of 2 min, and the subsequent disk expansion takes 1 min. To prevent errors, avoid filling the disk during this 3 min process.

Prerequisites

Before you begin, check the status of the storage-operator component and enable the automatic expansion feature if needed, based on your component version.

Confirm that the component is installed
The storage-operator component is installed in your cluster by default. In the navigation pane of the cluster management page, choose Operations Management > Components. On the Storage tab, confirm its installation status and version. To upgrade the component, see Upgrade components.
Enable automatic expansion based on the component version
The storage-auto-expander module in the storage-operator component is responsible for automatically expanding storage resources.
- For version v1.33.1 and later: The feature is enabled by default. No manual action is required.
- For versions earlier than v1.33.1: Modify the storage-operator ConfigMap to manually enable the feature.
  Sample command:
```
kubectl patch configmap/storage-operator \
  -n kube-system \
  --type merge \
  -p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'
```

Step 1: Configure an automatic disk expansion policy

Check the StorageClass and confirm that expansion is enabled.

kubectl get sc

Note

We recommend that you use the alicloud-disk-topology-alltype StorageClass that is created by default in ACK clusters. This StorageClass can automatically select disk types to avoid disk creation failures caused by instance type limits or insufficient disk inventory in the specified zone. For more information about how to create a StorageClass, see Use a dynamically provisioned disk volume.

The expected output is as follows: the ALLOWVOLUMEEXPANSION parameter is set to true, which indicates that volume expansion is enabled.

NAME                                    PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-efficiency                diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-essd                      diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-ssd                       diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-topology-alltype          diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   26h

Create an automatic expansion policy using a CRD.

Use the following YAML template to create a file named StorageAutoScalerPolicy.yaml.

apiVersion: storage.alibabacloud.com/v1alpha1
kind: StorageAutoScalerPolicy
metadata:
  name: hybrid-expand-policy
spec:
  pvcSelector:
    matchLabels:
      app: nginx
  namespaces:
    - default
    - nginx
  conditions:
    - name: condition1
      key: volume-capacity-used-percentage
      operator: Gt
      values:
        - "80"
  actions:
    - name: action1
      type: volume-expand
      params:
        scale: 50Gi
        limits: 100Gi
    - name: action2
      type: volume-expand
      params:
        scale: 50%
        limits: 300Gi

Configure the parameters based on your application and PVC. The following table describes the parameters.

Parameter	Description
`pvcSelector`	Filter target PVCs using labels. For example, `app: nginx`.
`namespaces`	The namespace where the target PVC resides. If you specify multiple namespaces, the OR logic applies. If you do not configure this parameter, the default value is `default`.
`conditions`	The conditions that trigger the rule. If you specify multiple conditions, the AND logic applies. Each condition includes the following parameters: `name`: The name of the condition. You can configure a custom name. `key`: The type of the metric. `volume-capacity-used-percentage` represents the capacity usage percentage. `operator`: The comparison operator, which can be `Gt` (greater than), `Lt` (less than), `Eq` (equal to), or `Ne` (not equal to). The value is case-insensitive. `values`: The values of the rule. In this example, the action is triggered when the PVC usage exceeds 80%.
`actions`	The operations to perform when the conditions are met. You can specify multiple operations. Each action includes the following parameters: `name`: The name of the action. You can configure a custom name. `type`: The operation type. Valid value: `volume-expand`, which indicates expansion. `params`: The operation parameters. `scale` specifies the scale-out size in GiB or as a percentage. `limits` specifies the maximum size limit for the PVC in this action. If multiple actions are specified, the system checks them sequentially and performs the first action whose conditions are met. The other actions are skipped. For example, if the conditions for action1 are met, action1 is performed and action2 is skipped. If the conditions for action1 are not met, the system checks and performs action2. action1 specifies that when the disk capacity is less than 100 GiB, an expansion is triggered to increase the capacity by 50 GiB each time, up to a maximum of 100 GiB. action2 specifies that when the disk capacity is greater than or equal to 100 GiB and less than 300 GiB, an expansion is triggered to increase the capacity by 50% of its current size. This means the total capacity after each expansion is 150% of the capacity before the expansion. The maximum disk capacity is 300 GiB.

Create the automatic expansion policy.

kubectl create -f StorageAutoScalerPolicy.yaml

Step 2: Verify the automatic disk expansion

You can create a StatefulSet to test auto scaling.

Use the following content to create a file named StatefulSet.yaml.

The following YAML creates a StatefulSet that contains a pod with a 25 GiB disk mounted to the /data path.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: pvc-disk
          mountPath: /data
      volumes:
        - name: pvc-disk
          persistentVolumeClaim:
            claimName: disk-pvc
  volumeClaimTemplates:
    - metadata:
        name: pvc-disk
        labels:
          app: nginx
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "alicloud-disk-topology-alltype"
        resources:
          requests:
            storage: 25Gi

Create the StatefulSet.
```
kubectl create -f StatefulSet.yaml
```

View the pod deployment status.

kubectl get pod -l app=nginx

Expected output:

NAME      READY   STATUS    RESTARTS   AGE
nginx-0   1/1     Running   0          99s

View the disk capacity.

kubectl exec -it nginx-0 -- df -h /data

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb         25G   24K   25G   1% /data

Write data to the mount directory to make the disk usage exceed 80% and trigger the first expansion.

Write data to the mount directory.
The following sample command allocates 22 GB of data to /data/test1 so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 22G /data/test1
```

View the expansion events.

kubectl get events

When the disk usage exceeds 80%, the disk capacity is 25 GiB. This meets the conditions of action1. Therefore, action1 is automatically executed to expand the disk by 50 GiB. The following related events appear in the event list.

2m1s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 25Gi to 75Gi, usedCapacityPercentage:90%, freeSize:2498MB.
2m1s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m1s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
118s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
116s   Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 25Gi to 75Gi, this action action2 will skip.

View the capacity of the PVC.

kubectl get pvc

The following output shows that after action1 is executed, the disk is expanded from 25 GiB to 75 GiB.

NAME               STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
pvc-disk-nginx-0   Bound    d-uf66kkzltnq6xgi9****   75Gi       RWO            alicloud-disk-topology-alltype   <unset>                 26m

You can continue to write data to the mount directory to increase the cloud disk usage to over 80% again and trigger a second scale-out.

Write data to the mount directory.
The following sample command assigns 40 GB of data to the /data/test2 directory to increase the disk usage to more than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 40G /data/test2
```

View the expansion events.

kubectl get events

When the usage of the 75 GiB disk exceeds 80%, the conditions for action1 are met. Therefore, action1 is automatically executed to expand the disk by 50 GiB. However, because the expansion limit for action1 is 100 GiB, the disk is expanded from 75 GiB to only 100 GiB. The following related events appear in the event list.

7m4s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0      Start to expand of pvc pvc-disk-nginx-0 from 75Gi to 100Gi, usedCapacityPercentage:84%, freeSize:11927MB.
7m4s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0      waiting for an external controller to expand this PVC
7m4s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0      External resizer is resizing volume d-uf66kkzltnq6xgi9****
7m1s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0      Require file system resize of volume on node
5m59s  Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0      Pvc pvc-disk-nginx-0 is expanding status from 75Gi to 100Gi, this action action2 will skip.

Continue writing data to the mount directory to make the disk usage exceed 80% again and trigger the third expansion.

Write data to the mount directory.
The following sample command assigns 20 GB of data to the /data/test3 directory to increase the disk usage to more than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 20G /data/test3
```

View the expansion events.

kubectl get events

When the disk usage exceeds 80% on the 100 GiB disk, the conditions for action2 are met. Therefore, action2 is automatically executed to expand the disk from 100 GiB to 150 GiB. The following related events appear in the event list.

2m40s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 100Gi to 150Gi, usedCapacityPercentage:83%, freeSize:16637MB.
2m40s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m40s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m37s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
109s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 100Gi to 150Gi, this action action2 will skip.

Continue writing data to the mount directory to increase the disk usage above 80% again and trigger the fourth expansion.

Write data to the mount directory.
The following sample command assigns 50 GB of data to the /data/test4 directory to increase the disk usage to more than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 50G /data/test4
```

View the expansion events.

kubectl get events

When the disk usage exceeds 80%, the disk capacity is 150 GiB, which meets the conditions for action2. Therefore, action2 is automatically executed to expand the disk by 50% of its current capacity, from 150 GiB to 225 GiB. The following related events appear in the event list.

2m42s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 150Gi to 225Gi, usedCapacityPercentage:87%, freeSize:19621MB.
2m42s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m42s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m38s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
114s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 150Gi to 225Gi, this action action2 will skip.

Continue to write data to the mount directory to increase the disk usage beyond 80% and trigger the fifth expansion.
1. Write data to the mount directory.
  The following sample command allocates 50 GB of data to the /data/test5 directory so that the disk usage exceeds 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 50G /data/test5
```
2. View the expansion events.
```
kubectl get events
```
  At this point, the disk capacity is 225 GiB. When the disk usage exceeds 80%, the conditions for action2 are met. Therefore, action2 is automatically executed to expand the disk by 50% of its current capacity. However, because the expansion limit for action2 is 300 GiB, this action can only expand the disk from 225 GiB to 300 GiB. The event list displays the following related events.
```
17m     Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 225Gi to 300Gi, usedCapacityPercentage:82%, freeSize:40351MB.
17m     Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
17m     Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
17m     Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
```
  At this point, both action1 and action2 have completed. If the cloud disk capacity exceeds 80% again, a scale-out is not triggered because the trigger conditions for these actions are no longer met.

References

If you encounter issues with disk volumes, see the Disk volume FAQ.