Automatically expand a disk volume when the disk volume usage exceeds a specific threshold - Container Service for Kubernetes

Container Service for Kubernetes (ACK) allows you to automatically expand a disk volume. You can use a CustomResourceDefinition (CRD) to define an automatic disk expansion policy. This way, the disk can be automatically resized when the volume usage exceeds a specified threshold. This topic describes how to configure an automatic disk expansion policy and verify the automatic expansion of a disk volume.

Applicable scope

In this topic, a CRD is used to create an automatic expansion policy (StorageAutoScalerPolicy) to automatically expand a disk volume. The following section describes the scenarios in which you can use this method:

The PVC bound to the persistent volume (PV) of the disk is associated with a StorageClass, and the allowVolumeExpansion: true parameter is specified in the StorageClass.
Note
By default, the allowVolumeExpansion: true parameter is specified in StorageClass provided by ACK. You must specify this parameter for the StorageClass that you create. You cannot modify the properties of a StorageClass. You can only create a StorageClass.
The application pod to which the disk is attached must be in the Running state.
If the disk is a basic disk, you cannot expand the disk volumes by using this method.
If the Kubernetes version of the cluster is earlier than 1.16, you cannot use this method to expand the disk volumes. For more information about how to upgrade the Kubernetes version of a cluster, see Upgrade clusters.
You can expand only the disk volume of disks that can be resized. For more information, see ResizeDisk.

Usage notes

The maximum interval between two expansion operations is 2 minutes. The disk expansion process requires 1 minute. Make sure that you do not exhaust the disk space within 3 minutes.

Step 1: Configure the storage-operator component to enable automatic expansion

The storage-operator component is installed in the cluster and the component version is v1.18.8.28-18cca7b-aliyun or later.
Note
By default, the storage-operator component is installed in the cluster. In the left-side navigation pane of the cluster management page, choose Operations > Add-ons. On the Storage tab, you can check the installation status and version information of storage-operator. For more information, see Manage the storage-operator component.

Connect to the cluster and run the following command to modify the ConfigMap of storage-operator and enable the auto expansion feature:

The default storage-auto-expander of the storage-operator component is responsible for automatically expanding storage resources. Sample command:

kubectl patch configmap/storage-operator \
  -n kube-system \
  --type merge \
  -p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'

Step 2: Configure an automatic expansion policy

Check the StorageClass and whether expansion is enabled.

kubectl get sc

Note

We recommend that you create an alicloud-disk-topology-alltype StorageClass in ACK clusters. The StorageClass can automatically select disk types to avoid disk creation failures caused by instance type limits or insufficient disk inventory in the specified zone. For more information about how to create a StorageClass, see Use a dynamically provisioned disk volume.

The following expected response is returned. The ALLOWVOLUMEEXPANSION parameter is set to true, which indicates that expansion is enabled.

NAME                                    PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-efficiency                diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-essd                      diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-ssd                       diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   26h
alicloud-disk-topology-alltype          diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   26h

Use a CRD to create an automatic expansion policy.

Use the following template to create a file named StorageAutoScalerPolicy.yaml.

apiVersion: storage.alibabacloud.com/v1alpha1
kind: StorageAutoScalerPolicy
metadata:
  name: hybrid-expand-policy
spec:
  pvcSelector:
    matchLabels:
      app: nginx
  namespaces:
    - default
    - nginx
  conditions:
    - name: condition1
      key: volume-capacity-used-percentage
      operator: Gt
      values:
        - "80"
  actions:
    - name: action1
      type: volume-expand
      params:
        scale: 50Gi
        limits: 100Gi
    - name: action2
      type: volume-expand
      params:
        scale: 50%
        limits: 300Gi

Configure the parameters based on your application and PVC. The following table describes the parameters.

Parameter	Description
`pvcSelector`	Filter target PVCs by using labels. In this example, `app: nginx` is specified.
`namespaces`	Specifies the namespace of the PVC. If multiple namespaces are specified, the logical operator between the namespaces is OR. Default value: default.
`conditions`	Specifies the conditions that trigger the action. The logical operator between multiple conditions is AND. Each condition includes the following parameters: `Name`: The name of the condition. You can configure a custom name. `key`: The type of the metric. `volume-capacity-used-percentage`: The storage usage threshold is expressed as a percentage. `operator`: The operator, which can be `Gt` (greater than), `Lt` (less than), `Eq` (equal to), or `Ne` (not equal to). The parameter value is not case-sensitive. `values`: The threshold value. In the example, the condition specifies that the action is triggered when the PVC usage exceeds 80%.
`actions`	The operations that are performed when the preceding conditions are met. Multiple operations are allowed. Each action includes the following parameters: `name`: The name of the action. You can configure a custom name. `type`: The operation type. Valid value: `volume-expand`, which indicates expansion. `params`: The operation parameter. `scale` indicates the capacity of the PVC to be expanded. Unit: GiB. You can also use a percentage. `limits` indicates the maximum limit of the PVC in this action. If multiple actions are specified, the first action whose limits condition is met is performed. The other actions are skipped. In this example, if the limits condition of action1 is met, action1 is performed and action2 is skipped. If the limits condition of action1 is not met, action2 is performed. action1 specifies that when the disk capacity is less than 100 GiB, an expansion operation is triggered to increase the capacity by 50 GiB each time, up to the maximum capacity of 100 GiB. action2 specifies that the disk is expanded by 50% if the disk capacity is greater than 100 GiB but smaller than 300 GiB. The total capacity after each expansion is 150% of the capacity before the expansion. The maximum disk capacity is 300 GiB.

Create an automatic expansion policy.

kubectl create -f StorageAutoScalerPolicy.yaml

Step 3: Verify automatic disk expansion

Create a StatefulSet to test automatic disk expansion.

Use the following template to create a file named StatefulSet.yaml.

Use the following YAML template to create a StatefulSet. The StatefulSet contains a pod that has a 25 GiB disk mounted to it. The mount path is /data.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: pvc-disk
          mountPath: /data
      volumes:
        - name: pvc-disk
          persistentVolumeClaim:
            claimName: disk-pvc
  volumeClaimTemplates:
    - metadata:
        name: pvc-disk
        labels:
          app: nginx
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "alicloud-disk-topology-alltype"
        resources:
          requests:
            storage: 25Gi

Create a StatefulSet.
```
kubectl create -f StatefulSet.yaml
```

View the pod deployment.

kubectl get pod -l app=nginx

Expected output:

NAME      READY   STATUS    RESTARTS   AGE
nginx-0   1/1     Running   0          99s

View the disk capacity.

kubectl exec -it nginx-0 -- df -h /data

Expected output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb         25G   24K   25G   1% /data

Write data to the mounted directory to increase the disk usage to more than 80%. This triggers the first expansion.

Write data to the mounted directory.
The following sample command indicates that 22 GB of data is assigned to the /data/test1 directory so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 22G /data/test1
```

View expansion events.

kubectl get events

When the disk usage is higher than 80%, the disk capacity is 25 GiB, which meets the conditions of action1. Therefore, action1 is automatically executed to increase the capacity by 50 GiB. The following events are displayed in the event list.

2m1s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 25Gi to 75Gi, usedCapacityPercentage:90%, freeSize:2498MB.
2m1s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m1s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
118s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
116s   Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 25Gi to 75Gi, this action action2 will skip.

View the capacity of the PVC.

kubectl get pvc

The expected output shows that the disk was expanded from 25 GiB to 75 GiB after action1 was executed.

NAME               STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
pvc-disk-nginx-0   Bound    d-uf66kkzltnq6xgi9****   75Gi       RWO            alicloud-disk-topology-alltype   <unset>                 26m

Write data to the mounted directory to increase the disk usage to more than 80%. This triggers the second expansion.

Write data to the mounted directory.
The following sample command indicates that 40 GB of data is assigned to the /data/test2 directory so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 40G /data/test2
```

View expansion events.

kubectl get events

When the disk usage is higher than 80%, the disk capacity is 75 GiB, which meets the conditions of action1. Therefore, action1 is automatically executed to increase the capacity by 50 GiB. The limits parameter of action1 specifies that the maximum disk capacity is 100 GiB. Therefore, the disk capacity is expanded from 75 GiB to 100 GiB, as shown in the expected output. The following events are displayed in the event list.

7m4s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0      Start to expand of pvc pvc-disk-nginx-0 from 75Gi to 100Gi, usedCapacityPercentage:84%, freeSize:11927MB.
7m4s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0      waiting for an external controller to expand this PVC
7m4s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0      External resizer is resizing volume d-uf66kkzltnq6xgi9****
7m1s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0      Require file system resize of volume on node
5m59s  Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0      Pvc pvc-disk-nginx-0 is expanding status from 75Gi to 100Gi, this action action2 will skip.

Write data to the mounted directory to increase the disk usage to more than 80%. This triggers the third expansion.

Write data to the mounted directory.
The following sample command indicates that 20 GB of data is assigned to the /data/test3 directory so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 20G /data/test3
```

View expansion events.

kubectl get events

When the disk usage is higher than 80%, the disk capacity is 100 GiB, which meets the conditions of action2. Therefore, action2 is automatically executed to resize the disk from 100 GiB to 150 GiB. The following events are displayed in the event list.

2m40s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 100Gi to 150Gi, usedCapacityPercentage:83%, freeSize:16637MB.
2m40s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m40s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m37s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
109s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 100Gi to 150Gi, this action action2 will skip.

Write data to the mounted directory to increase the disk usage to more than 80%. This triggers the fourth expansion.

Write data to the mounted directory.
The following sample command indicates that 50 GB of data is assigned to the /data/test4 directory so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 50G /data/test4
```

View expansion events.

kubectl get events

When the disk usage is higher than 80%, the disk capacity is 150 GiB, which meets the conditions of action2. Therefore, action2 is automatically executed to increase the current capacity by 50%. Resize the disk from 150 GiB to 225 GiB. The following events are displayed in the event list.

2m42s   Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 150Gi to 225Gi, usedCapacityPercentage:87%, freeSize:19621MB.
2m42s   Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
2m42s   Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
2m38s   Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
114s    Warning   SkipExpand                   persistentvolumeclaim/pvc-disk-nginx-0    Pvc pvc-disk-nginx-0 is expanding status from 150Gi to 225Gi, this action action2 will skip.

Write data to the mounted directory to increase the disk usage to more than 80%. This triggers the fifth expansion.
1. Write data to the mounted directory.
  The following sample command indicates that 50 GB of data is assigned to the /data/test5 directory so that the disk usage is higher than 80%.
```
kubectl exec -it nginx-0 -- fallocate -l 50G /data/test5
```
2. View expansion events.
```
kubectl get events
```
  When the disk usage is higher than 80%, the disk capacity is 225 GiB, which meets the conditions of action2. Therefore, action2 is automatically executed to increase the current capacity by 50%. The limits parameter of action2 specifies that the maximum disk capacity is 300 GiB. Therefore, the disk capacity is expanded from 225 GiB to 300 GiB, as shown in the expected output. The following events are displayed in the event list.
```
17m     Warning   StartExpand                  persistentvolumeclaim/pvc-disk-nginx-0    Start to expand of pvc pvc-disk-nginx-0 from 225Gi to 300Gi, usedCapacityPercentage:82%, freeSize:40351MB.
17m     Normal    ExternalExpanding            persistentvolumeclaim/pvc-disk-nginx-0    waiting for an external controller to expand this PVC
17m     Normal    Resizing                     persistentvolumeclaim/pvc-disk-nginx-0    External resizer is resizing volume d-uf66kkzltnq6xgi9****
17m     Normal    FileSystemResizeRequired     persistentvolumeclaim/pvc-disk-nginx-0    Require file system resize of volume on node
```
  At this time, both action1 and action2 have been triggered to their limits. If the disk capacity exceeds 80% again, the conditions of the two actions cannot be met and expansion operations are no longer triggered.

References

For more information about the issues that can occur when you use disk volumes, see FAQ about disk volumes.