All Products
Search
Document Center

Container Service for Kubernetes:Use CNFS to automatically expand NAS volumes

Last Updated:Jul 01, 2024

You can define automatic expansion policies for Apsara File Storage NAS (NAS) volumes that are managed by Container Network File System (CNFS) to automatically expand the NAS volumes when the storage usage exceeds a specific threshold. This topic describes how to use CNFS to automatically expand NAS volumes.

Prerequisites

  • A Container Service for Kubernetes (ACK) is created. The Container Storage Interface (CSI) plug-in is used as the volume plug-in of the cluster.

    • If you want to use a new cluster, select the CSI volume plug-in and the Dynamically Provision Volumes by Using Default NAS File Systems and CNFS, Enable NAS Recycle Bin, and Support Fast Data Restore option when you create the cluster.

    • If you want to use an existing cluster and the Dynamically Provision Volumes by Using Default NAS File Systems and CNFS, Enable NAS Recycle Bin, and Support Fast Data Restore option is not selected when you create the cluster, you can use CNFS to manage NAS file systems. For more information, see Use CNFS to manage NAS file systems.

  • The versions of csi-plugin and csi-provisioner are 1.20.5-ff6490f-aliyun or later. For more information about how to update csi-plugin and csi-provisioner, see Update csi-plugin and csi-provisioner.

  • The default dnsPolicy of csi-plugin is ClusterFirst. If you want to use CNFS to automatically expand NAS volumes, you must change the default dnsPolicy from ClusterFirst to ClusterFirstWithHostNet.

  • The version of storage-operator is 1.18.8.56-2aa33ba-aliyun or later. For more information, see storage-operator.

  • A kubectl client is connected to the cluster.

Step 1: Enable automatic scale-up

The default storage-auto-expander in the storage-operator component manages automatic storage expansion. To enable automatic scale-up, run the following command to modify the ConfigMap file of the storage-operator component.

kubectl patch configmap/storage-operator \
  -n kube-system \
  --type merge \
  -p '{"data":{"storage-auto-expander":"{\"imageRep\":\"acs/storage-auto-expander\",\"imageTag\":\"\",\"install\":\"true\",\"template\":\"/acs/templates/storage-auto-expander/install.yaml\",\"type\":\"deployment\"}"}}'

Step 2: Configure an auto expansion policy

  1. View the status of the CNFS object. Make sure that the CNFS object is in the Available state.

    1. Run the following command to query the CNFS object:

      kubectl get cnfs

      Expected output:

      NAME                                      AGE
      default-cnfs-nas-837d6ea-20210819155623   14d
    2. Run the following command to view the status of the CNFS object:

      kubectl get cnfs <The CNFS object queried in the previous step> -o yaml | grep Available

      Expected output:

      status: Available
  2. Use the following YAML template to create an auto expansion policy for a NAS volume.

    cat << EOF | kubectl apply -f -
    apiVersion: storage.alibabacloud.com/v1alpha1
    kind: StorageAutoScalerPolicy
    metadata:
      name: hybrid-expand-policy
    spec:
      pvcSelector:
        matchLabels:
          app: nginx      # You must specify the label that is added to the persistent volume claim (PVC) and the Deployment to which the NAS volume is mounted. In this example, app: nginx is specified. 
      namespaces:
        - default
        - nginx
      conditions:
        - name: condition1
          key: volume-capacity-used-percentage
          operator: Gt
          values:
            - "80"
      actions:
        - name: action1
          type: volume-expand
          params:
            scale: 100%
            limits: 500Gi
    EOF

    Parameter

    Description

    pvcSelector

    The label that is used to match the expansion policy with the PVC that is used to mount the NAS volume. In this example, nginx is used.

    namespaces

    The namespace of the PVC that is used to mount the NAS volume. If multiple namespaces are specified, the logical operator between the namespaces is OR. Default value: default. In this example, default and nginx are specified.

    conditions

    The conditions that trigger the action. The logical operator between multiple conditions is AND. Each condition includes the following parameters:

    • key: the type of the metric.

    • volume-capacity-used-percentage: specifies that the storage usage threshold is expressed in percentage.

    • operator: the operator, which can be Gt (greater than), Lt (less than), Eq (equal to), or Ne (not equal to). The parameter value is not case-sensitive.

    • values: the threshold value.

    In the example, the condition specifies that the action is triggered when the storage utilization of the PVC exceeds 80%.

    actions

    The actions that are performed when the preceding conditions are met. Multiple actions are allowed. Each action includes the following parameters:

    • type: the type of the action. Only volume-expand is supported.

    • scale: the size of the storage that you want to add to the volume. Valid units: GiB and percentage (%).

    • limits: the maximum capacity of the PVC if the action is performed.

    If multiple actions are specified in the actions section, the first action whose conditions are met is performed. The other actions are skipped.

    In this example, action1 specifies that if the conditions are met, the NAS volume is expanded by 100%. The NAS volume can be expanded to up to 500 GiB.

  3. Use the following YAML template to create a PVC and a Deployment.

    Important

    The auto expansion policy is applied to the PVC and the Deployment that have the specified label. In this example, pvcSelector.matchLabels is set to app: nginx in the auto expansion policy. In this case, the policy applies to the PVC and the Deployment whose labels parameter is set to app: nginx.

    cat << EOF | kubectl apply -f -
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: cnfs-nas-pvc
      labels:
        app: nginx   # You must specify the same value as the pvcSelector.matchLabels parameter in the YAML template of the expansion policy. In this example, app: nginx is specified. 
    spec:
      accessModes:
        - ReadWriteMany
      storageClassName: alibabacloud-cnfs-nas
      resources:
        requests:
          storage: 50Gi
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: cnfs-nas-deployment
      labels:
        app: nginx
    spec:
      selector:
        matchLabels:
          app: nginx   # You must specify the same value as the pvcSelector.matchLabels parameter in the YAML template of the expansion policy. In this example, app: nginx is specified. 
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.14.2
            volumeMounts:
            - mountPath: "/data"
              name: cnfs-nas-pvc
          volumes:
          - name: cnfs-nas-pvc
            persistentVolumeClaim:
              claimName: cnfs-nas-pvc
    EOF

Step 3: Verify that the NAS volume can be automatically expanded

  1. Run the following command to query all pods in the default namespace:

    kubectl get pod

    Expected output:

    NAME                                                            READY   STATUS    RESTARTS   AGE
    cnfs-nas-deployment-56dbcc7fb7-wh79z   1/1      Running            0          20m
  2. Run the following command to log on to the application pod:

    kubectl exec cnfs-nas-deployment-56dbcc7fb7-wh79z -ti sh
  3. Run the following command to write 50 GiB of data to the directory to which the NAS volume is mounted:

    cd /data
    dd if=<data path> of=<mounted path>
  4. Run the following command to check events related to volume expansion:

    kubectl get events

    Expected output:

    default    0s    Warning   NotEnoughDiskSpace           persistentvolumeclaim/cnfs-nas-pvc              Pvc cnfs-nas-pvc is not enough disk space, namespace: default, totalSize:50Gi, usedSize:49Gi, usedPercentage:98.00%, threshold:85.00%
    default    1s    Warning   StartExpand                  persistentvolumeclaim/cnfs-nas-pvc              Start to expand of pvc cnfs-nas-pvc from 50Gi to 100Gi, usedCapacityPercentage:98%, freeSize:1024MB.
    default    0s    Normal    Resizing                     persistentvolumeclaim/cnfs-nas-pvc              External resizer is resizing volume nas-1acba306-e626-46f3-8441-110c10a6****
    default    0s    Warning   ExternalExpanding            persistentvolumeclaim/cnfs-nas-pvc              Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
    default    0s    Normal    FileSystemResizeRequired     persistentvolumeclaim/cnfs-nas-pvc              Require file system resize of volume on node
    default    0s    Normal    FileSystemResizeSuccessful   pod/cnfs-nas-deployment-56dbcc7fb7-wh79z        MountVolume.NodeExpandVolume succeeded for volume "nas-1acba306-e626-46f3-8441-110c10a6****"

    The preceding output indicates that the storage usage of the NAS volume exceeds 80%, which triggers a volume expansion. The NAS volume is expanded from 50 GiB to 100 GiB.

  5. Check the monitoring information about CNFS in the CSI Nodes dashboard.

    1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

    2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Operations > Prometheus Monitoring.

    3. On the Prometheus Monitoring page, click the Storage Monitoring tab, and then click the CSI - Nodes tab.

    4. On the CSI - Nodes tab, set StorageType and Namespace based on the NAS volume, and select the PVC that is used to mount the NAS volume. Then, you can view the capacity of the NAS volume in the Total Capacity chart.

      In this example, set StorageType to nas, Namespace to default, and Pvc to cnfs-nas-pvc. NAS存储卷容量

      The preceding chart shows that the expansion occurred at 16:55:30 and the NAS volume is expanded to 100 GiB.