Container Service for Kubernetes:Use CNFS to manage General-purpose CPFS dynamic volumes

Workflow

When you use dynamic volumes, the system automatically provisions a persistent volume (PV) based on the specified StorageClass after your application creates a persistent volume claim (PVC). This method eliminates pre-provisioning and supports automatic volume expansion.

The following figure shows the workflow for mounting a CPFS dynamic volume in a cluster.

Create a CNFS object: Register an existing General-purpose CPFS file system in your cluster. Create a StorageClass: Define a template for dynamic volume provisioning, such as the reclaim policy and mount options, and reference the CNFS object as its backend. Create an application: Create a PVC that specifies the StorageClass. The system automatically creates a PV, binds it to the PVC, and then creates the pod and mounts the volume.

Prerequisites

• Ensure your components meet the version requirements.

  To upgrade your cluster, see Upgrade an ACK cluster.

  1.26 and later

  • Ensure the CSI component version is v1.32.2-757e24b-aliyun or later.

    To upgrade the CSI component, see Manage CSI components.

  • The cnfs-nas-daemon component is installed, and you have added AlinasMountProxy=true to the FeatureGate of the csi-plugin component to enable cnfs-nas-daemon. For more information, see Manage the cnfs-nas-daemon component.

  Earlier than 1.26

  • Ensure the CSI component version is v1.24.11-5221f79-aliyun or later.

    To upgrade the CSI component, see Manage CSI components.

  • Install the client dependencies and restart the csi-plugin component.

    Expand to view the steps

    Configure the csi-plugin ConfigMap to automatically install client dependencies at startup.

    1. Check whether a ConfigMap named csi-plugin exists.

       kubectl -n kube-system get cm csi-plugin

       If the ConfigMap exists

       If a ConfigMap named csi-plugin already exists, update the ConfigMap.

       kubectl edit configmap csi-plugin -n kube-system

       Add the cnfs-client-properties field under data and set cpfs-efc=true to install client dependencies.

       ...
       data:
         cnfs-client-properties: |
           cpfs-efc=true   # Install client dependencies when csi-plugin starts.

       If the ConfigMap does not exist

       If no ConfigMap named csi-plugin exists, create one in the kube-system namespace.

       cat <<EOF | kubectl apply -f -
       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: csi-plugin
         namespace: kube-system
       data:
         cnfs-client-properties: |
           cpfs-efc=true   # Install the client dependencies when csi-plugin starts.
       EOF

    2. Restart the csi-plugin component.

       kubectl -n kube-system rollout restart daemonset csi-plugin

• You have created a General-purpose CPFS file system and its corresponding protocol service in the same VPC as the cluster and obtained the mount target address. For more information, see Create a protocol service and obtain a mount target address.

  For optimal performance, place the CPFS protocol service mount target and the cluster in the same vSwitch.

Limits

• The CPFS general-purpose edition is available only in select regions. For more information, see Available regions for the CPFS general-purpose edition.

• CPFS file systems can be mounted only to clusters within the same VPC.

For more information about the limits of CPFS, see Limits.

Step 1: Create a CNFS object

Create a CNFS object to register the existing CPFS file system with your cluster.

1. Create a file named cnfs.yaml by using the following template.

   apiVersion: storage.alibabacloud.com/v1beta1
   kind: ContainerNetworkFileSystem
   metadata:
     name: cnfs-nfs-cpfs
   spec:
     # Specify the backend storage type as CPFS.
     type: cpfs  
     # Deleting this CNFS object does not delete the backend CPFS file system.
     reclaimPolicy: Retain  
     parameters:
       # The domain name of the mount target for the CPFS protocol service.
       protocolServer: cpfs-xxxx.xxxx.cpfs.aliyuncs.com  
       # Use the NFS client to mount the volume.
       useClient: NFSClient  

   Parameters:

   Parameter	Description
   spec.type	The backend storage type. Set this to cpfs.
   spec.reclaimPolicy	The reclaim policy. Only Retain is supported. This setting prevents the backend CPFS file system from being deleted when the CNFS object is deleted.
   spec.parameters.protocolServer	The mount target domain of the protocol service for the General-purpose CPFS file system.
   spec.parameters.useClient	Set this to NFSClient to use the NFS client for mounting.

2. Create the CNFS object.

   kubectl apply -f cnfs.yaml

3. Check the status of the CNFS object and confirm it is Available.

   kubectl get cnfs cnfs-nfs-cpfs -o jsonpath='{.status.status}'

Step 2: Create a StorageClass

Create a StorageClass to serve as a template for dynamic provisioning and to reference the CNFS object you created.

1. Create a file named sc.yaml by using the following template.

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: cnfs-nfs-cpfs-sc
   # Mount options.
   mountOptions:  
     - nolock,tcp,noresvport
     - vers=3
   parameters:
     # Create a subdirectory-based PV.
     volumeAs: subpath  
     # Reference the CNFS object created earlier.
     containerNetworkFileSystem: "cnfs-nfs-cpfs"  
     # The export path of the General-purpose CPFS protocol service.
     path: "/share"  
     # Data handling policy when a PVC is deleted. `true` archives data instead of deleting it.
     archiveOnDelete: "true"  
   provisioner: nasplugin.csi.alibabacloud.com
   # PV reclaim policy. Only Retain is supported in this example.
   reclaimPolicy: Retain  
   # Allow volume expansion.
   allowVolumeExpansion: true  

   Parameters:

   Parameter	Description
   mountOptions	The mount options. You can use the default values shown in the example.
   parameters.volumeAs	subpath: Subpath mode. Each PV corresponds to an independent subdirectory on the CPFS file system, providing data isolation. sharepath: Sharepath mode. All PVs created from this StorageClass share the same directory on the CPFS file system.
   parameters.containerNetworkFileSystem	The CNFS object to reference.
   parameters.path	The export path of the General-purpose CPFS protocol service, such as /share. You can also specify a subdirectory, such as /share/dir. Currently, only regular directories are supported. Filesets are not supported.
   parameters.archiveOnDelete	This parameter takes effect only when volumeAs is set to subpath and reclaimPolicy is set to Delete. General-purpose CPFS is a shared storage service. This parameter provides a second confirmation before deletion. Controls how subdirectory data is handled when the PVC is deleted. true (default): The subdirectory is archived and renamed to archived-{pvName}.{timestamp} instead of being deleted. You can manually recover the data. false: Permanently deletes the subdirectory and all its files. This data cannot be recovered. This operation deletes only the CPFS subdirectory and its files, not the CPFS file system itself. To delete the CPFS file system, see Delete a file system.
   reclaimPolicy	If csi-provisioner is a managed component, its version must be v1.34.1 or later. There is no version restriction for unmanaged components. This parameter takes effect only when volumeAs is set to subpath. The reclaim policy for the PV. It controls how resources are handled after the PVC is deleted. Delete: Automatically releases the PV when the PVC is deleted. The archiveOnDelete parameter then determines if the backend subdirectory data is also deleted. Retain: When the PVC is deleted, the PV is retained and the backend data is not affected. You must manually clean up the resources. This option is suitable for scenarios with high data security requirements.
   allowVolumeExpansion	If this parameter is enabled, the CPFS volume can be automatically expanded.

2. Create the StorageClass.

   kubectl apply -f sc.yaml

3. Check if the StorageClass was created successfully.

   kubectl get sc cnfs-nfs-cpfs-sc

   Expected output:

   NAME               PROVISIONER                      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
   cnfs-nfs-cpfs-sc   nasplugin.csi.alibabacloud.com   Retain          Immediate           true                   9s

Step 3: Create an application

Create a workload and specify the StorageClass you created in its volumeClaimTemplates section.

1. Create a file named sts.yaml by using the following template.

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: cnfs-nfs-cpfs-sts
   spec:
     selector:
       matchLabels:
         app: nginx
     serviceName: "nginx"
     replicas: 2
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
           volumeMounts:
           - name: pvc
             # Mount the storage volume to the /data directory in the container.
             mountPath: /data  
     # Define a PVC template. The StatefulSet controller dynamically creates a PVC for each pod.
     volumeClaimTemplates:
     - metadata:
         name: pvc
       spec:
         accessModes: [ "ReadWriteOnce" ]
         # Reference the StorageClass created earlier.
         storageClassName: "cnfs-nfs-cpfs-sc"     
         resources:
           requests:
             storage: 50Gi

2. Create the application.

   kubectl apply -f sts.yaml

3. Check if the pods are in the Running state.

   kubectl get pod | grep cnfs-nfs-cpfs-sts

   Expected output:

   cnfs-nfs-cpfs-sts-0   1/1     Running   0          11s
   cnfs-nfs-cpfs-sts-1   1/1     Running   0          9s

4. Confirm that the pods have mounted the CPFS storage volume.

   kubectl exec cnfs-nfs-cpfs-sts-0 -- mount | grep nfs

   The following output indicates that CNFS has successfully mounted the CPFS file system by using the NFS client.

   cpfs-********-********.cn-shanghai.cpfs.aliyuncs.com:/share/nas-804e8cb1-2355-4026-87fc-ee061e14f5f9 on /data type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,noresvport,proto=tcp,port=30000,timeo=600,retrans=2,sec=sys,mountaddr=127.XX.XX.255,mountvers=3,mountport=30000,mountproto=tcp,local_lock=all,addr=127.XX.XX.255)

Step 4: Verify the results

After the application is created, the system automatically provisions and mounts the volume.

kubectl get pvc -l app=nginx

NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       VOLUMEATTRIBUTESCLASS   AGE
pvc-cnfs-nfs-cpfs-sts-0   Bound    nas-8d2530eb-fcf9-4ad5-b5f8-bae02a1*****   50Gi       RWO            cnfs-nfs-cpfs-sc   <unset>                 63s
pvc-cnfs-nfs-cpfs-sts-1   Bound    nas-e84cadaf-ce35-4745-8cbb-2a80026*****   50Gi       RWO            cnfs-nfs-cpfs-sc   <unset>                 61s

kubectl exec cnfs-nfs-cpfs-sts-0 -- mount | grep /data

cpfs-....aliyuncs.com:/share/nas-804e8cb1-2355-4026-87fc-ee061e1****** on /data type nfs (rw,...)

Production best practices

• Performance tuning: General-purpose CPFS is suitable for high-throughput and high-IOPS scenarios. You can adjust NFS mount options, such as rsize and wsize, in the StorageClass's mountOptions to optimize performance for specific workload requirements.

• Data protection: Set reclaimPolicy to Retain and archiveOnDelete to true in the StorageClass to prevent accidental data loss from PVC deletion.

Clean up resources

To avoid unexpected charges and ensure data security, follow this procedure to release resources that you no longer need.

1. Delete the workload

   • Action: Delete all applications that use the relevant PVCs, such as the StatefulSet. This action stops the running pods and unmounts the storage volumes.

   • Example command: kubectl delete statefulset <your-statefulset-name>

2. Delete PVCs

   • Action: Delete the PVCs associated with the application. The handling of backend data is determined by the combination of volumeAs, reclaimPolicy, and archiveOnDelete.

     • volumeAs: subpath

       Each PVC corresponds to an independent subdirectory. The deletion behavior is as follows:

       • reclaimPolicy: Retain: The PV is retained, and the data in the CPFS subdirectory remains unchanged. You must perform a manual cleanup.

       • reclaimPolicy: Delete + archiveOnDelete: "true": The PV is automatically released. The subdirectory is archived and renamed to archived-{pvName}.{timestamp}, and no data is lost.

       • reclaimPolicy: Delete + archiveOnDelete: "false": Automatically releases the PV and permanently deletes the subdirectory and its files. This data cannot be recovered. Proceed with caution.

     • volumeAs: sharepath

       Multiple PVCs share the same directory. Deleting a PVC does not affect the backend directory or its data. The reclaimPolicy and archiveOnDelete parameters have no effect.

     Regardless of the configuration, the CPFS file system itself is not deleted. To delete the CPFS file system, see Delete a file system.

   • Example command: kubectl delete pvc <your-pvc-name>

3. Delete Kubernetes storage resources

   This action only removes the resource definitions within the cluster. It does not delete the backend CPFS file system.

   1. Delete the PV

      • Action: If reclaimPolicy is set to Retain, the PV's status changes to Released after the PVC is deleted. You can then manually delete the PV if you confirm the data is no longer needed.

      • Example command: kubectl delete pv <your-pv-name>

   2. Delete the StorageClass

      • Action: If you no longer need this type of storage, you can delete the corresponding StorageClass.

      • Example command: kubectl delete sc <your-storageclass-name>

   3. Delete the CNFS object

      • Action: If you no longer need to use the CPFS file system with the cluster, you can delete the CNFS object. Deleting a CNFS object does not delete the backend CPFS file system.

      • Example command: kubectl delete cnfs <your-cnfs-name>