Best practice for using disk volumes in ACS clusters - Container Compute Service

This topic describes the best practice for using disk volumes in ACS clusters, including suggested StorageClass configurations and suggested configurations for mounting disk volumes as ephemeral storage or persistent storage.

Introduction to disk volumes

Alibaba Cloud Elastic Block Storage (EBS) provides a high-throughput, low-latency persistent storage solution for computing instances. It is the core infrastructure that supports compute-intensive scenarios, such as high-performance computing (HPC), AI, and big data analysis. For more information, see Overview of Block Storage

ACS supports the following types of disks:

cloud_essd_entry: ESSD Entry disk.
cloud_auto: ESSD AutoPL disk.
cloud_essd: Enterprise SSD (ESSD). This is the default value.
cloud_ssd: standard SSD.
cloud_efficiency: ultra disk.

Note

Standard SSDs and ultra disks are previous-generation disks, which are discontinued in some regions and zones. ACS GPU-accelerated pods do not support standard SSDs or ultra disks. We recommend that you preferably use the ESSD series to ensure compatibility.

Suggested StorageClass configurations

You can configure StorageClasses to dynamically provision storage resources and conduct topology-aware scheduling.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-essd
parameters:
  type: cloud_auto,cloud_essd
  fstype: xfs
provisioner: diskplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The key parameters of a StorageClass must follow these principles:

Storage type binding: Explicitly claim the disk type by specifing parameters.type. In this example, cloud_auto,cloud_essd indicates that cloud_auto (ESSD AutoPL disks) is preferably used, and cloud_essd (ESSD disks) is used when ESSD AutoPL disks are out of stock. ESSD disks support performance levels (from PL0 to PL3) defined by performanceLevel. You can define performance levels to meet I/O requirements in HPC, AI training, and other scenarios.
Binding mode optimization: We recommend that you set volumeBindingMode to WaitForFirstConsumer in order to schedule pods and then create disks based on the zones of the pods. With the help of the elastic provisioning mechanism of Alibaba Cloud, you can adjust the topology of pods and volumes based on affinity rules in case pods and volumes (PVs and PVCs) are spread across different zones.
File system selection: Explicitly claim the file system type by specifying parameters.fstype. XFS file systems are suggested because they support large metadata blocks and delayed logging, which can augment large-scale parallel I/O performance.

How to use disk volumes in ACS clusters

Overview

Characteristic	Ephemeral storage	Persistent storage
Data TTL	Disks are bound to pods, and disks are deleted together with pods.	Disks are independent of pods. You need to manually delete PVs and PVCs.
Data restoration	Not supported. Data is permanently deleted and cannot be restored.	Supported. PVCs are reserved for recreating pods.
Storage cost	Disks are created on demand and billed based on the lifecycle of the pods.	Disks are billed until they are released.
Applicable scenarios	Temporary caching and intermediate computing result storage.	Databases, logging systems, and stateful applications.

Note

ACK clusters whose version is 1.24 or earlier do not support ephemeral storage.

Based on the preceding characteristics, we recommend that you follow these rules when you choose between ephemeral storage and persistent storage:

Prioritize ephemeral storage: If your system can tolerate business data loss and requires fast scaling, such as stateless web applications, we recommend that you choose ephemeral storage.
Use persistent storage only: In scenarios that require data persistence, cross-pod data sharing, or high availability, such as running MySQL clusters within Kubernetes clusters, you must use persistent storage.

A proper storage solution can efficiently help you balance resource utilization and data reliability in ACS clusters. In addition, with the help of Alibaba Cloud storage services, such as disk snapshots and cross-zone replication, the disaster recovery capability of your system can be further improved.

Ephemeral storage

Ephemeral storage is a lightweight storage solution in Kubernetes. This solution is based on volumes of the ephemeral type. The mechanism behind this solution is binding the lifecycle of storage resources to pods. When a pod is deleted, the PVC and PV associated with the pod are also deleted. The deleted data cannot be restored. This solution is suitable for temporary storage scenarios, such as temporary caching, log caching, intermeidate computing result storage, and temporary data processing.

This solution has the following advantages:

Resource isolation: Each pod occupies separate storage resources. This helps avoid performance contention or data contamination caused by storage resource sharing.
Simplified declarative configuration: volumeClaimTemplate is built into the pod template, which eliminates the need to create PVCs and PVs in advance and simplifies the resource management procedure.

The following YAML template creates a Deployment and mounts a disk volume as ephemeral storage to the pods created by the Deployment.

Click to view the YAML template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ephemeral-example
  labels:
    a1libabacloud.com/compute-class: performance
spec:
  replicas: 2
  selector:
    matchLabels:
      pod: example-pod
  strategy:
    type: Recreate      
  template:
    metadata:
      labels:
        pod: example-pod
    spec:
      containers:
        - name: nginx
          image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
          resources:
            requests:
              cpu: 500m
              memory: 2Gi
              ephemeral-storage: 2Gi
          volumeMounts:
          - mountPath: "/scratch"
            name: scratch-volume
      nodeSelector:
        kubernetes.io/hostname: virtual-kubelet-cn-hangzhou-j
      tolerations:
      - operator: Exists
        key: virtual-kubelet.io/provider
      volumes:
        - name: scratch-volume
          ephemeral:      # Explicitly claim ephemeral storage.
            volumeClaimTemplate:
              metadata:
                labels:
                  type: scratch-volume
              spec:
                accessModes: [ "ReadWriteOncePod" ]
                storageClassName: alicloud-essd
                resources:
                  requests:
                    storage: 30Gi

In the YAML template, take note of the following key parameters:

      volumes:
        - name: scratch-volume
          ephemeral:      # Explicitly claim ephemeral storage.
            volumeClaimTemplate:
              metadata:
                labels:
                  type: scratch-volume
              spec:
                accessModes: [ "ReadWriteOncePod" ]
                storageClassName: alicloud-essd
                resources:
                  requests:
                    storage: 30Gi

StorageClass selection: Configure StorageClassName to specify an existing StorageClass.
Access mode configuration: We recommend that you set accessModes to ReadWriteOncePod (RWXO). The RWXO mode is suggested for ephemeral volumes. In this mode, a volume can be read and written by only one pod, which ensures data consistency.
Storage capacity planning: Adjust the value of storage based on the actual needs of pods. If the value is too small, the storage space will be insufficient. An excessively large value results in resource waste. We recommend that you monitor the usage of ephemeral-storage in Prometheus or CloudMonitor and adjust the value accordingly.

Persistent storage

Persistent storage is implemented through the StatefulSet controller, with the core being that the lifecycle of PVC and PV is independent of the pod. Even if a pod is terminated due to scaling in, failure, or manual deletion, the PVC and PV still retain the data and can be rebound to a new pod. This mechanism is suitable for stateful applications (such as databases, message queues) or scenarios that require cross-pod data sharing.

StatefulSet ensures storage persistence through the following mechanisms:

Independent PVC management: Each pod is associated with a separate PVC, which is generated based on volumeClaimTemplates. However, the lifecycle of the PVC is decoupled from the pod.
Ordered restoration: The StatefulSet controller prioritizes volume restoration when recreating pods in order to ensure data consistency.

The following YAML template creates a StatefulSet and mounts a disk volume as persistent storage to the pods created by the StatefulSet.

Click to view the YAML template

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-statefulset
  labels:
    alibabacloud.com/compute-class: performance
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      nodeSelector:
        kubernetes.io/hostname: virtual-kubelet-cn-hangzhou-j
      tolerations:
      - operator: Exists
        key: virtual-kubelet.io/provider
      containers:
        - name: mysql
          image: mysql:5.7
          volumeMounts:
          - name: mysql-data
            mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alicloud-essd
      resources:
        requests:
          storage: 50Gi

In the YAML tempalte, take note the following key parameters:

  volumeClaimTemplates:
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alicloud-essd
      resources:
        requests:
          storage: 50Gi

StorageClass selection: Configure StorageClassName to specify an existing StorageClass.
Access mode configuration: We recommend that you set accessModes to ReadWriteOnce (RWO) to ensure that the volume is mounted only to one node.
Data persistence: The PVC will be retained after the pod is deleted. If you recreate the pod by running the kubectl delete pod -n <namespace> <pod-name> command, and the new pod will be reassociated with the retained PVC.