Overview of CSI-based storage solutions - Container Service for Kubernetes

When running workloads in clusters, you will encounter various storage requirements, including the need for persistent application data, storage for sensitive information and configurations, and dynamic provisioning of storage resources. Alibaba Cloud Container Service for Kubernetes (ACK) integrates with Alibaba Cloud's storage services through Container Storage Interface (CSI) plugins, allowing you to provision and manage both statically and dynamically provisioned volumes.

Supported volumes

CSI is the standard and recommended mechanism for exposing storage systems to containerized workloads on Kubernetes. The storage capabilities in ACK are built on this CSI framework, offering deep integration with Alibaba Cloud storage services such as Elastic Block Storage (EBS), Network Attached Storage (NAS), Cloud Parallel File Storage (CPFS), and Object Storage Service (OSS).

In addition to these CSI-based volumes, ACK also fully supports local disks and Kubernetes-native volume types, such as emptyDir, hostPath, secret, and configMap.

The following figure shows the supported CSI-powered storage volumes:

ACK allows pods to be automatically associated with storage services such as Alibaba Cloud disks, NAS, OSS, CPFS, and local volumes. The following table describes the key features, use cases, and billing rules of the volumes.

Note

The CSI plugin allows you to mount statically and dynamically provisioned volumes. To mount a statically provisioned volume, you must manually create both a Persistent Volume (PV) and Persistent Volume Claim (PVC). If you need many PVs and PVCs, use dynamically provisioned volumes instead.

Storage service	Statically provisioned volume	Dynamically provisioned volume	Default ACK storage	Key feature	Scenario	Billing
Alibaba Cloud disks	Supported	Supported	Yes	Non-shared storage. A disk can be mounted only to one node.	High I/O and low latency Disks are block storage devices and are suitable for scenarios that require high I/O performance and low latency, such as databases and middleware services. Non-data sharing A disk can be provisioned only for one pod. You can use disk volumes in scenarios that do not require data sharing. For more information, see Disk volumes.	For more information about the billable items of disks, see Block storage devices. For more information about the pricing of disks, visit the ECS product page.
NAS	Supported	Supported	Yes	Shared storage that provides high performance and high throughput.	Data sharing NAS file systems allow multiple pods to access the same data. We recommend that you use NAS file systems if data needs to be shared. Big data analysis NAS file systems provide high throughput and meet the requirement of shared storage access when large numbers of jobs are involved. Web applications NAS file systems can provision storage for web applications and content management systems. Log storage We recommend that you use NAS volumes to store log data. For more information, see NAS volumes.	Billing of General-purpose NAS file systems
OSS	Supported	Supported	Yes	OSS provides a huge, low-cost, and shared storage space. If you do not need to frequently modify written data, we recommend that you store the data in OSS volumes.	Data sharing OSS is a shared storage type. You can access data on OSS volumes from multiple pods at the same time. The data on OSS volumes is not deleted when the pod is deleted. OSS volumes can be used to share data between pods. Read-only configuration files of websites and applications ossfs provides limited network performance and can be used to read small files. Read-only media files, such as images and audio and video files OSS is suitable for storing unstructured data, such as images, audios, and videos. OSS volumes are mounted using `ossfs`, which may exhibit poor performance in write-intensive workloads, especially those involving random writes. For such cases, use other storage volume types instead. For more information, see OSS volumes.	Billing
CPFS General Edition	Supported	Supported	No	High-performance and high-bandwidth shared storage	CPFS delivers the high throughput required for demanding workloads such as genomic computing and big data analytics, meeting the exceptional performance needs of large-scale clusters. It can also be used as a high-speed cache, allowing you to stage data from slower storage tiers onto a CPFS volume for faster access by your applications. For details, see Use a statically provisioned volume of CPFS General-purpose Edition and Use CNFS to manage isolated CPFS volumes.	CPFS billing overview
CPFS for Lingjun	Supported	Not supported	No	Exceptional throughput and input/output operations per second (IOPS)	Ideal for demanding AI computing workloads such as AI-Generated Content (AIGC) and autonomous driving.	Billing of CPFS for Lingjun
Edge Node Service (ENS)	Supported	Supported	No	Low-latency storage located at the network edge, close to end-users	Not highly available and can only be mounted to a single node. For details, see What is ENS and Use ENS disks in ACK Edge clusters	Pay-as-you-go is supported. For details, see Billing overview.

Limitations

When using the CSI plugin, take note of the following limitations.

Cluster versions

Make sure that the ACK cluster runs Kubernetes 1.14 or later, and the --enable-controller-attach-detach parameter is set to true for kubelet. For more information about how to update an ACK cluster, see Manually upgrade a cluster.

Node OS

Windows nodes are not supported.

CSI plugin

The CSI plugin is officially supported on ACK clusters.

For non-ACK clusters, such as self-managed Kubernetes on Alibaba Cloud or on-premises clusters, the plugin is not officially supported and may require manual adaptation. This is due to potential variations in cluster configuration, permission management, and networking. Users attempting to run the plugin in these environments are encouraged to review the source code and modify the configuration as needed. For more information, see alibaba-cloud-csi-driver.

Limits on volumes

Volume type	Limits
Disk volumes	Disks are non-shared storage. If multi-attach is not enabled for a disk, it can be mounted to only one pod at a time. For more information about multi-attach, see Use the multi-attach and reservation features of NVMe disks. You can mount a disk only to a pod that resides in the same zone as the disk. Cross-zone mounting is not supported. The ECS instance types to which a cloud disk can be attached depend on the category of the disk. When you mount a disk volume to a pod, make sure that the instance type of the ECS instance on which the pod runs supports the category of the disk that you want to mount. For more information about the matching rules between disk categories and ECS instance types, see Overview of instance families.
NAS volumes	NAS is a shared storage service. A PVC that is used to mount a NAS file system can be shared among pods. You cannot use the CSI plugin to mount Server Message Block (SMB) file systems. We recommend that you use the NFSv3 file sharing protocol. You can mount a NAS volume only to ECS instances in the same virtual private cloud (VPC) as the NAS file system. General-purpose and Extreme NAS file systems have different limits, such as the difference in mounting connectivity, the number of file systems, and file sharing protocols. For more information, see Limits.
OSS volumes	To mount a subdirectory in an OSS bucket, we recommend that you set the path field of the PV instead of using subPath. If your business needs to use subpath or subpathExpr configuration, you can avoid mount exceptions caused by incorrect permission configuration or other reasons. For more information, see An exception occurs when you mount an OSS volume using subpath or subpathExpr You cannot perform the chmod or chown operation when OSS volumes are mounted to the root path. To perform these operations, modify the mp_umask setting. For more information, see OSS volume mount permission issues
CPFS (General Edition)	CPFS General-purpose Edition is available only in specific regions. For more information, see Available regions. Only the NFS protocol is supported for mounting. The POSIX protocol is not supported. CPFS supports only nodes that use the x86 architecture. Volumes can be mounted only to clusters that are in the same VPC. You cannot mount volumes to nodes that run the ContainerOS operating system.
CPFS for Lingjun	Region availability: Only available in select regions. For details, see Available regions. Access control: Currently in invitational preview. To request access, submit a ticket. Network isolation: Only supports mounting to clusters within the same virtual private cloud (VPC). Cross-VPC mounting is not supported.

Container storage features

The following table describes the storage features supported by different ACK clusters.

Storage type	Feature	ACK managed cluster and ACK dedicated cluster	ACK Serverless cluster	ACK cluster that supports sandboxed containers
EBS	Mounting and unmounting disks
	Online resizing
	Snapshot
	Container I/O monitoring
	File systems	XFS and ext4 are supported.	XFS and ext4 are supported.	XFS and ext4 are supported.
	Block devices and bare metal devices
	Data restoration from snapshots
	Disk queue settings
	Customer managed key (CMK)-based encryption and Bring Your Own Key (BYOK)-based encryption
	Multi-zone awareness
	Custom labels
	Cross-host migration
NAS	Creating, mounting, and unmounting NAS file systems
	Mounting and unmounting Samba file systems
	Recycle bin (CNFS)
	Subdirectories or shared directories of dynamically provisioned volumes (CNFS)
	CMK-based encryption (CNFS and Extreme NAS file systems)
	Quota limits (CNFS)	Only ACK managed clusters support this feature.
	Capacity and I/O monitoring (CNFS)
	Online resizing (CNFS)
OSS	Mounting and unmounting OSS buckets
OSS	BYOK-based encryption
Local storage	Linux Volume Manager (LVM)-managed block storage
	Automated volume groups
	LVM-managed capacity-aware scheduling
	Persistent memory (PMem) that is directly accessible
	LVM-managed PMem

CNFS

Use Container Network File System (CNFS) to manage container storage resources in ACK Pro clusters for improved the performance of NAS and OSS volumes and quality of service (QoS) control. CNFS allows ACK to create, delete, describe, mount, monitor, and scale individual file storage of Alibaba Cloud by using Kubernetes CustomResourcecDefinitions (CRDs). CNFS also provides features such as recycle bin, resource quota, and I/O performance monitoring of volumes. For details, see CNFS, Manage the lifecycles of NAS file systems, and Manage the lifecycle of OSS buckets.

CSI components

The CSI plugin contains the csi-plugin and csi-provisioner components that are used to automatically create, mount, and unmount volumes. By default, the CSI components are deployed in ACK managed clusters and ACK dedicated clusters. For more information about CSI components and how to update the components, see Manage the csi-plugin and csi-provisioner components.

RBAC permissions

PVs are cluster-level resources, while PVCs are namespace-level resources. If the default roles provided by ACK, such as administrator and O&M engineer, do not meet your access control requirements, you can configure custom RBAC rules. For example, the O&M engineer role grants read and write permissions for PVCs within authorized namespaces and read-only access to PVs across the cluster. However, this role restricts actions such as creating new PVs. You can customize RBAC rules in such cases.

For more information, see Grant RBAC permissions to a RAM user or RAM role.

FAQ

How do I check the storage plugin used by a cluster?

You can check the storage plugin used by a cluster by checking node annotations in the ACK console or checking kubelet parameters in kubectl.

Check node annotations in the ACK console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the one you want to change. In the left navigation pane, choose Nodes > Nodes.
On the Nodes page, find a node that you want to manage, click More in the Actions column, and then select Details.
On the Overview tab, check the annotations of the node. If the volumes.kubernetes.io/controller-managed-attach-detach: true annotation exists, the cluster uses the CSI plugin. Otherwise, the cluster uses the FlexVolume plugin.

Check kubelet parameters in kubectl

Run the following command to check kubelet parameters:

ps -ef | grep kubelet

Expected output:

--enable-controller-attach-detach=true

If the value of the --enable-controller-attach-detach parameter is true, the cluster uses the CSI plugin.
If the value of the --enable-controller-attach-detach parameter is false, the cluster uses the FlexVolume plugin.

How do I manually grant permissions to the CSI plugin?

Before you can use the CSI plugin to mount, unmount, create, and delete volumes, you must grant the plugin the permissions to access other cloud resources. In most cases, the CSI plugin is installed in the cluster by default and granted the relevant permissions. If you want to manually grant permissions to the CSI plugin in your cluster, you can use an AccessKey pair or a Resource Access Management (RAM) role. By default, the system grants permissions to the CSI plugin by using a RAM role.

Use an AccessKey pair
- Specify an AccessKey pair in the CSI deployment template.
- Create a Secret to pass an AccessKey pair as environment variables.
Use a RAM role: The CSI plugin uses the AliyunCSManagedCsiRole role to access your resources of other Alibaba Cloud services. For more information, see ACK roles. For more information about how to grant permissions to RAM roles, see Grant permissions to a RAM role.
- ACK managed clusters: The token of the RAM role used by the CSI plugin is stored in a Secret named addon.csi.token. To grant permissions to the CSI plugin by using the RAM role and allow the plugin to call API operations, you need to only mount the Secret to the plugin.
- ACK dedicated clusters: The CSI plugin uses the RAM role assigned to the Elastic Compute Node (ECS) node on which the pod resides.