container storage selection - Container Service for Kubernetes

Running workloads in an ACK cluster often necessitates storage solutions for application data persistence, secure storage of sensitive and configuration data, and dynamic provisioning of storage resources. The ACK cluster delivers container storage capabilities through the Container Storage Interface (CSI) plug-in, utilizing Alibaba Cloud storage services to manage both static and dynamic persistent volume resources effectively.

Overview of supported volumes

The Container Storage Interface (CSI) plug-in is the Kubernetes community's recommended solution for implementing container storage. ACK's container storage capabilities are built on the CSI plug-in, seamlessly integrating Elastic Block Storage (EBS), File Storage NAS, Object Storage Service (OSS), and local disks. It is fully compatible with Kubernetes native volume types, including EmptyDir, HostPath, Secret, ConfigMap, and more. The volumes supported by the CSI plug-in include the following:

Container Service for Kubernetes (ACK) facilitates the automatic binding of various Alibaba Cloud storage services—including disks, NAS, OSS, , and local volumes—to pods. Key features, use cases, and associated billing details for each type of storage volume are outlined below.

Note

The Container Storage Interface (CSI) plug-in is capable of both static and dynamic volume provisioning. With static provisioning, you typically need to manually create and edit a PersistentVolume (PV) and PersistentVolumeClaim (PVC) for attachment. For scenarios requiring numerous PVs and PVCs, dynamic provisioning can be utilized to streamline the process. The definitions of PV and PVC are as follows:

PersistentVolume (PV)

A storage resource within the cluster, a PV has a lifecycle independent of any pod that utilizes it. PVs can be provisioned in various forms, depending on the StorageClass type.
PersistentVolumeClaim (PVC)

A request for storage within a cluster. For instance, while pods consume node resources through PVs, PVCs are the mechanism through which PVs are consumed. Should there be a shortage of PVs, PVCs can dynamically provision additional PVs.

Storage service	Static volume	Dynamic volume	Default deployment	Main features	Application scenarios	Billing description
Alibaba Cloud Disk	Supported	Supported	Yes	Non-shared storage. Each cloud disk can only be mounted on a single node.	High I/O, low latency scenarios. Disks are block storage devices and are suitable for use in scenarios that require high I/O performance and low latency. For example, databases and middleware services. Non-shared scenarios Disk volumes are in exclusive mode and can only be used by one pod at a time. They are recommended for use in non-shared data business scenarios. For more information, see disk volumes.	For information about billing items for disks, see block storage billing. For information about disk prices, see ECS product page.
Alibaba Cloud NAS	Supported	Supported	Yes	Shared storage that can provide high-performance and high-throughput storage services.	Shared data scenarios NAS supports multiple pods accessing the same data simultaneously. If you have shared data requirements, NAS is recommended. Big data analysis scenarios NAS file systems provide high throughput and meet the requirement of shared storage access when large numbers of jobs are involved. Web application scenarios NAS file systems can provide storage for web applications and content management systems. Log preservation scenarios If you want to save logs to a storage volume, NAS volumes are recommended. For more information, see NAS volumes.	NAS billing description
Alibaba Cloud OSS	Supported	Supported	Yes	Shared storage, user-mode file system.	Media, image, and other media file read-only scenarios You can read these file types through OSS volumes. Web page, application configuration file read-only scenarios OSSFS has poor network performance but can support some small file reading scenarios. Note OSS volumes are mounted using OSSFS, which is a user-mode file system simulated through FUSE. The write performance is limited when you use OSS volumes. We recommend that you use other storage volumes in scenarios that require high write performance. For more information, see OSS volumes.	OSS billing overview

Alibaba Cloud storage services each have specific usage limits for storage volumes. Please refer to the information below.

Limits

When utilizing the ACK storage CSI plug-in, be mindful of both the plug-in's usage limits and those of the Alibaba Cloud storage products involved.

Cluster version limits

When utilizing the Container Storage Interface (CSI) plug-in, ensure that your ACK cluster is version 1.14 or higher, and set the kubelet runtime parameter --enable-controller-attach-detach to true. Should you need to upgrade your cluster, please refer to how to manually upgrade the cluster.

CSI Plug-in Usage Limits

The Alibaba Cloud CSI plug-in, open-sourced in the community, primarily supports ACK clusters. For non-ACK environments, such as self-built clusters on Alibaba Cloud, configuration, permission management, and networking differences may prevent seamless support. Adaptation may require source code modifications and configuration changes. For more information, visit alibaba-cloud-csi-driver.

Storage Volume Usage Limits

Volume type	Usage limits
Disk volumes	See usage limits.
NAS volumes	NAS is a shared storage service. A persistent volume claim (PVC) that is used to mount a NAS file system can be shared among pods. The CSI component does not support mounting NAS file systems that use the SMB protocol. We recommend that you use file systems that use the NFSv3 protocol. NAS does not support cross-VPC mounting. It only supports mounting to clusters in the same VPC. General-purpose NAS file systems and Extreme NAS file systems have different limits on mounting connectivity, the number of file systems, and file sharing protocols. For more information, see usage limits.
OSS volumes	An OSS bucket can be shared by multiple pods. The names of PVs used by each application must be unique. To mount a subdirectory in an OSS bucket, we recommend that you set the path field of the PV instead of using subPath. If subPath or subPathExpr is used in your business, to avoid mounting errors caused by permission issues, we recommend that you read What do I do if a mounting error occurs when I use subPath or subPathExpr to mount an OSS volume?. OSS volumes are FUSE file systems mounted by using ossfs. OSS volumes are suitable for scenarios where you need to read objects. For example, use OSS volumes when you need to read configuration files, video files, or images. For more information about the limits, see ossfs limits. OSS volumes are not suitable for scenarios where you need to write objects. We recommend that you use the SDK when you need to write objects. For more information about how to use the SDK to write objects, see Best practice for OSS read/write splitting. To directly write data to OSS volumes, we recommend that you use ossfs 1.91 or later. For more information, see Introduction to ossfs 1.91 and later versions and stress testing. When you use a CSI version earlier than 1.28, ossfs runs as a process on each node. It supports the following node OSs: CentOS, Alibaba Cloud Linux, ContainerOS, and Anolis OS. If the OS of the nodes is not supported, update CSI to run ossfs. You cannot perform the chmod or chown operation when OSS volumes are mounted to the root path. To perform these operations, modify the mp_umask setting. For more information, see How do I manage the permissions related to OSS volume mounting?.

Overview of container storage capabilities

The table below describes the storage features supported by different types of ACK clusters.

Storage classification	Storage capabilities	ACK managed and dedicated clusters	ACK Serverless cluster	ACK secure sandbox
Block storage	Mount and unmount disks
	Online resize
	Snapshots
	Container I/O monitoring
	File system	Supports XFS, Ext4	Supports XFS, Ext4	Supports XFS, Ext4
	Block, raw device
	Snapshot restoration
	Disk queue settings
	Data encryption CMK and BYOK
	Multi-zone awareness
	Custom tags
	Cross-host migration
File Storage NAS	Create, mount, and unmount NAS file systems
	Mount and unmount Samba file systems
	Recycle bin (CNFS)
	Dynamic volume - subdirectory/shared directory (CNFS)
	Data encryption CMK (CNFS, Extreme NAS)
	Quota control (CNFS)	Supported only in managed version
	Capacity/I/O monitoring (CNFS)
	Online resize (CNFS)
Object Storage Service (OSS)	Mount and unmount OSS buckets
Object Storage Service (OSS)	Data encryption BYOK
Local storage	Block storage LVM management
	VolumeGroup automated configuration
	Node LVM capacity-aware scheduling
	PMEM Direct Mem
	LVM-managed persistent memory (PMEM)

Container Network File System CNFS

Within the ACK Pro cluster, you can leverage the Container Network File System (CNFS) for managing container storage resources. CNFS enhances NAS and OSS volume performance and Quality of Service (QoS) control by representing Alibaba Cloud's file storage as a Kubernetes object (CRD). This allows for independent management tasks such as creation, deletion, detailing, mounting, monitoring, and scaling. CNFS also offers functionalities including a recycle bin, resource quotas, and storage volume I/O performance monitoring. For additional details on CNFS, refer to Container Network File System CNFS, manage NAS lifecycle, and manage OSS lifecycle.

CSI components

The Container Storage Interface (CSI) plug-in comprises two key components: the CSI-Plugin and the CSI-Provisioner, which facilitate the automatic creation, mounting, and dismounting of volumes. These CSI components come pre-deployed in both ACK managed clusters and ACK dedicated clusters. For detailed information on the CSI components and instructions for upgrading, refer to managing CSI-Plugin and CSI-Provisioner components.

FAQ

How To Determine The Storage Plug-in Mode Used By The Cluster?

To identify the storage plug-in mode, view node annotations in the console or use kubectl to check kubelet parameters.

View node annotations through the console

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Nodes.
In the Operation column of the desired node, click More > Details.

Check the node's Annotations under the Basic Information tab. The presence of volumes.kubernetes.io/controller-managed-attach-detach: true signifies that the cluster storage plug-in is CSI, while its absence indicates FlexVolume.

View kubelet parameters by using commands

Execute the command below to view kubelet parameters.

ps -ef | grep kubelet

Expected output:

--enable-controller-attach-detach=true

If --enable-controller-attach-detach is set to true, this signifies that the cluster's storage plugin is the Container Storage Interface (CSI).
If the --enable-controller-attach-detach parameter is set to false, this signifies that the cluster is utilizing a FlexVolume storage plugin.

How to manually authorize the CSI plug-in?

The CSI plug-in necessitates granting permissions for operations like mounting, unmounting, creating, and deleting volumes, which involve accessing resources from other products. Generally, CSI components come pre-installed in the cluster with the requisite permissions. Should manual authorization be required within the cluster, the CSI plug-in allows for direct authorization via AccessKey or by using RAM roles, which is the default method.

Authorize using AccessKey information
- Enter the AccessKey information directly into the CSI deployment template.
- You can also inject the AccessKey information into environment variables using the Secret method.
Authorize using RAM roles: The CSI plug-in utilizes the AliyunCSManagedCsiRole to access your resources across other cloud services. For more information, see ACK service roles. For details on RAM role authorization, see authorize RAM roles.
- ACK managed cluster: The Token for RAM role permissions required by CSI is stored in a Secret named addon.csi.token. The CSI plug-in can mount this Secret to facilitate RAM role authorization and enable OpenAPI access.
- ACK dedicated cluster: CSI leverages the RAM role associated with the ECS node where its pod resides.