The container storage capability of Container Service for Kubernetes (ACK) clusters is developed based on the Container Storage Interface (CSI) plug-in. This capability is deeply integrated with Elastic Block Storage (EBS), Apsara File Storage NAS (NAS) of Alibaba Cloud, Object Storage Service (OSS), and local disks of Alibaba Cloud, and is completely compatible with Kubernetes-native storage services, such as emptyDir volumes, hostPath volumes, Secrets, and ConfigMaps. This topic describes the overview, features, and limits of the CSI plug-in, and the permissions that are required to use the CSI plug-in.
ACK container storage architecture
ACK allows you to configure storage services to be automatically mounted to pods. The storage services include Alibaba Cloud disks, NAS, OSS, and local volumes. The following table describes the features and use scenarios of the storage services.
CSI supports dynamically provisioned OSS volumes. For more information about how to use dynamically provisioned OSS volumes, see Manage the lifecycle of OSS buckets.
Alibaba Cloud storage service | Statically provisioned volume | Dynamically provisioned volume | Default ACK storage | Key feature | Scenario |
Alibaba Cloud disks | Supported | Supported | Yes | Non-shared storage. A disk can be mounted only to one node. |
For more information, see Disk volume overview. |
NAS | Supported | Supported | Yes | Shared storage that provides high performance and high throughput. |
For more information, see NAS volume overview. |
OSS | Supported | Supported | Yes | Shared storage that supports file systems in user space. |
Note OSS volumes are mounted by using ossfs, which is implemented as a file system in user space (FUSE). The write performance is limited when you use OSS volumes. We recommend that you use other storage volumes in scenarios that require high write performance. For more information, see OSS volume overview. |
You can use the CSI plug-in to mount storage resources as statically provisioned volumes and as dynamically provisioned volumes in ACK clusters. To mount a storage resource as a statically provisioned volume, you must manually create a persistent volume (PV) and a persistent volume claim (PVC). When large numbers of PVs and PVCs are required, you can mount storage resources as dynamically provisioned volumes. Definitions of PV and PVC:
PV
A PV is a piece of storage in the cluster. The lifecycle of a PV is independent of the lifecycle of the pod that has the PV mounted. Different types of PVs can be provisioned by using different types of StorageClasses.
PVC
A PVC is a request for storage in the cluster. PVs are node resources consumed by pods. PVCs are claims that consume PVs. When PVs are insufficient, PVCs can dynamically provision PVs.
ACK container storage features
The following table describes the storage features supported by different types of ACK cluster.
Storage type | Feature | ACK cluster (Linux) | ACK Serverless cluster | Registered cluster (hybrid cloud or multi-cloud) | ACK Edge | ACK cluster (Windows) | Container Service cluster (Apsara Stack) | ACK cluster with sandboxed containers |
Block storage | Mount and unmount disks | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
Online resizing | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Snapshot | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Container I/O monitoring | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
File system | XFS, ext4, and dBFS are supported. | XFS and ext4 are supported. | ![]() | ![]() | NTFS is supported. | XFS and ext4 are supported. | XFS and ext4 are supported. | |
Block devices and bare metal devices | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Data restoration from snapshots | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Disk queue settings | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Customer managed key (CMK)-based encryption and Bring Your Own Key (BYOK) encryption | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Multi-zone awareness | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Custom labels | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Cross-host migration | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
File storage | Create, mount, and unmount NAS file systems | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
Mount and unmount Samba file systems | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Recycle bin (CNFS) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Subdirectories or shared directories of dynamically provisioned volumes (CNFS) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
CMK-based encryption (CNFS and Extreme NAS file systems) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Quota limits (CNFS) |
| ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Capacity and I/O monitoring (CNFS) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Online resizing (CNFS) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Object storage | Mount and unmount OSS buckets | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
BYOK-based encryption | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
Local storage | Linux Volume Manager (LVM)-managed block storage | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
Automated volume groups | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
LVM-managed capacity-aware scheduling | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
PMEM Direct Mem | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
LVM-managed persistent memory (PMEM) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
CSI deployment architectures
The CSI plug-in consists of two parts: CSI-Plugin and CSI-Provisioner. The following figure shows the deployment architectures of the CSI plug-in in an ACK managed cluster and an ACK dedicated cluster.
The CSI plug-in is automatically installed in ACK managed clusters and ACK dedicated clusters. If you use an ACK Serverless cluster or an ACK Edge cluster, you must manually install the CSI plug-in.
ACK managed clusters | ACK dedicated clusters |
![]() In ACK managed clusters, CSI-Provisioner and CSI-Plugin are deployed on worker nodes. | ![]() In ACK dedicated clusters, CSI-Provisioner is deployed on master nodes. CSI-Plugin is automatically deployed as DaemonSets on master and worker nodes. |
Permissions required to use CSI
Before you can use the CSI plug-in to mount, unmount, create, and delete volumes, you must grant the plug-in the permissions to access other cloud resources. You can use an AccessKey pair or a Resource Access Management (RAM) role to grant permissions to the CSI plug-in. The default method is to use a RAM role. The following table describes the two authorization methods.
Use an AccessKey pair | Use a RAM role |
| The CSI plug-in uses the RAM role AliyunCSManagedCsiRole to access resources of other cloud services. For more information the RAM role, see ACK default roles. For more information about how to grant permissions to RAM roles, see Grant permissions to a RAM role.
|
Limits on CSI
When you use the CSI plug-in in ACK clusters, take note of the limits on the CSI plug-in and Alibaba Cloud storage services.
Limits on Alibaba Cloud storage services
Alibaba Cloud storage service
Limit
Alibaba Cloud disks
You can provision a disk as a volume only for one pod.
You cannot mount disks of all types to ECS instances of all types. For more information, see Overview of instance families.
You cannot mount or unmount subscription disks as volumes.
You can mount a disk only to an ECS instance in the same zone as the disk.
We recommend that you create StatefulSets instead of Deployments to use disk volumes.
NoteDeployments are used to create stateless applications. When a pod is restarted, the start time of the new pod may overlap with the end time of the old pod. If multiple pods are created for a Deployment, no dedicated volume is provisioned for each pod.
The minimum capacity of a disk volume is 20 GiB.
NAS
You can mount a NAS volume only to ECS instances in the same virtual private cloud (VPC) as the NAS file system.
You cannot use the CSI plug-in to mount Server Message Block (SMB) file systems.
The number of NAS file systems that you can create is limited. To request a quota increase, join the DingTalk group 35532895.
OSS
We recommend that you do not perform data write operations on OSS volumes. Use other storage media for data write operations.
NoteOSS volumes are mounted by using ossfs, which is implemented as a file system in user space (FUSE). The write performance is limited when you use OSS volumes. We recommend that you use other storage volumes in scenarios that require high write performance.
Limits on the CSI plug-in
The CSI plug-in is an open source plug-in for ACK clusters. In other types of clusters, such as clusters deployed in third-party clouds and self-managed clusters on Alibaba Cloud, you cannot directly use the CSI plug-in for reasons such as cluster configurations, permission management, and network differences. If you want to use the CSI plug-in in these types of clusters, you must modify the cluster configurations based on the source code. For more information, see alibaba-cloud-csi-driver.
Requirements on Kubernetes versions
To use the CSI plug-in in an ACK cluster, the Kubernetes version of the cluster must be 1.14 or later. In addition, the --enable-controller-attach-detach
parameter must be set to true
.
Installation and updates of the CSI plug-in
For more information about how to install and update the CSI plug-in, see Install and upgrade the CSI plug-in.
Differences between the CSI and FlexVolume plug-ins
Plug-in | Feature | Reference |
Flexvolume | FlexVolume is a traditional mechanism to extend the storage systems developed by the Kubernetes community. ACK supports FlexVolume. FlexVolume consists of the following parts:
| For more information about FlexVolume, see FlexVolume overview. For more information about how to update FlexVolume, see Manage components. |
CSI | The Kubernetes community recommends the CSI plug-in. The CSI plug-in provided by ACK is compatible with the features of the community version. CSI consists of the following two parts:
| For more information about CSI, see CSI overview and alibaba-cloud-csi-driver. |
You must select a plug-in when you create an ACK cluster.
You cannot use CSI and FlexVolume in the same cluster.
You cannot change the plug-in from FlexVolume to CSI for a cluster.
Recommendation
For newly created ACK clusters, we recommend that you use CSI. The ACK technical team will continuously update CSI to support more features of the CSI community version.
For existing clusters, we recommend that you use the plug-in that is already installed. The ACK technical team will continue its support for FlexVolume.
How to check the storage plug-in used in a cluster
You can confirm the volume plug-in used by the cluster by checking node annotations in the ACK console or checking kubelet parameters in the CLI.
Check node annotations in the ACK console
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
Select a node and click
in the Actions column.On the Overview tab, check the annotations of the node.
If
volumes.kubernetes.io/controller-managed-attach-detach: true
is displayed, the cluster uses the CSI plug-in.If
volumes.kubernetes.io/controller-managed-attach-detach: true
is not displayed, the cluster uses the FlexVolume plug-in.
Check kubelet parameters in the CLI
Run the following command to check kubelet parameters:
ps -ef | grep kubelet
Expected output:
--enable-controller-attach-detach=true
If the value of
--enable-controller-attach-detach
istrue
, the cluster uses the CSI plug-in.If the value of
--enable-controller-attach-detach
isfalse
, the cluster uses the FlexVolume plug-in.