Persistent volumes and storage components FAQ - Container Service for Kubernetes

Typical issues

The following issues apply regardless of storage type.

Cloud disk persistent volumes

Type	Issue
Creation	Dynamic Persistent Volume creation fails with an 'InvalidDataDiskCatagory.NotSupported' error Dynamic Persistent Volume creation fails with a 'The specified AZone inventory is insufficient' error Dynamic Persistent Volume creation fails with a 'disk size is not supported' error Dynamic Persistent Volume creation fails with a 'waiting for first consumer to be created before binding' error Dynamic Persistent Volume creation fails with a 'no topology key found on CSINode node-XXXX' error Dynamic Persistent Volume creation fails with a 'selfLink was empty, can't make reference' error Dynamic Persistent Volume creation fails when a Persistent Volume Claim requests less than 20 GiB of capacity
Mounting	A Pod with a mounted Cloud Disk fails to start with a 'had volume node affinity conflict' error A Pod with a mounted Cloud Disk fails to start with a 'can't find disk' error A Pod with a mounted Cloud Disk fails to start with a 'Previous attach action is still in process' error A Pod with a mounted Cloud Disk fails to start with an 'InvalidInstanceType.NotSupportDiskCategory' error A Pod with a mounted Cloud Disk fails to start with a 'diskplugin.csi.alibabacloud.com not found in the list of registered CSI drivers' error A Pod with a mounted Cloud Disk fails to start with a 'Multi-Attach error for volume' error A Pod with a mounted Cloud Disk fails to start with an 'Unable to attach or mount volumes: unmounted volumes=[xxx], unattached volumes=[xxx]: timed out waiting for the condition' error A Pod with a mounted Cloud Disk fails to start with a 'validate error Device /dev/nvme1n1 has error format more than one digit locations' error A Pod with a mounted Cloud Disk fails to start with an 'ecs task is conflicted' error A Pod with a mounted Cloud Disk fails to start with a 'wrong fs type, bad option, bad superblock on /dev/xxxxx missing codepage or helper program, or other error' A Pod with a mounted Cloud Disk fails to start with an 'exceed max volume count' error A Pod with a mounted Cloud Disk fails to start with a 'The amount of the disk on instance in question reach its limits' error How do I modify the configuration of the default StorageClass for Cloud Disks? Can multiple Container Applications use the same Cloud Disk volume?
Usage	An application reports an 'input/output error' during I/O operations on a Cloud Disk mount directory How do I set user access permissions for a Cloud Disk mount directory?
Expansion	Do Cloud Disk volumes support automatic expansion? Cloud Disk expansion fails with a 'Waiting for user to (re-)start a pod to finish File System resize of volume on node' error Cloud Disk expansion fails with an 'only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize' error
Unmounting	Deleting a Pod with a mounted Cloud Disk fails with a 'The specified disk is not a portable disk' error A Cloud Disk fails to unmount when its Pod is deleted, and an 'Orphaned pod' (a pod not managed by ACK) is found in the kubelet logs After a Pod with a mounted Cloud Disk is deleted, the Pod fails to restart due to an unrecoverable mount failure Deleting a Pod with a mounted Cloud Disk fails with a 'target is busy' error Why does a Cloud Disk remain after its associated PVC is deleted? Why does a PVC still exist after it is deleted?
Other	Can a Cloud Disk used as a Volume be converted to the Subscription billing model? In the ECS Console, how do I identify which Cloud Disks on the Block Storage page are associated with volumes?

NAS persistent volumes

Category	Issue
Mounting	Error during mount: chown: Operation not permitted Controller task queue is full when mounting a dynamically provisioned NAS volume The time required to mount a NAS persistent volume increases Error during mount: unknown filesystem type "xxx" Why is my pod stuck in ContainerCreating when I mount two NAS PVCs? How do I mount a NAS file system with TLS using CSI? How do I implement user or group isolation on NAS? Can multiple applications use the same NAS volume? Error when mounting a NAS volume in ACS: failed to do setup volume
Usage	Cannot create or modify directories on a NAS volume NFS stale file handle error during read/write operations
Unmounting	Unmount times out and pod is stuck in Terminating state

OSS persistent volumes

ossfs 1.0

Category	Issue
Mounting	OSS volume mount time is extended OSS volume mount permission issues OSS volume mount failure How do I mount only a specific file in OSS using an OSS volume? How do I use specified ARNs or a ServiceAccount for RRSA authentication? How do I mount an OSS bucket across accounts? How do I enable exclusive mount mode after ossfs is containerized? An exception occurs when you mount an OSS volume using subPath or subPathExpr
Usage	Access to an OSS bucket through an OSS volume is slow File size is 0 in the OSS console An application reports the "Transport endpoint is not connected" error when accessing a mount target An application reports the "Input/output error" error when accessing a mount target A directory is displayed as a file object after being mounted Many abnormal requests are monitored on the OSS server The Content-Type metadata of file objects written through an OSS volume is always application/octet-stream The "Operation not supported" or "Operation not permitted" error is returned when you create a hard link How do I view the records of access to OSS through an OSS volume? How do I restart the ossfs process in shared mount mode? How do I check the ossfs version used to mount an OSS volume?
Scaling	Do I need to scale out a volume when the actual storage capacity exceeds the volume's configuration?
Unmounting	Unmounting a statically provisioned OSS volume fails and the pod remains in the Terminating state

ossfs 2.0

Category	Issue
Mounting	OSS persistent volume mount fails How to mount a single file from an OSS bucket by using a persistent volume How to mount an OSS bucket across different accounts How to use a specific ARN or ServiceAccount for RRSA-based authorization
Scaling	Do I need to scale out a volume when the actual storage capacity exceeds the volume's configuration?
Usage	How to restart the ossfs 2.0 process

Storage components

Category	Issue
Component issues	The CSI component fails to start, and the component log shows a "403 - Forbidden" error The CSI component fails to start due to an image pull failure, with the error "exec /usr/bin/plugin.csi.alibabacloud.com: exec format error" OOM issues caused by storage components High network traffic is observed in the monitoring data of the csi-plugin pod The csi-provisioner component log shows the error "failed to renew lease xxx timed out waiting for the condition"
Component upgrade failures	The pre-check for the csi-plugin component fails The pre-check for the csi-plugin component passes, but the upgrade fails The csi-plugin component is present in the console, but the csi-provisioner component is missing The pre-check for the csi-provisioner component fails The pre-check for the csi-provisioner component passes, but the upgrade fails The csi-provisioner component upgrade fails because the number of cluster nodes or the permissions do not meet the requirements The csi-provisioner component upgrade fails due to changes in StorageClass properties

CNFS

"IPAddress ... for Service ... has a wrong reference" event alert after an ACK cluster upgrade

Symptom

After upgrading the cluster, running kubectl get events -A returns continuous Warning events:

IPAddress: <IP_ADDRESS> for Service kube-system/cnfs-cache-ds-service has a wrong reference; cleaning up

This issue usually occurs in the following scenarios:

The storage-operator component version in the cluster is earlier than v1.33.1.
The cluster is upgraded from a version earlier than 1.33 to version 1.33 or later.

Cause

Versions of storage-operator earlier than v1.33.1 have a known issue where they continuously try to create an already-existing Service. In Kubernetes 1.33 and later, the MultiCIDRServiceAllocator feature is enabled by default. This repetitive creation behavior triggers the feature, causing the system to enter a loop of rapidly creating and deleting temporary IPAddress resources.

Solution

Upgrade the storage-operator component.

Why is `kube-system/cnfs-cache-ds-service` automatically recreated after I manually delete it?

Symptom

After manually deleting cnfs-cache-ds-service in the kube-system namespace, the deletion appears to succeed, but the service reappears shortly after.

Cause

The storage-operator component manages the desired state of cluster resources. Its reconciliation loop works as follows:

Desired state: In the storage-operator ConfigMap, the installation status of cnfs-cache-ds-service is set to true.
Continuous monitoring: The component continuously checks the cluster to confirm the service exists.
Automatic reconciliation: When you delete the service, the controller detects the mismatch and immediately recreates it to match the desired state.

Solution

Option 1: Upgrade the storage-operator component (recommended)

See Upgrade the storage-operator component.

Option 2: Modify the storage-operator configuration (temporary)

This option prevents the cnfs-cache-ds service from being automatically recreated by updating the operator's desired state.

Edit the storage-operator ConfigMap in the kube-system namespace.
```
kubectl edit configmap storage-operator -n kube-system
```
In the data field, locate cnfs-cache-ds and change the value of its install key from "true" to "false".
```
cnfs-cache-ds:
  install: "false"
  # ...other configurations...
```
Save the changes and exit the editor. The storage-operator applies the new configuration.

Delete the service.

kubectl delete service cnfs-cache-ds-service -n kube-system

Typical issues

Cloud disk persistent volumes

NAS persistent volumes

OSS persistent volumes

ossfs 1.0

ossfs 2.0

Storage components

CNFS

"IPAddress ... for Service ... has a wrong reference" event alert after an ACK cluster upgrade

Symptom

Cause

Solution

Why is kube-system/cnfs-cache-ds-service automatically recreated after I manually delete it?

Symptom

Cause

Solution

Why is `kube-system/cnfs-cache-ds-service` automatically recreated after I manually delete it?