Container Service for Kubernetes:ossfs 2.0 persistent volume FAQ

Pod fails to start with FailedMount due to AccessKey permissions

The pod reports FailedMount because the AccessKey lacks required permissions.

To resolve this:

1. Verify that the RAM user's access policy meets OSS mounting requirements. See Use a statically provisioned ossfs 2.0 volume.

2. Confirm the AccessKey is not disabled or rotated.

Pod on a virtual node fails with "secrets is forbidden"

The pod event contains:

failed to get secret secrets "xxx" is forbidden: User "serverless-xxx" cannot get resource "secrets" in API group "" in the namespace "xxx"

On virtual nodes (Container Compute Service pods), the Secret referenced by nodePublishSecretRef must be in the same namespace as the PersistentVolumeClaim (PVC).

Create the Secret in the PVC namespace. In the PV, set nodePublishSecretRef to reference this Secret. See Authenticate using a RAM user's AccessKey.

Pod fails with "mounter.sock: connect: no such file or directory"

The pod event contains:

FailedMount /run/fuse.ossfs/xxxxxx/mounter.sock: connect: no such file or directory

The ossfs 2.0 pod did not start correctly or was unexpectedly deleted.

1. Check whether the ossfs 2.0 pod exists on the node where the application pod is scheduled:

   • If the pod exists but is not Running, troubleshoot it. After it reaches Running, restart the application pod to trigger a remount.

   • If the pod does not exist, proceed to the next step.

      kubectl -n ack-csi-fuse get pod -l csi.alibabacloud.com/volume-id=<PV_NAME> -owide | grep <NODE_NAME>

2. (Optional) Check audit logs for unexpected pod deletion. Common causes: cleanup scripts, node drains, and auto-healing. Adjust configurations to prevent recurrence.

3. Check for remaining VolumeAttachment resources:

      kubectl get volumeattachment | grep <PV_NAME> | grep <NODE_NAME>

4. Restart the application pod to trigger a remount and verify that a new ossfs 2.0 pod is created.

Mount fails because the bucket uses mirroring-based back-to-origin

If the OSS bucket uses mirroring-based back-to-origin and the mount directory is not synchronized from the source, the mount fails.

Synchronize the data from the source before mounting. See Back-to-origin overview.

Mount fails because the bucket has static website hosting enabled

With static website hosting enabled, ossfs 2.0 directory checks redirect to files such as index.html, causing the mount to fail.

Disable or adjust static website hosting before mounting. See Static website hosting.

Mount a single file from an OSS bucket

ossfs 2.0 mounts an OSS bucket path as a file system -- it cannot mount a single file. To access a specific file, mount the parent directory and use subPath in volumeMounts.

For example, to mount a.txt and b.txt from bucket:/subpath to different pods at /path/to/file/:

PV configuration -- mount the parent directory:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-oss
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: pv-oss
    volumeAttributes:
      bucket: bucket
      path: subpath              # Parent path containing a.txt and b.txt
      url: "oss-cn-hangzhou.aliyuncs.com"
      fuseType: ossfs2

Pod volumeMounts -- use subPath to select the file:

  volumeMounts:
    - mountPath: /path/to/file   # Maps to bucket:/subpath
      name: oss-pvc              # Must match the volume name
      subPath: a.txt             # Relative path within bucket:/subpath

After mounting, /path/to/file/a.txt in the pod maps to bucket:/subpath/a.txt.

See Use a statically provisioned ossfs 2.0 volume.

Mount an OSS bucket from a different account

Use RRSA (RAM Roles for Service Accounts) to mount a bucket across accounts.

Verify that your cluster and CSI component versions meet RRSA requirements.

The following procedure mounts a bucket in Account B to a cluster in Account A.

In Account B (where the bucket resides):

1. Create a RAM role named roleB that trusts Account A. See Create a RAM role for a trusted Alibaba Cloud account.

2. Grant roleB permissions to access the target OSS bucket.

3. In the RAM console, open the roleB details page and copy its ARN. Example: acs:ram::130xxxxxxxx:role/roleB.

In Account A (where the cluster resides):

1. Create a RAM role named roleA for RRSA. Set the trusted entity type to OIDC IdP.

2. Grant roleA permission to assume roleB. roleA needs no OSS access policy -- attach a policy with the sts:AssumeRole action, such as AliyunSTSAssumeRoleAccess. See Enable RRSA in a cluster (static volumes) or Use a dynamically provisioned ossfs 1.0 volume (dynamic volumes).

Configure the volume -- set assumeRoleArn to the ARN of roleB:

• Statically provisioned volume (PV): Add to volumeAttributes:

    assumeRoleArn: <ARN of roleB>

• Dynamically provisioned volume (StorageClass): Add to parameters:

    assumeRoleArn: <ARN of roleB>

Use CoreDNS to resolve OSS access endpoints

To resolve OSS endpoints via an internal cluster domain, configure a DNS policy for the ossfs pod to prioritize CoreDNS during mounting.

Requires CSI component v1.34.2 or later. To upgrade, see Manage CSI components.

• Static volume (PV): Add dnsPolicy to spec.csi.volumeAttributes.

  dnsPolicy: ClusterFirstWithHostNet

• Dynamic volumes (StorageClass): Add dnsPolicy to parameters.

  dnsPolicy: ClusterFirstWithHostNet

Use a custom ARN or ServiceAccount for RRSA

By default, setting roleName in the PV lets CSI automatically retrieve the Role ARN and OIDC Provider ARN. For more control -- such as using a third-party OIDC IdP or non-default ServiceAccount -- specify roleArn and oidcProviderArn directly.

roleArn and oidcProviderArn must be configured together. When set, they override roleName.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-oss
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ossplugin.csi.alibabacloud.com
    volumeHandle: pv-oss               # Must match the PV name
    volumeAttributes:
      bucket: "oss"
      url: "oss-cn-hangzhou.aliyuncs.com"
      authType: "rrsa"
      fuseType: "ossfs2"
      oidcProviderArn: "<oidc-provider-arn>"
      roleArn: "<role-arn>"
      # roleName: "<role-name>"        # Ignored when roleArn and oidcProviderArn are set
      serviceAccountName: "csi-fuse-<service-account-name>"

Exceeding configured volume capacity

OSS does not enforce capacity limits on buckets or subdirectories. The .spec.capacity (PV) and .spec.resources.requests.storage (PVC) values are ignored -- ensure they match between the bound PV and PVC.

If actual usage exceeds the configured capacity, the volume operates normally. No scaling is needed.

Restart the ossfs 2.0 process

After modifying credentials or the ossfs 2.0 version, the running process does not pick up changes automatically. Applying changes requires restarting both the ossfs 2.0 pod (csi-fuse-ossfs2-* in ack-csi-fuse) and all application pods using the volume, which causes a service interruption.

Procedure:

1. Identify the target ossfs 2.0 pod. Replace <pv-name> and <node-name>:

      kubectl -n ack-csi-fuse get pod -l csi.alibabacloud.com/volume-id=<pv-name> -owide | grep <node-name>

2. Identify all application pods using this volume. Replace <ns> and <pvc-name>. The Used By field lists the pods:

      kubectl -n <ns> describe pvc <pvc-name>

      Used By:       oss-static-94849f647-4****
                     oss-static-94849f647-6****
                     oss-static-94849f647-h****

3. Find application pods on the same node as the ossfs 2.0 pod:

      kubectl -n <ns> get pod -owide | grep <node-name>

4. Delete all application pods simultaneously. Use kubectl scale or equivalent. When no pods reference the mount, CSI reclaims the csi-fuse-ossfs2-* pod. After replicas restore, the volume remounts with updated configuration.

Alternative approach -- if simultaneous deletion is not possible (for example, a Deployment, StatefulSet, or DaemonSet immediately recreates deleted pods), or the application tolerates temporary OSS read/write failures:

1. Find the VolumeAttachment for the volume. Expected output:

      kubectl get volumeattachment | grep <pv-name> | grep <node-name>

      csi-bd463c719189f858c2394608da7feb5af8f181704b77a46bbc219b**********   ossplugin.csi.alibabacloud.com   <pv-name>   <node-name>   true   12m

2. Delete the VolumeAttachment. Application pods will return a disconnected error on OSS operations.

3. Restart application pods one by one. Each connects to a new csi-fuse-ossfs2-* pod with updated configuration.