Optimize OSS small file reads with strmvol volumes - Container Service for Kubernetes

In containerized applications, a conventional method for mounting object storage data is to use a Filesystem in Userspace (FUSE)-based file system, such as ossfs. However, this approach often falls short in read-intensive scenarios that involve millions of small files, such as loading AI training datasets or analyzing time-series logs, where high throughput and low latency are critical. For these use cases, we recommend using a virtual block device solution. By mounting object storage data directly with a strmvol volume, you can significantly optimize the read performance for small files.

Usage notes

strmvol volumes

Only internal endpoints can be used to access OSS data.
When a virtual block device is initialized, it builds a complete file metadata index. This process consumes node resources and causes the application pod to remain in the ContainerCreating state. The time and resources required are directly proportional to the number of files in the OSS bucket mount path.
- When you select an OSS bucket mount path, follow the principle of least privilege.
  If Business A accesses content under /app/a/ in an OSS Bucket and Business B accesses content under /app/b/, create separate volumes for Business A and Business B with mount target paths of /app/a/ and /app/b/, respectively.
- For information about the time and resource overhead during initialization, see Metadata index construction.
The settings in the PersistentVolume (PV) limit the resource consumption of the virtual block device during its mount lifecycle, excluding the initialization phase. Ensure that you reserve sufficient node resources.
A single virtual block device has a capacity of 16 TiB, which means the mounted OSS path is also limited to 16 TiB of data.

Cluster and node requirements

The cluster must be an ACK managed Pro cluster or an ACK dedicated cluster that runs Kubernetes 1.20 or later. The storage plug-in must be a Container Storage Interface (CSI) plug-in.
Nodes must use kernel version 4.19 or later. We recommend resource specifications of at least 4 vCPU and 8 GiB of memory.
Supported node operating systems: Alibaba Cloud Linux 3, Alibaba Cloud Linux 2, and CentOS 7. To optimize data access performance with the erofs operating system, we recommend using Alibaba Cloud Linux 3.
Note
CentOS 7 and Alibaba Cloud Linux 2 have reached their end of life (EOL). For more information, see [Product Change] Announcement on the Discontinuation of Support for Alibaba Cloud Linux 2 and CentOS 7.
You cannot mount strmvol volumes to a serverless pod that is scheduled to a virtual node.

Prerequisites

Step 1: Deploy the strmvol-csi-driver

To use strmvol volumes, you must deploy a separate CSI driver, the strmvol-csi-driver component. After deployment, this CSI driver operates independently and does not conflict with the csi-provisioner and csi-plugin components managed by ACK.

Log on to the ACK console. In the left-side navigation pane, choose Marketplace > Marketplace.
On the Marketplace page, search for strmvol-csi-driver and click its card.
On the application details page, click Deploy in the upper-right corner.
In the panel that appears, configure the basic information and parameters, and then click OK.

Step 2: Configure OSS access permissions

Create a RAM user and grant the required permissions.
1. Create a RAM user. You can skip this step if you already have a RAM user.
  Go to the Create User page on the RAM Console and follow the on-screen instructions to create a RAM user. You must specify parameters such as the logon name and password.
2. Create a custom policy for OSS access. For more information, see Create a custom policy.
  Choose from the following read-only and read-write permission policies based on your requirements, and replace mybucket with the name of your bucket.
  - OSS read-only policy
    Click to view the read-only policy
    { "Statement": [ { "Action": [ "oss:Get*", "oss:List*" ], "Effect": "Allow", "Resource": [ "acs:oss:*:*:mybucket", "acs:oss:*:*:mybucket/*" ] } ], "Version": "1" }
  - OSS read/write policy
    Click to view the read/write policy
```
{
    "Statement": [
        {
            "Action": "oss:*",
            "Effect": "Allow",
            "Resource": [
                "acs:oss:*:*:mybucket",
                "acs:oss:*:*:mybucket/*"
            ]
        }
    ],
    "Version": "1"
}
```
3. (Optional) If you encrypt OSS objects with a customer master key (CMK) from Key Management Service (KMS), you must also grant the RAM user KMS permissions. For more information, see Encrypt ossfs 1.0 volumes.
4. Grant the OSS permissions to the RAM user. For more information, see Manage RAM user permissions.
5. The AccessKey you create will be stored as a Secret for the PV.
  1. Go to the Users page on the RAM Console, click the name of your RAM user, and then in the AccessKey section, click Create AccessKey.
  2. Follow the on-screen instructions to create an AccessKey. Copy and securely store the AccessKey ID and AccessKey Secret.
Create a secret to store the authentication credentials for accessing OSS data.
The following is an example command. Replace akId and akSecret with your AccessKey ID and AccessKey secret.
```
kubectl create -n default secret generic strmvol-secret --from-literal='akId=xxxxxx' --from-literal='akSecret=xxxxxx'
```

Mount a strmvol volume

Step 1: Create a strmvol volume

Static volume

Create a PersistentVolume (PV).

Create a file named strmvol-pv.yaml with the following content.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-strmvol
spec:
  capacity:
  # The mounted OSS path can store a maximum of 16 TiB of data.
    storage: 20Gi
  # Only the ReadOnlyMany access mode is supported.
  accessModes:
    - ReadOnlyMany
  # To prevent accidental data loss, only the Retain policy is supported. 
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: strmvolplugin.csi.alibabacloud.com
    volumeHandle: pv-strmvol
    # Use the secret created in the prerequisite steps.
    nodeStageSecretRef:
      name: strmvol-secret
      namespace: default
    volumeAttributes:
      bucket: cnfs-oss-test
      path: /subpath
      # strmvol volumes only support accessing OSS data over an internal network.
      url: oss-cn-hangzhou-internal.aliyuncs.com
      umask: "000"
      directMode: "false"
      resourceLimit: "2c4g"

Parameters in nodeStageSecretRef
Parameter
Required
Description
name
Yes
The name of the secret that stores the AccessKey information.
namespace
Yes
The namespace of the secret that contains the AccessKey information.

Parameters in volumeAttributes

Parameter	Required	Description
`bucket`	Yes	The OSS bucket to mount.
`path`	No	The path of the directory to mount within the OSS bucket. This path is relative to the bucket's root. Important Select a mount path based on the principle of least privilege.
`url`	Yes	The internal endpoint for OSS. This must match the endpoint shown on the bucket's Overview page in the OSS console. Common formats for internal endpoints include: `http://oss-{{regionName}}-internal.aliyuncs.com` or `https://oss-{{regionName}}-internal.aliyuncs.com`. Important The `vpc100-oss-{{regionName}}.aliyuncs.com` internal endpoint format is deprecated. Update to a new format as soon as possible.
`umask`	No	The permission mask for the file system after the virtual block device is mounted. For example, to set the default file permissions to `755`, set `umask` to `022`.
`directMode`	No	Specifies whether to enable direct mode. `"true"`: Enables direct mode, which disables data prefetching and local caching. This is suitable for random read scenarios, such as the random batch reading of training datasets. `"false"`: Disables direct mode (default). This is suitable for general-purpose scenarios, such as sequential reads of small files or reads of large files. If your application does not have a specific data access pattern, keep this mode disabled.
`resourceLimit`	No	The maximum node resources that the virtual block device can consume. For example, `"2c4g"` indicates that the virtual block device can use up to 2 vCPUs and 4 GiB of memory from the node. Note Memory is primarily used for data prefetching and local caching. When direct mode is enabled, memory usage is significantly lower than the preset value. For operating systems other than Alibaba Cloud Linux 3, read performance is similar regardless of the configuration. Therefore, high resource limits are not recommended. For more information, see Data read performance tests.

Create the PV.
```
kubectl create -f strmvol-pv.yaml
```

Check the status of the PV.

kubectl get pv pv-strmvol

Expected output:

NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pv-strmvol   20Gi       ROX            Retain           Available                          <unset>                          18s

Create a PersistentVolumeClaim (PVC).

Create a file named strmvol-pvc-static.yaml with the following content.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-strmvol
  namespace: default
spec:
  # The following settings must match the PV configuration.
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 20Gi
  volumeName: pv-strmvol

Create the PVC.

kubectl create -f strmvol-pvc-static.yaml

Check the status of the PVC.

kubectl get pvc pvc-strmvol

The expected output shows that the PVC is now bound to the PV.

NAME            STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-strmvol-2   Bound    pv-strmvol   20Gi       ROX                           <unset>                 16s

Dynamic volume

Create a StorageClass.

Create a file named strmvol-sc.yaml with the following content.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: strmvol-test
parameters:
  # Use the secret created in the prerequisite steps.
  csi.storage.k8s.io/node-stage-secret-name: strmvol-secret  
  csi.storage.k8s.io/node-stage-secret-namespace: default
  bucket: cnfs-oss-test 
  path: /subpath
  # strmvol volumes only support accessing OSS data over an internal network.
  url: oss-cn-hangzhou-internal.aliyuncs.com 
  umask: "000"
  directMode: "false"
  resourceLimit: "2c4g"
provisioner: strmvolplugin.csi.alibabacloud.com
# To prevent accidental data loss, only the Retain policy is supported. 
reclaimPolicy: Retain
volumeBindingMode: Immediate

The parameters in parameters are described as follows:

Secret configuration

Parameter	Required	Description
`csi.storage.k8s.io/node-stage-secret-name`	Yes	The name of the secret that stores the AccessKey information.
`csi.storage.k8s.io/node-stage-secret-namespace`	Yes	The namespace of the secret that contains the AccessKey information.

Volume configuration

Parameter	Required	Description
`bucket`	Yes	The OSS bucket to mount.
`path`	No	The path of the directory to mount within the OSS bucket. This path is relative to the bucket's root. Important Select a mount path based on the principle of least privilege.
`url`	Yes	The internal endpoint for OSS. This must match the endpoint shown on the bucket's Overview page in the OSS console. Common formats for internal endpoints include: `http://oss-{{regionName}}-internal.aliyuncs.com` or `https://oss-{{regionName}}-internal.aliyuncs.com`. Important The `vpc100-oss-{{regionName}}.aliyuncs.com` internal endpoint format is deprecated. Update to a new format as soon as possible.
`umask`	No	The permission mask for the file system after the virtual block device is mounted. For example, to set the default file permissions to `755`, set `umask` to `022`.
`directMode`	No	Specifies whether to enable direct mode. `"true"`: Enables direct mode, which disables data prefetching and local caching. This is suitable for random read scenarios, such as the random batch reading of training datasets. `"false"`: Disables direct mode (default). This is suitable for general-purpose scenarios, such as sequential reads of small files or reads of large files. If your application does not have a specific data access pattern, keep this mode disabled.
`resourceLimit`	No	The maximum node resources that the virtual block device can consume. For example, `"2c4g"` indicates that the virtual block device can use up to 2 vCPUs and 4 GiB of memory from the node. Note Memory is primarily used for data prefetching and local caching. When direct mode is enabled, memory usage is significantly lower than the preset value. For operating systems other than Alibaba Cloud Linux 3, read performance is similar regardless of the configuration. Therefore, high resource limits are not recommended. For more information, see Data read performance tests.

Create the StorageClass.
```
kubectl create -f strmvol-sc.yaml
```

Create a PVC.

Create a file named strmvol-pvc-dynamic.yaml with the following content.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-strmvol
  namespace: default
spec:
  # Only the ReadOnlyMany access mode is supported.
  accessModes:
    - ReadOnlyMany
  # Specify the StorageClass.
  storageClassName: strmvol-test
  resources:
    requests:
      # The mounted OSS path can store a maximum of 16 TiB of data.
      storage: 20Gi

Create the PVC.

kubectl create -f strmvol-pvc-dynamic.yaml

Check the status of the PVC.

kubectl get pvc pvc-strmvol

The expected output shows that the PVC is bound to a PV automatically provisioned by the CSI driver.

NAME          STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
pvc-strmvol   Bound    strmvol-d8d1d22a-e1d7-4caa-b875-54f378dec769   20Gi       ROX            strmvol-test   <unset>                 3m

Step 2: Create an application and mount the volume

Create a file named strmvol-test.yaml with the following content.

The following YAML example creates a StatefulSet with one pod. The pod requests storage resources by using a PVC named pvc-strmvol, and the mount path is /data.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: strmvol-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: strmvol-test
  template:
    metadata:
      labels:
        app: strmvol-test
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: pvc-strmvol
          mountPath: /data
      volumes:
        - name: pvc-strmvol
          persistentVolumeClaim:
            claimName: pvc-strmvol

Create the StatefulSet.
```
kubectl create -f strmvol-test.yaml
```

Check the deployment status of the pod in the StatefulSet.

kubectl get pod -l app=strmvol-test

Expected output:

NAME             READY   STATUS    RESTARTS   AGE
strmvol-test-0   1/1     Running   0          14s

Verify that the mount point is a block device and that the application can access the data in OSS.

kubectl exec -it strmvol-test-0 -- sh -c "df /data && ls /data"

Expected output:

Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/ublkb1        24812 24812         0 100% /data
<data in OSS mount path>

Container Service for Kubernetes:Optimize OSS small file reads with strmvol volumes

Usage notes

Prerequisites

Step 1: Deploy the strmvol-csi-driver

Step 2: Configure OSS access permissions

Mount a strmvol volume

Step 1: Create a strmvol volume

Static volume

Dynamic volume

Step 2: Create an application and mount the volume

Related documentation