Container Service for Kubernetes:Best practices for OSS persistent volume performance tuning

Internet endpoint configured on the PV

When your cluster and OSS bucket are in the same region, use an internal endpoint to significantly reduce network latency.

Diagnosis:

kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeAttributes.url}'

Fix: If the output shows an Internet endpoint such as http://oss-<region-id>.aliyuncs.com, recreate the PV with an internal endpoint: http://oss-<region-id>-internal.aliyuncs.com.

OSS server throttling

OSS enforces limits on the total bandwidth and queries per second (QPS) for each bucket. In some regions and high-concurrency scenarios (such as workflows), bandwidth or QPS throttling can occur.

Diagnosis: In the OSS console, check the monitoring metrics for the OSS bucket. Look for bandwidth usage and request counts that approach or exceed the limits.

Fixes:

Node bandwidth bottleneck

The client and the application pod run on the same node and share its network resources. In serverless scenarios, they share the resources of a single ACS instance. The node's network bandwidth limit constrains the maximum OSS persistent volume performance.

Diagnosis: Check the node monitoring information or ECS instance monitoring information for a bandwidth bottleneck.

Fixes:

• Schedule the application pod to a node with higher available network bandwidth.

• Upgrade or downgrade the node configuration to get more bandwidth.

• If the task is I/O-intensive and neither solution is feasible, switch to a disk-based persistent volume.

ossfs 1.0 disk throughput limit

The ossfs 1.0 client caches data to disk by default to support full POSIX write operations and ensure data consistency for a single client. This caching uses the /tmp directory of the ossfs pod, which maps to the node's container runtime data disk. The input/output operations per second (IOPS) and throughput caps of this disk directly limit ossfs 1.0 performance.

Diagnosis:

This method does not apply to ContainerOS nodes.

1. Find the container runtime data disk ID on the node where the application pod runs: Example output:

   If the ID is empty, go to the ECS console - Elastic Block Storage - Disks. Locate the disk by its attached instance, then query its monitoring information.

       kubectl get node <node-name> -o yaml | grep alibabacloud.com/data-disk-serial-id

       alibabacloud.com/data-disk-serial-id: uf69tilfftjoa3qb****

2. View the disk monitoring information to determine whether the disk has an IOPS or throughput bottleneck.

Fixes:

High disk usage for ossfs 1.0 temporary data

If the container runtime data disk for ossfs 1.0 caching stays at consistently high usage (especially in serverless scenarios), frequent eviction and rotation of temporary data further degrades performance.

Diagnosis:

1. Identify the container runtime data disk as described in ossfs 1.0 disk throughput limit.

2. Check disk usage through monitoring. If usage stays high but the application itself writes little data to disk, the performance issue is likely caused by ossfs 1.0 temporary data rotation.

Fixes:

1. Check application logic. Confirm whether the application holds file descriptors for extended periods (keeps files open). While a file descriptor is held, its related temporary data cannot be released, consuming large amounts of local disk space.

2. Increase disk capacity. If the application logic is correct and the current disk's maximum throughput meets requirements, increase the disk capacity.

Short RAM authentication TTL

When you use a Resource Access Management (RAM) role for authentication, the ossfs client (versions 1.0 and 2.0) obtains a token and records its expiration time on first access to OSS. To ensure session continuity, the client requests a new token 20 minutes before the current token expires.

The max_session_duration for the RAM role must be longer than 20 minutes.

Example: With a 30-minute session duration, the client refreshes the token after 10 minutes of use (30 - 20 = 10), which does not affect throughput. With a session duration under 20 minutes, the client treats the token as always about to expire and tries to get a new token before every OSS request. This triggers excessive RAM requests and degrades performance.

Diagnosis: The default client log level may not capture frequent token refreshes. Verify that the RAM role configuration is correct.

Fix: See Set the maximum session duration for a RAM role to check and update the maximum session duration.

Tune metadata caching

Metadata caching reduces time-consuming network requests for high-frequency operations such as ls and stat. If you disable or misconfigure it, the client frequently requests data from OSS to maintain consistency, creating a performance bottleneck.

Diagnosis:

kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeAttributes.otherOpts}'

Fixes:

• Remove cache-disabling options. If the output includes max_stat_cache_size=0 (ossfs 1.0) or close_to_open=true (ossfs 2.0) and your scenario does not require strong consistency, recreate the PV without these parameters.

• Increase cache size and TTL. If your application reads data that is updated infrequently, increase the metadata cache size and expiration time:

  Client	Configuration reference
  ossfs 1.0	Metadata cache
  ossfs 2.0	Reduce OSS requests and improve mount target performance
  strmvol	All metadata under the mount target is cached by default and does not expire. No configuration needed.

Disable extended object information (ossfs 1.0 only)

File system metadata such as mode, gid, and uid are extended information for OSS objects. The ossfs 1.0 client makes extra HTTP requests to retrieve this information. Although the client caches metadata (including basic and extended properties) by default, frequent retrieval of extended properties remains a bottleneck.

Two optimization principles apply:

1. Minimize the number of metadata retrieval operations. See Tune metadata caching.

2. Avoid requesting high-overhead extended properties to reduce the time cost of each retrieval.

Fixes:

Optimize write operations

OSS objects support overwrite only. When the client handles file modifications, it reads the complete object from OSS to the local machine, then uploads the entire file after the file is closed. If the application frequently opens, writes, and closes files, it repeatedly downloads and uploads the full data.

Fixes:

• Use batch writes instead of multiple writes. Prepare content in memory and write it in a single operation. Some library functions such as Java's FileUtils.write include a complete open, write, and close flow. Calling such functions in a loop causes repeated full-file uploads.

• Use temporary files for frequent modifications. If a file requires multiple modifications during a processing task, copy it from the OSS mount target to a temporary path in the container (such as /tmp). Perform all modifications on the local file. Copy the final version back to the OSS mount target in one operation.

• Enable appendable objects for append-write scenarios. If your application only appends small amounts of data (such as writing logs), use ossfs 2.0 with enable_appendable_object=true. The client then uses OSS appendable objects and does not download and re-upload the entire file on each append. Limitations:

  • If the object already exists but is not an appendable object, this approach does not work.

  • enable_appendable_object is a global parameter for the entire persistent volume. All write operations on the volume use the AppendObject API. This affects the performance of overwriting large files but not reading them. Create a dedicated persistent volume for append-write workloads.

Optimize concurrent reads (model loading scenarios)

When multiple processes concurrently read different locations within a single large file, the access pattern resembles random reads. This causes the client to read data redundantly, resulting in abnormally high bandwidth usage. This commonly happens during AI model loading, where multiple concurrent processes read the same model parameter file.

If ossfs 2.0 performance in this scenario is worse than ossfs 1.0, or significantly different from the ossfs 2.0 stress testing benchmarks, this is likely the cause.

Fixes:

Data prefetching (recommended). Before the application pod starts, prefetch data by using high-bandwidth concurrent sequential reads to load model files into the node's page cache. Set the container memory limit slightly higher than the total model size to reduce the risk of the page cache being reclaimed under memory pressure.

Standalone prefetching script (reads file content to /dev/null concurrently):

#!/bin/bash
# Set the maximum concurrency.
MAX_JOBS=4

# Check if a folder is provided as an argument.
if [ -z "$1" ]; then
    echo "Usage: $0 <folder_path>"
    exit 1
fi

DIR="$1"

# Check if the folder exists.
if [ ! -d "$DIR" ]; then
    echo "Error: '$DIR' is not a valid folder."
    exit 1
fi

# Use find to locate all regular files and use xargs to concurrently run cat to /dev/null.
find "$DIR" -type f -print0 | xargs -0 -I {} -P "$MAX_JOBS" sh -c 'cat "{}" > /dev/null'

echo "Finished reading all files."

Integrate this logic into the pod's lifecycle.postStart hook:

# ...
spec:
  containers:
  - image: your-image
    env:
    # Folder path where model files are located. If not configured or the folder does not exist, prefetching is skipped.
    - name: MODEL_DIR
      value: /where/is/your/model/
    # Concurrency for the prefetching script. Default is 4.
    - name: PRELOADER_CONC
      value: "8"
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "CONC=${PRELOADER_CONC:-4}; if [ -d \"$MODEL_DIR\" ]; then find \"$MODEL_DIR\" -type f -print0 | xargs -0 -I {} -P \"$CONC\" sh -c 'cat \"{}\" > /dev/null'; fi"]
# ...

Refactor application logic. Switch from multi-process concurrent reads within a single file to concurrent reads of multiple different files. This leverages the high throughput advantage of OSS.

Why is an OSS persistent volume much slower than a local disk?

This is expected. An OSS persistent volume is a network file system. All operations go through network transmission and protocol conversion, so its latency and throughput differ from directly attached local storage. Evaluate performance against the published benchmarks:

• ossfs 1.0 baseline performance

• ossfs 2.0 baseline performance

• strmvol baseline performance

Why is write performance much worse than read performance?

This is related to the overwrite-only nature of OSS objects. When the client handles file modifications, it first downloads the entire object, then uploads the entire file after modification. This round-trip has high overhead. For optimization guidance, see Optimize write operations.

How do I determine if my application performs random writes?

In a test environment, switch the mounted persistent volume type from ossfs 1.0 to ossfs 2.0. If the application encounters an EINVAL (Invalid argument) error related to file writing, it is performing random write operations that ossfs 2.0 does not support.