Object Storage Service:Local data cache

Background information

When you use ossfs 2.0 to access OSS, data must be fetched from the remote end each time a file is read. For scenarios that require multiple rounds of repeated reads of the same data, such as AI training and model loading, frequent remote access results in high network latency and bandwidth consumption.

The local data cache of ossfs 2.0 retains previously read file data on a local disk so that subsequent access is served directly from the local disk. This significantly improves the performance of repeated reads and reduces OSS request costs.

The local data cache of ossfs 2.0 has the following features:

• Better performance: In repeated read scenarios for both large and small files, the bandwidth is significantly better than that of ossfs 1.0. For more information, see the performance comparison below.

• Automatic eviction: A built-in LRU eviction policy automatically reclaims the least recently accessed files when the cache is full, without the need for manual cleanup.

• Support for massive numbers of small files: When used together with metadata cache (--max_inode_cache_count and --attr_timeout), it can greatly accelerate multi-round loading of AI training datasets.

Working principles

Cache granularity

The cache uses OSS objects (files) as the basic unit. Each file that has been read corresponds to a cache file in the cache directory, and the cache is filled on demand.

Read process

• First read: Data is fetched from OSS and synchronously written to the local cache directory.

• Subsequent reads: Data is read directly from the local cache.

Space reclamation

When the cache approaches its capacity limit, old files are automatically evicted based on the LRU (least recently used) policy to free up space for new data. Reclamation is performed with a single cache file as the smallest unit.

Configuration methods

Constraints:

• The directory that --disk_data_cache_dir points to must exist and be an empty directory. We recommend that you ensure the cache path resides on an independent partition or independent disk.

• --disk_data_cache_size cannot exceed the available space of the disk where the cache directory resides.

• When you mount multiple ossfs2 instances on the same machine, each instance must use an independent cache directory.

• When the ossfs2 process exits, the cache directory is not automatically cleaned up and must be cleaned up manually.

• After mounting, avoid performing operations on the cache directory path to prevent unknown errors.

Example mount command:

ossfs2 mount /mnt/oss/ \
  --oss_bucket <your-bucket> \
  --oss_endpoint <your-endpoint> \
  --oss_access_key_id <ak> \
  --oss_access_key_secret <sk> \
  --disk_data_cache_dir /mnt/disk/ossfs2/cache \
  --disk_data_cache_size 256G

Environment requirements

Usage recommendations

Local data cache is suitable for scenarios that are read-heavy and write-light and require repeated access to the same data.

Recommendations for configuring the IO engine (disk_data_cache_io_engine):

• psync: Uses synchronous read/write system calls, and data passes through the operating system PageCache during reads. When the amount of cached data is close to or smaller than the available memory, hot data naturally resides in PageCache, achieving read speeds close to memory speeds. This is suitable for general-purpose scenarios, low-specification cloud disks, and tmpfs cache disks (tmpfs does not support libaio).

• libaio: Uses the Linux asynchronous IO interface and bypasses PageCache to operate the disk directly. This is suitable for scenarios with large-capacity local NVMe and datasets much larger than memory, and typically provides an additional performance improvement of about 20% compared with psync.

The following table lists typical scenarios and recommended configurations:

Performance comparison

Test targets: ossfs 2.0.8 vs ossfs 1.91.10. In the table, "first read / subsequent read" correspond to a cache miss (data must be fetched from OSS) and a local cache hit, respectively. "—" indicates that the configuration has no cache and does not distinguish between first and subsequent reads.

Scenario 1: Sequential read of large files (4 threads reading a 100 GB file)

Test environment

• Machine: ecs.i4.4xlarge (16 vCPUs, 128 GiB)

• Local NVMe disk: 3576 GiB NVMe (read bandwidth 6 GB/s, write bandwidth 3 GB/s, IOPS 900,000)

• ossfs 2.0: --disk_data_cache_dir=<path> --disk_data_cache_size=3T

• ossfs 1.0: -ouse_cache=<path> -oparallel_count=128

Results

Conclusion:

• After the cache is enabled, in the subsequent read scenario for large files, ossfs 2.0 delivers about 2.2 times the performance of ossfs 1.0 (libaio mode: 5947 vs 2648 MB/s), reaching the performance ceiling of local NVMe. Meanwhile, CPU usage is significantly reduced (≈250% vs 800%).

Scenario 2: Reading massive numbers of small files (loading the ImageNet training dataset)

Test environment

• Machine: ecs.i4.32xlarge (128 vCPUs, 1024 GiB)

• Local NVMe disks: 8 × 3576 GiB NVMe

• Dataset: ImageNet training set (about 1.3M images), loaded repeatedly across multiple epochs

• ossfs 2.0: --disk_data_cache_dir=<path> --disk_data_cache_size=1500G --disk_data_cache_io_engine=libaio --attr_timeout=36000 --max_inode_cache_count=11000000 (metadata cache enabled)

• ossfs 1.0: -ouse_cache=<path> -oreaddir_optimize -omax_stat_cache_size=11000000 -ostat_cache_expire=72000

Results

Conclusion:

• After disk cache and metadata cache are enabled, the loading performance of ossfs 2.0 in subsequent epochs reaches 2.7 times that of the first read (895 vs 329 MB/s), and OSS network requests are completely eliminated. For AI training workloads that repeatedly load the same dataset across multiple rounds, we strongly recommend ossfs 2.0 local data cache (libaio engine) + metadata cache.