Object Storage Service:Use OSS Connector for AI/ML to access and store OSS data in PyTorch training jobs

Item

Do not use OSS Connector for AI/ML

Use OSS Connector for AI/ML

Performance

You must manually optimize performance, which may be inefficient.

OSS Connector for AI/ML automatically optimizes the performance of OSS data download and checkpoint storage.

Data loading method

You must download data in advance, which increases costs and management workloads.

OSS Connector for AI/ML supports stream load to reduce cost and management complexity.

Data access

You must read and write data by using adapters, which increases access complexity.

OSS Connector for AI/ML directly reads and writes data in OSS to simplify access.

Configuration difficulty

You must compile code, which makes configuration difficult.

OSS Connector for AI/ML provides simple configuration options to improve development efficiency.

Item

Feature

Class

Method

Map-style dataset

Supports random access for quick retrieval of specific data during training.

OssMapDataset

The OssMapDataset and OssIterableDataset classes provide the same methods to build a dataset.

• from_prefix()

  Builds a dataset from an OSS_URI prefix. Ideal when OSS data storage paths follow uniform rules.

• from_objects()

  Builds a dataset from a list of OSS_URIs. Ideal when OSS data storage paths are known but scattered.

• from_manifest_file()

  Create a manifest file and use the manifest file to build a dataset. Ideal for datasets with a large number of files (tens of millions), frequent loading, and data indexing enabled on the bucket.

Iterable-style dataset

Supports sequential streaming reads for efficiently processing large volumes of continuous data.

OssIterableDataset

Checkpoint API operations

Loads checkpoints from OSS during model training and saves checkpoints to OSS at regular intervals, simplifying the training workflow.

OssCheckpoint

• OssCheckpoint()

  Initializes an OssCheckpoint object for reading and writing checkpoints during model training.

• reader()

  Reads checkpoints from OSS.

• writer()

  Writes checkpoints to OSS.