You can configure a Deep Learning Containers (DLC) training job to use storage services like OSS, NAS, CPFS, or MaxCompute. To do this, you can either mount the storage or access it directly from your code, which allows your training job to read and write data. This topic explains how to configure these storage services.
Prerequisites
Activate PAI (DLC) and create a workspace. For more information, see Activate PAI and create a default workspace.
(Optional) To configure OSS storage, complete the following:
Activate OSS and grant the required permissions to PAI. For more information, see Activate OSS and Authorize the PAI service account.
Create an OSS bucket. For more information, see Quick start in the console.
(Optional) To use NAS storage, create a general-purpose NAS file system. For more information, see Create a file system.
(Optional) To configure MaxCompute storage, you must activate MaxCompute and create a MaxCompute project. For more information, see Activate MaxCompute and Create a MaxCompute project.
Use OSS storage
Configuration by mounting
When you create a DLC training job, you can mount OSS storage. The following mount types are supported. For specific configuration steps, see Create a training job.
Mount type | Description |
Mount dataset | Mount a custom or public dataset.
Select a dataset of the OSS type and configure the Mount Path. When the DLC job runs, the system accesses the data in OSS using this path. |
Mount storage | Mount an OSS bucket path and use the Read-only switch to set read and write permissions. |
DLC supports mounting OSS using either JindoFuse or ossfs:
JindoFuse: JindoFuse is used by default, but the default configuration for DLC has functional limitations (for more information, see JindoFuse), which makes it unsuitable for all scenarios. You can adjust the parameters to adapt to specific scenarios. For specific instructions, see JindoFuse.
ossfs: When mounting an OSS bucket path using the Mount storage option, you can specify
{"mountType":"ossfs"}in Advanced Settings to use ossfs.
Configuration without mounting
DLC jobs can use the OSS PyTorch Connector or an OSS SDK to read from and write to OSS. When you create a training job, you can configure the relevant code files. For code examples, see OSS Connector for AI/ML or OSS SDK.
Use NAS/CPFS storage
You can use NAS/CPFS storage when you create a Deep Learning Containers (DLC) job by binding a custom dataset of the NAS/CPFS type or mounting storage. For details about the configuration, see Use NAS/CPFS.
Mount type | Description |
Mount dataset | Mount a custom dataset. You can use the Read-only switch to set read and write permissions. |
Mount storage | Mount a NAS/CPFS file system and use the Read-only switch to set read and write permissions. You can also set the nconnect parameter in Advanced Settings to improve NAS access throughput. This Linux NFS client mount option boosts performance by creating multiple TCP connections between the client and server. For more information, see How do I resolve poor NAS access performance on a Linux OS? The following is a parameter example: |
Use MaxCompute storage
To use MaxCompute storage, access it directly from your code instead of mounting it. When you create a training job, you can configure the relevant code files. For code examples, see Use MaxCompute.
FAQ
Q: 'Killed' message for paiio reads
A: This issue occurs due to insufficient memory. Because paiio has no resource limits, reading from MaxCompute can cause high memory consumption. The operating system and other components also consume available memory.