Distributed Training DLC (Deep Learning Containers) runs on Kubernetes and provides a ready-to-use training environment for distributed deep learning workloads. It lets you submit training jobs immediately without configuring clusters, using frameworks like PyTorch, TensorFlow, and Megatron across GPU and Lingjun AI Compute resources.
Key benefits
Diverse compute resources
Runs on Lingjun AI Compute and general-purpose compute resources, including cloud ECS, ECI, Shenlong bare-metal, and Lingjun bare-metal instances. Supports heterogeneous compute scheduling across these resource types.
Wide framework support
Run jobs across more than ten training frameworks — including Megatron, DeepSpeed, PyTorch, TensorFlow, MPI, and XGBoost — without setting up a cluster. DLC provides multiple pre-built images and supports custom runtime environments. Submit jobs from the console, SDK, or CLI.
High stability
For large model training, DLC reduces compute loss and improves training stability through the proprietary AIMaster fault-tolerance engine, the EasyCKPT high-performance checkpoint framework, SanityCheck health detection, and automatic node repair.
High performance
The proprietary AI training acceleration framework supports multiple parallelism strategies — including data parallelism, pipeline parallelism, and operator splitting — with automatic strategy search and memory optimization. Combined with topology-aware scheduling, gradient group fusion, and mixed-precision communication, DLC improves distributed training efficiency. Suited for large model pre-training, continued training, and alignment workloads.
Use cases
Data preprocessing: Run offline parallel preprocessing using custom runtime environments to reduce the complexity of preparing training data.
Large-scale distributed training: Train with thousands of nodes simultaneously using multiple open-source deep learning frameworks, significantly reducing training time.
Offline inference: Run offline inference jobs with DLC to improve GPU utilization during idle periods.
Resource types
PAI provides two resource types for you to choose based on your compute requirements:
Lingjun AI Compute: High-performance compute resources designed for large model training, supporting ultra-large-scale deep learning workloads. Suited for large model training, autonomous driving, and scientific research that require massive compute.
General-purpose compute: Flexible resources for standard training workloads, supporting machine learning tasks of various scales and types.
Both Lingjun AI Compute and general-purpose compute resources are available in the following usage modes:
Resource quota: Purchase Lingjun or general-purpose compute resources with a subscription commitment for long-term, steady training needs.
Public resources: Use Lingjun or general-purpose compute resources on demand when submitting jobs, with pay-as-you-go billing. No upfront purchase required.
Spot instances: Lingjun AI Compute offers spot instances to get AI compute capacity at a lower cost.
Key features
Job creation and management
Create a training job: Submit training jobs from the console, SDK, or CLI, configuring the image, dataset, resource type, and startup command.
View job details: View basic job information, resource view, and operation logs.
Manage training jobs: Stop, clone, share, generate scripts, and delete jobs.
Configure offline scheduling: Automatically submit DLC jobs for incremental training and model tuning when data or hyperparameters are updated.
Storage configuration: Mount or configure Object Storage Service (OSS), Apsara File Storage NAS (NAS), Cloud Parallel File Storage (CPFS).
Job monitoring and notifications
View training job analysis reports: Visualize training metrics and analysis reports with TensorBoard.
Monitor jobs with CloudMonitor or ARMS: View resource usage for DLC jobs or configure alert rules.
Configure notifications: Create notification rules in the PAI workspace event center to track job status changes in real time.
Network and communication acceleration
RDMA configuration: Configure high-performance RDMA (Remote Direct Memory Access) networking to accelerate inter-node communication when using Lingjun AI Compute resources.
ACCL: Alibaba Cloud's proprietary high-performance collective communication library: High-performance collective communication tuned for Alibaba Cloud network characteristics and large model communication patterns, with built-in fault diagnosis and self-healing.
Improve internet access speed with a private gateway: Create a public NAT gateway for the VPC where your instances are located, and configure SNAT so instances can access the internet through a dedicated public gateway at high speed.
Cost optimization
Spot instances: Create DLC jobs using Lingjun spot instances to get AI compute capacity at a lower cost.
Use idle-time resources: Submit low-priority compute jobs that use idle capacity from other quotas to improve resource utilization and reduce training costs without affecting normal workloads.
Elasticity and fault tolerance
Automatic fault tolerance: AIMaster monitors job status and automatically detects and recovers from failures.
Health check: Detect resource health before training starts and automatically isolate faulty nodes.
EasyCKPT: Provides lossless model checkpointing and recovery for PyTorch large model training, with support for resume from checkpoint.
PerfTracker: Online performance analysis and diagnostics: Collects CUDA kernel functions, Python function execution records, and hardware metrics from each worker in real time, and automatically generates analysis reports to identify slow nodes, bottleneck functions, and hang issues.
Billing
DLC charges primarily for compute resources:
Compute resource type | Billing method | Billing subject | Billing rule | Billing stops when |
Public resources | Pay-as-you-go | Duration the DLC job runs (time the job occupies public resources). |
|
|
AI compute resources (general-purpose compute and Lingjun AI Compute) | Subscription | For details, see Billing of AI computing resources. | For details, see Billing of AI computing resources. | Not applicable |
For more details, see Billing for DLC.
Quick start
Using the MNIST handwritten digit recognition example, learn how to run single-node single-GPU or multi-node multi-GPU distributed training with DLC. For details, see Get started with Deep Learning Containers (DLC).
Get help
DLC FAQ: For issues such as job start or stop failures, billing questions, or training errors, see DLC FAQ.
PAI AI assistant (Agentic PAI): Click the
icon in the upper-right corner of the PAI console to launch the PAI AI Assistant (Agentic PAI). Agentic PAI provides user manuals and operational guidelines for end-to-end PAI products. It supports O&M diagnostics for DSW instances, DLC tasks, and EAS services, automatically identifying the root causes of failures and providing troubleshooting recommendations.
References
Create a training job: Submit training jobs from the console, SDK, or CLI, and learn how to configure key parameters.
DLC use cases: Learn how to use DLC through practical examples.