The text-to-video distillation template builds high-quality text-video paired SFT datasets at scale. Starting from a small set of seed prompts, it runs a pipeline of prompt augmentation, video generation, and frame-level scoring to produce training data for text-to-video model fine-tuning.
Template overview
This template generates text-to-video training data at scale. Starting from a small set of seed prompts, it augments and diversifies the prompts, aligns them to the generation model's instruction format, has a text-to-video model batch-generate videos, then scores the generated videos on prompt consistency, visual quality, and subject consistency. Low-quality samples are filtered out, and the result is a text-to-video SFT dataset ready for fine-tuning.
This template supports single-stage creation only: distillation data construction. It doesn't include a student model training stage.
The text-to-video distillation pipeline consists of the following steps:
Step | Name | Description |
1 | Data augmentation | Augment a small set of seed prompts into a larger, more diverse set of video prompts, then balance and filter them. |
2 | Data filtering | Filter and balance the augmented prompts to retain high-quality samples. |
3 | Instruction alignment | Align the prompts to the instruction template preferred by the generation model. |
4 | Teacher generation | A text-to-video model batch-generates videos from the aligned prompts. |
5 | Quality scoring | Score the generated videos by frame sampling on dimensions such as prompt consistency, visual quality, and subject consistency. |
6 | Quality filtering | Filter out low-quality samples by threshold. The output is a text-to-video SFT dataset ready for fine-tuning. |
Procedure
Log on to the PAI console. In the left-side navigation pane, choose Model Application > Model Distillation (ModelDistill).
Click Create task to go to the Distillation Template tab, or navigate directly to the Distillation Template tab.
Select Text-to-Video Distillation, click Use Template, and go to the Create Distillation Task page. Configure the key parameters described below and keep the default values for other settings.
Configure the Basic Configuration:
Task name: Auto-filled by the system.
Output Path: Specify an OSS path that you have permission to access, such as
oss://mybucket.oss-cn-hangzhou-internal.aliyuncs.com/t2v-distill/.NoteCreate a separate output directory for each distillation task to prevent files from being overwritten. The OSS bucket must be in the same region as the PAI service.
Click Next to go to Build distillation data:
Teacher model: Text-to-video distillation requires three teacher models (instruction builder model is required; text-to-video model and judge model are optional). See the Configuration parameters section below for details.
Dataset: Select an OSS directory or file that you have permission to access. You can download and use the sample data.
Distillation Configuration: Default values are pre-filled. Modify them as needed.
generation.resolution: Video resolution. Default: 720P. Supported values: 480P, 720P, 1080P. Higher resolution increases generation cost.
generation.duration: Video duration in seconds. Default: 5. Valid range: [3, 15].
Compute Resource: Set Resource Type to General Computing, set Source to Public Resources, and select an appropriate Instance Type from Job Resource, such as
ecs.c6.large.
Click Create, and then click OK in the billing confirmation dialog. The page redirects to the basic configuration page of the task details, where you can track the task status.
Configuration parameters
This section describes the key configuration parameters specific to the text-to-video distillation template. Basic configuration and resource configuration are the same as general distillation. Only template-specific parameter differences are listed here.
Basic configuration
Basic configuration includes the task name and output path. The output path is the root directory for all task artifacts, including the distillation dataset. You don't need to specify a separate output path for the dataset.
Distillation data construction
Teacher model
The text-to-video distillation template requires multiple teacher models, each playing a different role in the pipeline:
No. | Model role | Description |
1 | Instruction-Building Model(required) | A text or vision language model responsible for prompt augmentation and instruction alignment. Choose a model with strong instruction-following and descriptive capabilities. |
2 | Text-to-Video Model(required) | The teacher model that generates the actual videos. Video generation is slow — a single video can take up to 30 minutes. |
3 | Judge Model (optional) | A vision language model that scores the generated videos by frame sampling. If left empty, the instruction builder model is reused. However, text-only models typically can't parse video frames — specify a separate model. |
Dataset
Select an input directory or file. The data format requirements are as follows:
Format: JSONL, one video prompt seed per line.
The prompt field must be named
prompt.Only text prompts are supported. Don't include image fields.
Sample data:
{"id": "t2v_001", "prompt": "A cat walks slowly across the lunar surface, kicking up fine dust, static camera"}
{"id": "t2v_002", "prompt": "Slow push-in on a rainy cyberpunk street at night, neon signs flickering in the reflections"}Distillation configuration
The distillation configuration is provided in YAML format with recommended defaults from the algorithm team. Adjust the parameters as needed. The following table describes key parameters.
Section | Description |
backend | Request parameters for the instruction builder model.
|
t2v_backend | Request parameters for the text-to-video model.
|
generation |
|
eval | Video scoring uses a VLM (vision language model) to score by frame sampling.
|
pipeline |
|