Platform For AI:Continued pre-training for LLMs

Prepare data for continued pre-training

Continued pre-training uses the Pai-Megatron-Patch toolkit, which applies Megatron-LM acceleration techniques to train LLMs and vision language models (VLMs).

Pai-Megatron-Patch requires pre-training data in MMAP format, a pre-tokenized format that reduces loading time for large datasets. Convert raw data to MMAP format by using the data conversion tutorial or the "Convert text data to mmap format" component in PAI-Designer. The output files must be named dataset.bin and dataset.idx. PAI provides a sample dataset for trial use:

wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/llama3-datasets/wudao_llama3bpe_content_document.bin
wget https://atp-modelzoo-wlcb-pai.oss-cn-wulanchabu.aliyuncs.com/release/models/pai-megatron-patch/llama3-datasets/wudao_llama3bpe_content_document.idx
mv wudao_llama3bpe_content_document.bin dataset.bin
mv wudao_llama3bpe_content_document.idx dataset.idx

Use PAI-QuickStart for continued pre-training

After preparing data, run continued pre-training in PAI-QuickStart. This procedure uses Qwen2-72B as an example.

1. Go to the Model Gallery page.

   1. Log on to the PAI console.

   2. In the upper-left corner, select a region.

   3. In the left-side navigation pane, choose Workspaces, and click a workspace name.

   4. In the left-side navigation pane, choose QuickStart > Model Gallery.

2. On the Model Gallery page, click Qwen2-72B-Pre-trained (Megatron Edition).

3. On the Model Details page, click Train. Configure the following settings:

   • Training output: Set the output to a NAS dataset (Create dataset). Megatron checkpoints are saved to the checkpoint subfolder in the output directory.

   • Compute resources: At least 32 A100/A800/H100/H800 (80 GB) GPUs across four nodes.

   • Hyperparameters: Adjust based on your dataset and compute resources, or use the defaults.

     Parameter	Default	Type	Description
     job_name	qwen2-72b-cpt	string	Training task type. Do not modify.
     batch_size	1	int	Data samples processed per GPU per iteration.
     global_batch_size	32	int	Total samples across all GPUs per iteration. Equals batch_size * number of GPUs.
     learning_rate	5e-5	float	Learning rate.
     min_learning_rate	5e-6	float	Minimum learning rate.
     sequence_length	1024	int	Text sequence length.
     pad_length	128	int	Padding length for text sequences.
     save_interval	1000	int	Number of training iterations between checkpoints.
     train_tokens	1638400	int	Total tokens for training. Tokens consumed per iteration = global_batch_size * sequence_length.
     warmup_tokens	163840	int	Total tokens for the warmup phase.

4. Click Train. PAI-QuickStart starts training and displays the task status and logs.

(Optional) Convert model checkpoint to Hugging Face format

Qwen2-72B pre-training outputs Megatron Dense Checkpoint format. Convert to Hugging Face format by using Megatron-Core model format conversion.