Platform For AI:Fine-tuning large language models

Fine-tuning methods

SFT/DPO

In Model Gallery, you can fine-tune models using supervised fine-tuning (SFT) and direct preference optimization (DPO). The training process for a large language model (LLM) typically includes the following three stages:

1. Pre-training (PT)

Pre-training is the first stage in the LLM training process, where the model learns fundamental grammar, logical reasoning, and common-sense knowledge from a massive text corpus.

• Goal: To equip the model with language comprehension, logical reasoning, and common-sense knowledge.

• Example models: Model Gallery provides many pre-trained models, such as the Qwen and Llama series.

2. Supervised fine-tuning (SFT)

Supervised fine-tuning further adapts a pre-trained model to excel in specific domains, dialogue, and question-answering scenarios.

• Goal: To improve the model's output format and content.

• Example problem: Enabling the model to provide expert-level answers to questions in a specific domain, such as traditional Chinese medicine.

• Solution: Fine-tune the model using question-and-answer pairs from the specific domain, such as traditional Chinese medicine.

3. Preference optimization (PO)

After supervised fine-tuning, a model might produce responses that are grammatically correct but factually inaccurate or not aligned with human values. Preference optimization (PO) is used to further refine the model's dialogue capabilities and align it with human values. The main methods include reinforcement learning-based proximal policy optimization (PPO) and direct preference optimization (DPO), which directly optimizes the language model. DPO uses an implicit reinforcement learning paradigm that does not require an explicit reward model and does not directly maximize a reward signal, making its training process more stable than PPO.

3.1 Direct preference optimization (DPO)

• A DPO model consists of two main components: a large language model (LLM) that is being trained, and a reference model. The reference model, also a fine-tuned LLM, has frozen parameters to prevent the trained model from deviating from its intended behavior.

• Training data format: Triplets consisting of (prompt, chosen (preferred response $$y_{\text{win}}$$), rejected (less-preferred response $$y_{\text{lose}}$$))

• The loss function is shown below. The DPO algorithm aims to increase the probability of the model generating the chosen response relative to the reference model, while decreasing the probability of generating the rejected response.

  $$\text{loss}=\underset{{\pi }_{\theta }}{\max }\left\{{\mathbb{E}}_{(x,y_{\text{win}},y_{\text{lose}})\sim \mathcal{D}}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_{\text{win}}|x)}{{\pi }_{\text{ref}}(y_{\text{win}}|x)}-\beta \log \frac{{\pi }_{\theta }(y_{\text{lose}}|x)}{{\pi }_{\text{ref}}(y_{\text{lose}}|x)}\right)\right]\right\}$$

  • σ is the Sigmoid function, which maps the result to a range of (0, 1).

  • β is a hyperparameter, typically set between 0.1 and 0.5, that adjusts the sensitivity of the loss function.

PEFT: LoRA/QLoRA

Parameter-efficient fine-tuning (PEFT) techniques achieve competitive performance by freezing most of a model's parameters and fine-tuning only a small subset. Because fewer parameters are updated, this also reduces data and computational resource requirements. Model Gallery supports full-parameter fine-tuning as well as two PEFT methods: LoRA and QLoRA.

LoRA (low-rank adaptation)

The LoRA technique adds a parallel path next to a model's parameter matrix (for example, a matrix of size m×n). This path consists of the product of two low-rank matrices (of sizes m×r and r×n, where r is much smaller than m and n). During the forward pass, the input passes through both the original parameter matrix and the LoRA path, and their outputs are summed. During training, the original parameters are frozen, and only the LoRA path is trained. Because the LoRA path is composed of two low-rank matrices, it has far fewer parameters than the original matrix, which significantly reduces the computational overhead of training.

QLoRA (quantized LoRA)

QLoRA combines model quantization with the LoRA technique. In addition to introducing a LoRA path, QLoRA quantizes the large model to 4-bit or 8-bit precision when it is loaded. During computation, these quantized parameters are de-quantized to 16-bit for processing. This method not only optimizes storage for model parameters when they are not in use but also further reduces GPU memory consumption during training compared to LoRA.

Training method selection and data preparation

When choosing between SFT and DPO, consider your specific use case and refer to the technical descriptions in the previous section.

Choosing between full-parameter, LoRA, and QLoRA

• For complex tasks, full-parameter training is recommended as it allows the model to leverage all its parameters, leading to better performance.

• For simpler tasks, we recommend using LoRA or QLoRA, which offer faster training times and require fewer computational resources.

• For small datasets (a few hundred to a few thousand examples), LoRA or QLoRA can help prevent model overfitting.

• If you have limited computational resources, choose QLoRA to further reduce GPU memory consumption. However, be aware that the added quantization and de-quantization steps can increase QLoRA training time compared to LoRA.

Training data preparation

• Simple tasks do not require large amounts of data.

• SFT data requirements: For SFT, a few thousand data points are often sufficient. In this case, data quality is more important than quantity.

Hyperparameters

learning_rate

The learning rate determines the step size for parameter updates during each iteration. A larger learning rate can speed up training but may cause the parameters to be updated too aggressively, preventing convergence to the loss function's minimum. A smaller learning rate provides a more stable convergence process but can increase training time or lead to the model getting stuck in a local optimum. Using the AdamW optimizer can effectively prevent long-term non-convergence issues.

num_train_epochs

An epoch is a fundamental concept in machine learning and deep learning. One epoch represents a single complete pass through the entire training dataset. For example, if you set the number of epochs to 10, the model will iterate over the full dataset 10 times to update its training parameters.

Too few epochs can lead to underfitting, while too many can cause overfitting. We recommend setting this value between 2 and 10. For small datasets, you can increase the number of epochs to avoid underfitting. For large datasets, 2 epochs are often sufficient. Additionally, a smaller learning rate typically requires more epochs. You can monitor the validation set accuracy and stop training when it no longer improves.

per_device_train_batch_size

In practice, model parameters are not updated after processing the entire training dataset at once. Instead, training is broken into multiple iterations using smaller subsets of data (batches) to improve efficiency. The batch size is the number of training samples used in each iteration. For example, if the batch size is set to 32, the model uses 32 training samples in each training step. The per_device_train_batch_size parameter specifies the number of samples processed by each GPU in a single training iteration.

• The batch size parameter primarily adjusts training speed, not training effectiveness. A smaller batch size increases the variance of the gradient estimate, requiring more iterations to converge. Increasing the batch size can reduce the total training time.

• The ideal batch size is typically the maximum value that your hardware can support. You can go to Model Gallery > Job Management > Training Jobs, click a specific job name, and view the GPU Memory Usage Rate and Memory Usage Rate metrics on the Task monitoring page to select the largest batch size that does not cause a GPU memory overflow.

  

• Increasing the number of GPUs while keeping the per_device_train_batch_size constant is equivalent to increasing the total batch size.

seq_length

For an LLM, training data is processed by a tokenizer into a token sequence. The sequence length is the length of the token sequence the model accepts for a single training sample. If a training sample's token sequence exceeds this length, it is truncated; if it is shorter, it is padded.

During training, you can select an appropriate sequence length based on the distribution of token sequence lengths in your training data. For a given text sequence, different tokenizers typically produce token sequences of similar lengths, so you can generally use a tool like the OpenAI tokens online calculation tool to estimate the token sequence length of your text.

For the SFT algorithm, estimate the sequence length of system prompt + instruction + output. For the DPO algorithm, estimate the sequence lengths of both system prompt + prompt + chosen and system prompt + prompt + rejected, and use the larger value.

lora_dim/lora_rank

When applying LoRA in a Transformer, it is primarily applied to the multi-head attention component. Research has shown that:

• Adapting multiple weight matrices within the multi-head attention module yields better results than adapting only a single type of weight matrix.

• Increasing the rank does not necessarily capture a more meaningful subspace; a low-rank adaptation matrix may be sufficient.

In the llm_deepspeed_peft algorithm provided in Model Gallery, LoRA adapts all four types of weight matrices in the multi-head attention module, and the default rank value is 32.

lora_alpha

The LoRA scaling factor. A higher lora_alpha increases the influence of the LoRA matrix and is suitable for smaller training datasets. A lower lora_alpha decreases the influence of the LoRA matrix and is suitable for larger training datasets. The value of lora_alpha is typically between 0.5 and 2 times the value of lora_dim.

dpo_beta

This hyperparameter controls the degree of deviation from the reference model. The default value is 0.1. A higher beta value results in less deviation from the reference model. This parameter is ignored during SFT training.

load_in_4bit/load_in_8bit

Used in QLoRA to load the base model with 4-bit and 8-bit precision, respectively.

gradient_accumulation_steps

A large batch size requires more GPU memory and may cause a CUDA out of memory (OOM) error. Gradient accumulation enables a larger effective batch size (= batch size × gradient_accumulation_steps) by updating the model only after accumulating gradients over several batches. This helps avoid OOM errors while improving convergence speed.

apply_chat_template

When apply_chat_template is set to true, the training data is automatically formatted with the model's default chat template. If you want to use a custom chat template, set apply_chat_template to false and manually insert the required special tokens into your training data. Even if apply_chat_template is true, you can still customize the system prompt.

system_prompt

In the system prompt, you can provide instructions, guidance, and background information to help the model better respond to user inquiries. For example, you could use the following system prompt: "You are an enthusiastic and professional customer service agent. You are friendly, concise, and like to use examples to solve user problems."