Deploy fine-tune Mixtral-8x7B MoE model PAI Model Gallery - Platform For AI

Model overview

Mixtral-8x7B is a decoder-only LLM based on a Sparse Mixture-of-Experts (SMoE) architecture, released under the Apache 2.0 license. A router network selects two of eight expert groups per token and combines their outputs. Although the model has 47 billion total parameters, only 13 billion are active per token, yielding inference speed comparable to a 13B model.

Mixtral-8x7B supports English, French, German, Spanish, and Italian with a 32k-token context length. It matches or surpasses LLaMA2-70B and GPT-3.5 on all evaluated benchmarks, and significantly outperforms LLaMA2-70B in mathematics, code generation, and multilingual tasks.

Source: arXiv:2401.04088

Mistral AI also released Mixtral-8x7B-Instruct-v0.1, an instruction-tuned variant optimized with supervised fine-tuning and Direct Preference Optimization (DPO). It follows human instructions more accurately and surpasses other instruction-tuned open-source models in conversational tasks.

Source: Chatbot Arena Leaderboard

Prerequisites

Lingjun Intelligent Computing Service resources are activated. Create a resource group and purchase Lingjun Intelligent Computing Service resources.

Environment requirements

This example requires PAI Lingjun clusters in the China (Ulanqab) region.
GU108 GPUs (80 GB VRAM) are recommended. Inference needs at least two GPUs; LoRA fine-tuning needs at least four.

Use the model in the PAI console

Deploy and call the model

Go to the Model Gallery page.
1. Log in to the PAI console.
2. In the top navigation bar, select the China (Ulanqab) region.
3. In the left-side navigation pane, click workspaces. On the Workspaces page, click the name of the workspace that you want to use.
4. In the left-side navigation pane, choose QuickStart > Model Gallery to go to the Model Gallery page.
In the model list on the right, click the Mixtral-8x7B-Instruct-v0.1 model card to open the Model Details page.
In the upper-right corner, click Deploy. Configure the Lingjun computing resources and click Deploy to deploy the model to PAI-EAS.

This model requires Lingjun Intelligent Computing Service resources. The resource quota must include at least two GU108 GPUs.

In the deployment configuration panel, set Resource group type to Lingjun Intelligent Computing Service, set Number of instances to 1, CPU to 40, and Memory (MB) to 256000. Then, click Deploy.

Call the inference service.

HTTP API

The inference service supports an OpenAI-compatible API. Get the service endpoint and access token from the service details page, then call the service using cURL:

# Replace  and  with the actual endpoint and token of your service.
export API_ENDPOINT="<ENDPOINT>"
export API_TOKEN="<TOKEN>"
# List available models
curl $API_ENDPOINT/v1/models \
	-H "Content-Type: application/json" \
	-H "Authorization: Bearer $API_TOKEN"
# Call the general text completion API
curl $API_ENDPOINT/v1/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $API_TOKEN" \
    -d '{
			"model": "Mixtral-8x7B-Instruct-v0.1",
			"prompt": "San Francisco is a",
			"max_tokens": 256,
			"temperature": 0
	}'
# Call the chat completion API
curl $API_ENDPOINT/v1/chat/completions \
    -H "Authorization: Bearer $API_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
			"model": "Mixtral-8x7B-Instruct-v0.1",
      "messages": [
          {"role": "user", "content": "Tell me about the history of Shanghai."}
        ]
      }'

SDK

To call the service using the OpenAI SDK, first install the SDK:

# Install the SDK to call the service
python -m pip install openai

Call the inference service using the SDK:

import openai
# Replace  and  with the token and endpoint from the service details page.
openai.api_key = "<TOKEN>"
openai.base_url = "<ENDPOINT>" + "/v1"
# Call the chat completion API
completion = openai.chat.completions.create(
    model="Mixtral-8x7B-Instruct-v0.1",
    temperature=0.0,
    top_p=0.8,
    frequency_penalty=0.0,
    messages=[{"role": "user", "content": "Tell me about yourself."}],
    # stream=True,
)
# Print the output content
print(completion.choices[0].message.content)

Fine-tune the model

Model Gallery provides a ready-to-use fine-tuning algorithm for Mixtral-8x7B-Instruct-v0.1.

Training data must be in JSON format. Each entry requires an instruction field (question) and an output field (answer):

[
    {
        "instruction": "Write a poem on a topic 'On a Political Prisoner': \n",
        "output": "She that but little patience knew,\r\nFrom childhood on, had now so much\r\nA grey gull lost its fear and flew\r\nDown to her cell and there alit,\r\nAnd there endured her fingers' touch\r\nAnd from her fingers ate its bit.\r\nDid she in touching that lone wing."
    },
    {
        "instruction": "Write a poem on a topic 'To His Love': ",
        "output": "He's gone, and all our plans\r\n   Are useless indeed.\r\nWe'll walk no more on Cotswold\r\n   Where the sheep feed\r\n   Quietly and take no heed.\r\n His body that was so quick\r\n   Is not as you\r\nKnew it, on Severn river\r\n   Under the blue\r\n   Driving our small boat through."
    }
]

On the Model Details page, click Fine-tune in the upper-right corner. The key configurations are as follows:

Dataset configuration: Upload data to an OSS bucket, or specify a NAS or CPFS dataset. PAI also provides public datasets for testing.
Compute resource configuration: Four GU108 GPUs (80 GB VRAM) are required. Ensure sufficient quota.

Hyperparameter configuration: Adjust the following hyperparameters based on your data and compute resources, or use the defaults.

Parameter	Type	Default	Required	Description
learning_rate	float	5e-5	Yes	Controls the step size of weight updates during training.
num_train_epochs	int	1	Yes	The number of times the training dataset is iterated over.
per_device_train_batch_size	int	1	Yes	Samples processed per GPU per iteration. Larger values improve throughput but increase VRAM usage.
seq_length	int	128	Yes	Input sequence length per training step.
lora_dim	int	16	No	The LoRA dimension. If lora_dim > 0, this enables lightweight fine-tuning with LoRA or QLoRA.
lora_alpha	int	32	No	LoRA alpha scaling factor. Used when lora_dim > 0.
load_in_4bit	bool	true	No	Specifies whether to load the model in 4-bit. If lora_dim > 0, load_in_4bit is true, and load_in_8bit is false, lightweight fine-tuning uses 4-bit QLoRA.
load_in_8bit	bool	false	No	Specifies whether to load the model in 8-bit. If lora_dim > 0, load_in_4bit is false, and load_in_8bit is true, lightweight fine-tuning uses 8-bit QLoRA.
gradient_accumulation_steps	int	8	No	The number of gradient accumulation steps.
apply_chat_template	bool	true	No	Specifies whether the algorithm applies the model's default chat template to the training data. Example: Question: `<\|begin_of_text\|><\|start_header_id\|>user<\|end_header_id\|>\n\n + instruction + <\|eot_id\|>` Answer: `<\|start_header_id\|>assistant<\|end_header_id\|>\n\n + output + <\|eot_id\|>`

Clicking Fine-tune opens the training job page. The job starts automatically.

In the upper-right corner, click Tensorboard to open TensorBoard and monitor the model's convergence.

The trained model is automatically registered in AI Assets - Model Management. Register and manage models.

Use the model with the PAI SDK

You can also use Model Gallery models with the PAI SDK for Python. Install and configure the SDK:

# Install the PAI SDK for Python
python -m pip install alipai --upgrade
# Interactively configure your access credentials, PAI workspace, and other information
python -m pai.toolkit.config

Configure access credentials (AccessKey pair) and workspace as described in Installation and configuration.

Deploy and call the model

Deploy the Mixtral model to PAI-EAS using pre-configured inference settings from Model Gallery. Provide the service name and resource details:

from pai.session import get_default_session
from pai.model import RegisteredModel
from pai.common.utils import random_str
from pai.predictor import Predictor
session = get_default_session()
# Get the model provided by PAI QuickStart
m = RegisteredModel(
    model_name="Mixtral-8x7B-Instruct-v0.1",
    model_provider="pai",
)
# View the default deployment configuration of the model
print(m.inference_spec)
# Deploy the inference service
# You must provide the ID of a Lingjun resource quota (QuotaId) that has at least two GU108 (80 GB of VRAM) GPU cards.
predictor = m.deploy(
    service_name="mixtral_8_7b_{}".format(random_str(6)),
    options={
        # Resource quota ID
        "metadata.quota_id": "<LingJunResourceQuotaId>",
        "metadata.quota_type": "Lingjun",
        "metadata.workspace_id": session.workspace_id,
    }
)
# Get the endpoint and token of the inference service
endpoint = predictor.internet_endpoint
token = predictor.access_token

Call the inference service through the Call the inference service methods above, or use the PAI SDK directly:

from pai.predictor import Predictor
p = Predictor("<MixtralServiceName>")
res = p.raw_predict(
    path="/v1/chat/completions",
    method="POST",
    data={
        "model": "Mixtral-8x7B-Instruct-v0.1",
        "messages": [
            {"role": "user", "content": "Tell me about the history of Shanghai."}
        ]
    }
)
print(res.json())

After testing, delete the service from the console or by using the SDK to release its resources.

# Delete the service
predictor.delete_service()

Fine-tune the model

Retrieve a model from Model Gallery to view its fine-tuning algorithm, supported hyperparameters, and input/output configuration:

from pai.model import RegisteredModel
# Get the Mixtral-8x7B-Instruct-v0.1 model provided by PAI QuickStart
m = RegisteredModel(
    model_name="Mixtral-8x7B-Instruct-v0.1",
    model_provider="pai",
)
# Get the fine-tuning algorithm configured for the model
est = m.get_estimator()
# View the supported hyperparameters and the input/output information of the algorithm
print(est.hyperparameter_definitions)
print(est.input_channel_definitions)

The Mixtral-8x7B-Instruct-v0.1 fine-tuning algorithm currently supports only Lingjun Intelligent Computing Service resources. Obtain your resource quota ID from the PAI console and configure hyperparameters before submitting the job:

# Configure the Lingjun resource quota ID for the training job
est.resource_id = "<LingjunResourceQuotaId>"
# Configure the hyperparameters for the training job
hps = {
    "learning_rate": 1e-5,
    "per_device_train_batch_size": 2,
}
est.set_hyperparameters(**hps)

The fine-tuning algorithm supports three inputs:

model: The pre-trained Mixtral-8x7B-Instruct-v0.1 model.
train: The training dataset for fine-tuning.
validation: The validation dataset for fine-tuning.

The required dataset format is described in the Fine-tune the model section. Upload data to an OSS bucket using ossutil, the console, or the SDK:

from pai.common.oss_utils import upload
# View the inputs used by the model's fine-tuning algorithm
# Get the input data for the algorithm, including the model and a public dataset for testing.
training_inputs = m.get_estimator_inputs()
print(training_inputs)
# {
#     "model": "oss://pai-quickstart-cn-wulanchabu.oss-cn-wulanchabu-internal.aliyuncs.com/huggingface/models/Mixtral-8x7B-Instruct-v0.1/main/",
#     "train": "oss://pai-quickstart-cn-wulanchabu.oss-cn-wulanchabu-internal.aliyuncs.com/huggingface/datasets/llm_instruct/en_poetry_train_mixtral.json",
#     "validation": "oss://pai-quickstart-cn-wulanchabu.oss-cn-wulanchabu-internal.aliyuncs.com/huggingface/datasets/llm_instruct/en_poetry_test_mixtral.json",
# }
# Upload user data. Replace the following local file path and the destination OSS bucket path.
train_data_uri = upload("/path/to/local/train.json", "path/of/train/data")
validation_data_uri = upload("/path/to/local/validation.json", "path/of/validation/data")
# Replace the training data with your own.
# training_inputs["train"] = train_data_uri
# training_inputs["validation"] = validation_data_uri

Replace the train and validation inputs with your own datasets, then submit the fine-tuning job. Use the printed job link to view status and logs in the PAI console, or open TensorBoard to monitor convergence:

from pai.common.oss_utils import download
# Submit the training job and print the link to the job
est.fit(
    inputs=training_inputs,
    wait=False,
)
# Open TensorBoard to view training progress
est.tensorboard()
# Wait for the training job to complete
est.wait()
# View the model path on the OSS bucket
print(est.model_data())
# You can download the model to your local machine by using ossutil or the utility methods provided by the SDK.
download(est.model_data())

Full SDK integration guide: Use pre-trained models — PAI SDK for Python.

Platform For AI:Quick start: Deploy and fine-tune Mixtral-8x7B

Model overview

Prerequisites

Environment requirements

Use the model in the PAI console

Deploy and call the model

HTTP API

SDK

Fine-tune the model

Use the model with the PAI SDK

Deploy and call the model

Fine-tune the model

References