Use Quick Start to deploy Llama 2 models - Platform For AI

Background

Llama 2 is an open-source, pre-trained large language model from Meta that is primarily designed for English. It accepts natural language text as input and generates text as output. Llama 2 models range from 7 billion to 70 billion parameters, available in 7B, 13B, and 70B sizes. Each size has a Llama2-chat variant optimized for conversational use cases. Quick Start supports online inference for Llama 2 models and allows you to use them as base models for training to achieve better results in custom scenarios.

The llama-2-7b-chat model in Quick Start is from the Llama-2-7b-chat model on Hugging Face. It is a large language model based on the Transformer architecture and is trained on a mixture of open-source datasets, making it suitable for most general-purpose English scenarios.

This topic demonstrates how to use Quick Start to deploy the llama-2-7b-chat model to Model Online Service (EAS), and then create and call an inference service.

Limitations

Quick Start is currently available in the following regions: China (Beijing), China (Shanghai), China (Hangzhou), China (Shenzhen), and China (Ulanqab).

Note

To enable the China (Ulanqab) region, contact your account manager.

Billing

You are charged for using OSS. For more information, see OSS billing overview.
Quick Start is free, but you are charged for Model Online Service (EAS) and Distributed training (DLC) when deploying and training models. For more information, see Billing of Model Online Service (EAS) and Billing of Distributed training (DLC).

Prerequisites

PAI, including Model Online Service (EAS) and Distributed training (DLC), is activated and a default workspace is created. For more information, see Activate PAI and create a default workspace.
OSS is activated and a bucket is created.

Note
The bucket must be in the same region as the PAI workspace. A bucket's region cannot be changed after creation.
You have read and agree to the Llama model's custom commercial open source license.

Note
If you cannot access the link, configure a proxy and try again.

Procedure

If direct deployment predictions do not meet your requirements, or if you need to apply domain-specific knowledge, you can fine-tune the model. This helps improve its capabilities in your custom domain and better align it with your actual business requirements.

Large language models can learn relatively simple knowledge directly through conversation. You can choose whether to train the model based on your needs. The training method supported in this Quick Start is based on LoRA. Compared to other training methods such as SFT, LoRA training significantly reduces training cost and time.

Deploy the model directly

Go to the Model Gallery page.
1. Log on to the PAI console.
2. In the navigation pane on the left, click Workspaces, then select and enter your target workspace.
3. In the navigation pane on the left, click QuickStart > Model Gallery to open the Model Gallery page.
In the model list's search box, enter llama-2-7b-chat and press Enter.

Note
You can also select other models based on your business needs. A model requires at least 64 GiB of memory and 24 GiB or more of GPU memory. Make sure that your selected compute resources meet these requirements to avoid deployment failure.
Click the llama-2-7b-chat model card to go to the model details page. In the upper-right corner, click Deploy.
At the bottom of the deployment page, click Deploy.
In the Billing reminder dialog box that appears, click OK.
The Service Details page appears. The inference service is deployed when its Status changes to Running.
After the service is deployed, you can call it using a web UI or an API.
1. On the Service Details page, click View Web App in the upper-right corner.
2. Call the inference service.
  - Web UI: On the Chat tab, enter text in the dialog box and click Send to start a conversation.
    
    In the Model params pane on the left, you can configure inference parameters, including Max New Tokens (default: 2048), Top K (default: 0), Top P (default: 0), and Temperature (default: 0). You can also select the Do sample, KV cache, and Stream output options. The default value for Max round is 5.
  - API: At the bottom of the web UI page, click Use via API to view API call details, including the API endpoint, request examples, and code snippets.

Fine-tune the model

Go to the Model Gallery page.
1. Log on to the PAI console.
2. In the navigation pane on the left, click Workspaces, then select and enter your target workspace.
3. In the navigation pane on the left, click QuickStart > Model Gallery to open the Model Gallery page.
In the model list's search box, enter llama-2-7b-chat and press Enter.

Note
You can also select other models based on your business needs. A model requires at least 64 GiB of memory and 24 GiB or more of GPU memory. Make sure that your selected compute resources meet these requirements to avoid deployment failure.
Click the llama-2-7b-chat model card to go to the model details page, and then click Train.

Configure the model training parameters.

Quick Start provides default compute resources and hyperparameters suitable for most use cases. You can modify them as needed.

Parameter

Description

Output configuration

Model output path

Select an OSS bucket path to store model files generated during training.

Note

If you configured a storage path on the workspace details page, this field is automatically populated and does not need to be configured manually. For more information about how to configure a workspace storage path, see Manage workspaces.

Dataset configuration

Training dataset

To help you get started with the Llama 2 model, Quick Start provides a default training dataset that you can use directly. If you do not use the default dataset, you must prepare your training data in the format specified in the model documentation and upload the training data in one of the following ways:

Dataset selection: Use a public or custom dataset. For more information about how to create a custom dataset, see Create and manage datasets.
OSS file or directory: Upload data using OSS. For more information, see Quick start.

Training data must be in JSON format. Each record consists of a question, an answer, and an ID, which are represented by the instruction, output, and id fields. Example:

[
    {
        "instruction": "Does the following text belong to the world topic? Why do Americans rarely hold military parades?",
        "output": "Yes",
        "id": 0
    },
    {
        "instruction": "Does the following text belong to the world topic? Breaking news! The timetable for vehicle reform in public institutions has been released!",
        "output": "No",
        "id": 1
    }
]

To better validate training results, we recommend that you prepare a validation dataset in addition to the training dataset provided by PAI. This dataset is used to evaluate training results and help you optimize and tune training parameters.

Click Train to submit the training job.
In the Billing reminder dialog box that appears, click OK.
The Task details page appears. The model training is complete when the Task Status changes to Succeeded.

The trained model is saved to OSS. You can find its location in the Basic Information section under Output Path.

Note
With custom data and configurations, the training time may vary but typically completes within a few hours.
Deploy the fine-tuned model.

The process for deploying and calling the fine-tuned model is the same as for direct deployment. For more information, see Deploy the model directly.

Next steps

On the Quick Start>Model Gallery page, click Job Management. You can view the details of your training and deployment jobs on the Training Tasks and Deployment Tasks tabs.

The training job list includes the job name/ID, pre-trained model, training method, resource quota, status, creation time, update time, and duration. You can perform operations such as Stop, Clone, and Delete.