PAI ModelEval - evaluate and benchmark LLMs - Platform For AI

Use cases

Model benchmarking: Evaluate model capabilities on public datasets and compare results against industry models or baselines.
Domain-specific evaluation: Evaluate model performance in a specific domain by using a custom dataset. Compare the performance difference between a pretrained model and a fine-tuned model.
Regression testing: Build a regression test set to evaluate model performance in production scenarios and determine whether the model meets production standards.
Model comparison: Compare the response quality of two models on the same set of questions through online or offline battles, and select the better-performing model.

Quick start: complete your first evaluation in 5 minutes

This section walks you through a minimal evaluation task: use the public CMMLU dataset to evaluate the Chinese language understanding and reasoning capabilities of the Qwen3-4B model.

Log on to the PAI console. In the left-side navigation pane, choose Model Application > Model Evaluation (ModelEval).
On the Model Evaluation page, click Create Task.
Configure the following parameters:
- Basic Configuration: Use the auto-generated Job Nameand Result Output Path.
  
  Note
  If no default OSS storage path is configured for the workspace, manually specify the output path.
- Evaluation Mode Configuration: Select Single Model Online Evaluation.
- Evaluation Object Configurations: Set Evaluation Target to Public Model. In the Public Model drop-down list, search for and select Qwen3-4B.
- Evaluation Method Configuration: Select Public Dataset Evaluation and select CMMLU from the dataset list.
- Resource Configuration: Select Single-Node-Standard.
  - Resource Type: Select General Computing.
  - Source: Select Public Resources.
  - Job Resource: Select a GPU instance type, for example, ecs.gn7i-c8g1.2xlarge (24 GB).
    
    If this instance type is out of stock, you can select another GPU instance type.
Submit the task: Click OK at the bottom of the page. The page redirects to the task details page.

After the task status changes to Succeeded, go to the Evaluation Report tab to view the performance of the Qwen3-4B model on the CMMLU dataset.

Configuration

Evaluation modes

ModelEval supports the following three evaluation modes.

Evaluation Mode	Description
Single Model Online Evaluation	Evaluates the inference results of a single model.
Dual-Model Arena	Runs online inference tasks to compare the responses of two models on the same questions and selects the better-performing model.
Dual-Model Offline Competition	Uses a pre-generated question-answer dataset to compare the performance of two models on the same questions and selects the better-performing model.

Evaluation targets

ModelEval supports five evaluation target sources. Select the source based on where your model or service is deployed.

Evaluation Target	Description	Scenario
PAI Token Service	Model services provided by PAI Token Service.	Evaluate models deployed in Token Service.
Public Model	Models from the PAI Model Gallery.	Quickly evaluate mainstream open-source models.
Custom Model	Custom models registered in AI Asset Management > Model. Important Make sure the model is compatible with the vLLM framework.	Evaluate fine-tuned or custom models.
PAI-EAS Service	Deployed PAI-EAS online inference services.	Evaluate model services in production.
Custom Services	Any model service that is compatible with the OpenAI API specification.	Evaluate third-party or self-hosted model services.

Important

When the evaluation target is a public model or a custom model, the model must be compatible with the HuggingFace AutoModelForCausalLM architecture.

Evaluation methods

You can evaluate models on custom datasets or public datasets. Custom datasets are suitable for evaluations that closely match your business scenarios. Public datasets are suitable for general-purpose capability assessments against industry baselines.

Custom dataset evaluation

Custom dataset evaluation supports two methods (selectable): General NLP Metric Evaluation and Multi-Metric Evaluation with LLM-as-a-Judge.

Important

The dataset must be in JSONL format, UTF-8 encoded, with one JSON object per line. After uploading to OSS, specify the path on the configuration page.

	General NLP Metric Evaluation	Multi-Metric Evaluation with LLM-as-a-Judge
Scenario	Tasks with definitive answers where text similarity is computed. Suitable for translation, summarization, and knowledge base Q&A.	Tasks without a single correct answer. A judge model scores responses on multiple dimensions such as relevance and accuracy. Suitable for open-ended conversations and content creation.
Data requirements	Must contain the `question` and `answer` (reference answer) fields. `{"question": "What is the capital of China?", "answer": "Beijing"}`	Only the `question` field is required. The `answer` (reference answer) field is optional. `{"question": "Describe the history of artificial intelligence"}`
Key metrics	ROUGE (ROUGE-1, ROUGE-2, ROUGE-L): Recall-based. Measures how much of the reference answer is covered by the model output. BLEU (BLEU-1, BLEU-2, BLEU-3, BLEU-4): Precision-based. Measures how much of the model output is accurate.	Correctness, hallucination, harmfulness, instruction following, and relevance. Custom metrics are supported.

Public dataset evaluation

Evaluate models on authoritative public datasets. ModelEval loads each dataset, runs model predictions, and applies the evaluation framework specific to each dataset to provide industry-standard assessment references.

Note

The GSM8K and HellaSwag datasets are large. Tasks on these datasets take longer to complete.

Dataset	Description	Language
LiveCodeBench	Code processing capability evaluation.	en
Math500	Math reasoning evaluation (500 high-difficulty math competition problems).	en
AIME25	Math reasoning evaluation (based on the 2025 American Invitational Mathematics Examination).	en
AIME24	Math reasoning evaluation (based on the 2024 American Invitational Mathematics Examination).	en
CMMLU	Chinese multi-discipline language understanding evaluation.	zh
MMLU	English multi-discipline language understanding evaluation.	en
C-Eval	Chinese comprehensive capability evaluation.	zh
GSM8K	Math reasoning evaluation.	en
HellaSwag	Commonsense reasoning evaluation (an advanced benchmark developed by the Allen Institute for AI that tests context and logic understanding through sentence completion).	en
TruthfulQA	Truthfulness evaluation.	en

Manage tasks

Stop: Running tasks can be manually stopped. This operation is irreversible. The task cannot be resumed, and consumed compute resources are not refunded.
Delete: Delete a task record. This operation is irreversible.

View evaluation results

Single task results

In the evaluation task list, click View Report in the Actions column of the target task. On the Evaluation Report tab, view the evaluation scores.

Custom dataset evaluation results

General metric evaluation: A radar chart displays the model scores across ROUGE and BLEU metrics (13 metrics in total). For metric definitions, see Appendix: General NLP evaluation metrics.
Judge model evaluation: A table displays the statistical metrics of judge model scores:
- Mean: The average judge model score (excluding invalid scores). Range: 1–5. A higher value indicates better model performance.
- Median: The median judge model score (excluding invalid scores). A higher value indicates better performance.
- Skewness: The distribution skewness of judge model scores (excluding invalid scores). Positive skewness indicates a longer tail on the right (high-score end). Negative skewness indicates a longer tail on the left (low-score end).
- StandardDeviation: The standard deviation of judge model scores (excluding invalid scores). When the mean and median are the same, a smaller standard deviation indicates a better model.
The bottom of the page shows the evaluation details for each data entry, including the Question, Model Response, and Reference Answer columns. You can click Export Report in the upper-right corner to download the evaluation data.

Public dataset evaluation results

If the evaluation task uses public datasets, a radar chart displays the model scores across the selected datasets.

The left chart shows the model scores by domain. Each domain may contain multiple related datasets. The domain score is the average of the dataset scores within that domain.
The right chart shows the model scores on individual public datasets. For the evaluation scope of each dataset, see the official dataset documentation.

Below the radar chart, a results table lists the Dataset Name, Language, and Score. Click Export Report in the upper-right corner of the table to export the evaluation data.

Compare evaluation results

In the evaluation task list, select 2 to 5 successful tasks and click Compare in the upper-right corner.

On the Custom Dataset Evaluation Result tab, a table compares judge model metrics (Mean, Median, StandardDeviation, Skewness) across models, a radar chart compares BLEU and ROUGE scores, and a detailed metrics table is provided.
On the Evaluation Results of Public Datasets tab, radar charts and data tables compare model scores across domains and datasets.

Billing

Compute resources: Evaluation tasks run on DLC compute resources. Both shared resources and resource quotas are supported. For more information, see Billing for DLC.
OSS storage: Evaluation datasets and results are stored in OSS and billed according to OSS pricing. For more information, see OSS billing.
PAI Token Service: Token consumption fees are incurred when the evaluation target is set to PAI Token Service, or when the judge model is a model other than pai-judge/pai-judge-plus. For billing details, see PAI Token Service pricing.

You can also use the model evaluation feature through the PAI Python SDK:

Appendix: General NLP evaluation metrics

Detailed definitions of the general metrics used in custom dataset evaluation.

ROUGE metrics

ROUGE metrics measure the similarity between model predictions and reference answers.

rouge-n metrics compute the overlap of N-grams (sequences of N consecutive words). rouge-1 and rouge-2 correspond to unigram and bigram overlap, respectively.
The rouge-l metric is based on the longest common subsequence (LCS).
Metric suffixes: -p (precision), -r (recall), -f (F-score).

Metric	Description
rouge-1-p	Ratio of unigrams in the model output that match unigrams in the reference.
rouge-1-r	Ratio of unigrams in the reference that appear in the model output.
rouge-1-f	Harmonic mean of rouge-1-p and rouge-1-r.
rouge-2-p	Ratio of bigrams in the model output that match bigrams in the reference.
rouge-2-r	Ratio of bigrams in the reference that appear in the model output.
rouge-2-f	Harmonic mean of rouge-2-p and rouge-2-r.
rouge-l-p	LCS length divided by model output length (precision).
rouge-l-r	LCS length divided by reference length (recall).
rouge-l-f	Harmonic mean of rouge-l-p and rouge-l-r.

BLEU metrics

BLEU (Bilingual Evaluation Understudy) scores model output by measuring N-gram overlap with reference answers.

Metric	Description
bleu-1	Unigram matching.
bleu-2	Bigram matching.
bleu-3	Trigram matching.
bleu-4	4-gram matching.