Model evaluation - Alibaba Cloud Model Studio - Alibaba Cloud Documentation Center

Evaluating models across multiple dimensions using datasets helps you fully understand their performance and ensures they deliver effective results in real-world applications.

Note

Getting Started

What is model evaluation

Model evaluation is a model quality validation tool provided by the Model Studio platform. It helps you objectively and quantitatively assess large language model performance.

One-sentence summary: Give the model a test, score responses automatically or manually, then generate an evaluation report.

Why perform model evaluation

Scenario 1: Model selection decisions

Problem: Facing dozens of large models (Qwen, GPT, Claude, ERNIE Bot, etc.), you don't know which best fits your business needs.

Solution:

Prepare 100 test cases from your business scenarios.
Evaluate all candidate models using the same dataset.
Compare evaluation reports (scores, pass rates, sample performance).
Select the model that best meets your business requirements.

Value:

Avoid subjective judgments—let data drive decisions.
Save time compared to manual testing of each model.
Reduce risks of poor model selection.

Scenario 2: Validate fine-tuning effectiveness

Problem: You fine-tuned Qwen with 1,000 training samples but aren't sure if it actually improved performance.

Solution:

Prepare an evaluation dataset (non-overlapping with the training set).
Evaluate both pre-fine-tuned and post-fine-tuned models.
Compare the results of two evaluations:
- Pre-fine-tuning score: 75
- Post-fine-tuning score: 85
- Conclusion: Fine-tuning worked—improved by 10 points.

Value:

Quantify fine-tuning impact to validate ROI.
Verify if fine-tuning direction was correct.
Provide data support for continuous optimization.

Scenario 3: Quantify model capabilities

Problem: You need to report model performance on specific tasks to your team or management but lack objective data.

Solution:

Use evaluation features to generate a report.
The report includes:
- Overall score: 85/100
- Pass rate: 90% (samples scoring ≥3 points)
- Score distribution: 30 samples at 5 points, 40 at 4 points, 20 at 3 points, 10 at 2 points
- Typical cases: high-scoring and low-scoring samples
Use report data to support decisions and presentations.

Value:

Replace subjective descriptions ("pretty good") with data ("85 points, 90% pass rate").
Enable clear cross-team communication and alignment.
Establish baseline references for future optimization.

Scenario 4: Continuously monitor model performance

Problem: After deployment, model performance may degrade over time or with business changes, but this is hard to detect promptly.

Solution:

Establish regular evaluation cycles (e.g., monthly).
Use identical evaluation datasets and dimensions.
Track performance trends:
- January 2024: Score 85
- February 2024: Score 87 (improved)
- March 2024: Score 78 (declined—requires investigation)

Value:

Detect performance degradation early.
Provide data foundation for model iteration.
Build model capability profiles.

Core value summary

Value	Description
Objective quantification	Replace subjective judgments with data—avoid vague descriptions like "seems okay"
Efficient comparison	Quickly compare multiple models—save manual testing time
Risk governance	Thoroughly validate before deployment—reduce model selection risks
Continuous optimization	Provide quantitative feedback for model fine-tuning and iteration
Decision support	Provide data support for team collaboration and management reporting

Evaluation workflow

Phase	Sub-step description
1. Preparations	Evaluation dimensions (define scoring criteria) → Test data (prepare questions and answers)
2. Create evaluation task	Select model → Select data → Select dimensions → Choose whether to join leaderboard
3. Execute evaluation	Automatic evaluation (system completes automatically) or manual evaluation (requires manual annotation)
4. View results	Score report → Pass rate statistics → Detailed breakdown

Quick experience

Evaluation scenario: Evaluate Qwen-Max accuracy on common-sense question answering.

Complete example: Use 10 test questions to assess semantic similarity of model responses.

Experience steps:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Common-sense QA accuracy
Description	Evaluate model accuracy in answering common-sense questions
Type	Select Model Evaluation - Numeric
Judge Model	Select Qwen-Max
Scorer Template	Select Semantic Similarity
Scoring Range	Keep default 0~5
Pass Threshold	3.0

Related notes:

For Scorer Template selection, the interface displays three radio buttons—select one. For more information, see Prompt variable documentation and System Prompt differences.

Template	Description	Applicable scenarios	Need custom Prompt?
Comprehensive Evaluation (default)	A system preset template that scores across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement.	General conversations and customer service scenarios	No—use directly or modify as needed
Semantic Similarity	A system preset template that evaluates the semantic closeness between the output and the reference answer.	QA systems and knowledge retrieval	No—use directly or modify as needed
Custom Evaluation	Write your own prompt.	Specific business scenarios	Yes—write according to your requirements

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	QA test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_QA Test Data.xlsx (CMS attachment, file_id=151014, download online).

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	QA accuracy test
Evaluation Object	Select Qwen-Plus.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Semantic Similarity, and choose the previously created “QA test data” dataset.
Scoring Range	Select the previously created “Common-sense QA accuracy.”
Pass Threshold	This test won't participate.

4. Check task status

Find the "QA accuracy test" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Corresponds to the evaluation dimension name—for example, “Common-sense QA accuracy”—and displays scoring results.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	100
Pass Rate	Percentage of samples meeting the threshold (≥3 points)	90%
Total Evaluation Sets	Total number of test data items	10 items
Completed	Number of completed evaluations	10 items
Remaining	Number of incomplete evaluations	0 items

Core concepts

Custom evaluation overview

Model Studio provides custom evaluation features that let you assess model quality across multiple dimensions using your own evaluation standards and test data. Custom evaluation supports five evaluation dimension types—see the Custom evaluation section below for details.

Custom evaluation

Supported models

Supports the following preset models (including fine-tuned versions):

Qwen-QwQ/Max/Plus/Turbo/Coder/Math
Qwen open-source versions (Qwen3, Qwen2.5, Qwen2, Qwen1.5)
Tongyi Farui
Third-party text generation models (abab6.5g, abab6.5t, etc.)

The list above is incomplete and subject to change. Always refer to the Create Evaluation Task page for the current list of Evaluate Model. Models shown in gray without an Apply button are currently not supported for evaluation.

Evaluation dimensions

Evaluation dimensions define scoring criteria—how you score models. You must create evaluation dimensions before custom evaluation.

Evaluation dimension types (five types)

Type	Scoring Method	Applicable Scenarios	Cost	Recommendation
Model Evaluation - Numeric	AI judge scores (1–5 points)	QA quality, content generation quality	Medium	⭐⭐⭐⭐⭐
Rule Evaluation - Text Similarity	Algorithm calculates similarity	Translation, summarization, rewriting	Low	⭐⭐⭐⭐
Model Evaluation - Classification	AI judge classifies (Pass/Fail)	Content moderation, correctness verification	Medium	⭐⭐⭐
Rule Evaluation - String Matching	Exact string matching	Function Calling, NL2SQL	Very low	⭐⭐⭐
Manual Evaluation - Classification	Manual labeling	Creativity, professional judgment	High	⭐⭐

How to quickly select evaluation dimension types?

One evaluation task can combine multiple evaluation dimensions.

Evaluation data

Data source types

Data type	Description	Workflow	Applicable scenarios
Evaluation dataset	Contains questions (prompts) and reference answers (completions)	The system calls the model for real-time inference, generates output, and then the scorer assigns a score.	Initial evaluation Compare multiple models The model has not yet generated results
Inference result set	Already contains model output	The system directly reads the output and the scorer assigns a score (no inference required).	The model has already generated results Reduce inference costs Baseline model for control groups

Data format requirements: Excel

Evaluation dataset example:

Prompt	Completion
What is cloud computing?	Cloud computing is an on-demand pool of shared computing resources.
How does Python define functions?	You can use the def keyword to define functions.

Inference result set example:

Add one more Output column.

Prompt	Completion	Output
What is cloud computing?	Cloud computing is an on-demand pool of shared computing resources.	Cloud computing is a service model that delivers computing resources over the internet...

Data volume recommendations:

Phase	Recommended Quantity	Description
Small-scale validation	50–100 items	Verify configuration correctness.
Formal evaluation	200–500 items	Obtain reliable statistical results.
Comprehensive assessment	500+ items	Cover various edge cases.

Leaderboard

What is a leaderboard:

Compare multiple models' performance on the same evaluation dimension.
Visually display model ranking.
Support horizontal comparison across multiple evaluation tasks.

When do you need a leaderboard:

Compare multiple models' performance (e.g., GPT-4 vs Claude vs Qwen).
Visually demonstrate model strengths/weaknesses in business scenarios.
Share model comparison results within your team.

Core variables

Applicable scope: This section applies only to Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions.

During large model evaluation, the judge model receives these three variables:

Variable name	Description	Example	Usage scenario
`${prompt}`	Original input: the user's question or instruction	What is cloud computing?	Required in all scenarios
`${output}`	Model output: the evaluated model's response	Cloud computing is a service delivered over...	Required in all scenarios
`${completion}`	Reference answer: the standard answer or expected output	Cloud computing is an on-demand...	Use when standard answers exist

How to use in Prompt:

Please evaluate the quality of the following response:

**User question**: ${prompt}
**Model response**: ${output}
**Reference answer**: ${completion}

Scoring criteria:
- 5 points: Response is completely correct and semantically matches reference answer
- 4 points: Response is mostly correct with minor deviations
- 3 points: Response is partially correct
- 2 points: Response is mostly incorrect
- 1 point: Response is completely wrong or irrelevant

System Prompt vs. Scorer Prompt differences

In model evaluation, two easily confused Prompt configurations serve different purposes:

Prompt type	Configuration location	Target	Action Phase	Required?	Cost attribution
System Prompt	When creating Evaluation Task	Evaluated model	When the model generates answers	No	Evaluated model inference cost
Scorer Prompt	When creating Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions	Judge model	When the judge model scores answers	Yes	Judge model scoring cost

Detailed explanation

System Prompt
- Purpose: Define role or behavior guidelines for the evaluated model.
- Example scenarios:
  - Evaluating customer service: "You are a professional customer service representative. Answer customer questions politely and accurately."
  - Evaluating code assistant: "You are a programming expert. Provide accurate, concise code examples."
- When to use: Only fill when you need to assign a specific identity or constraints to the evaluated model—leave blank in most cases.
Scorer Prompt
- Purpose: Tell the judge model how to evaluate answer quality and what scoring criteria to use.
- Applicable scope: Required only for Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions.
- Configuration methods:
  - Select system preset templates (Comprehensive Evaluation, Semantic Similarity, Sentiment Analysis, etc.).
  - Write custom prompts (for specific business scenarios).
- Scorer Prompts can use ${prompt}, ${output}, ${completion} variables.

Vivid analogy

System Prompt: Tells the "student" (evaluated model) what role to assume when taking the exam.
Scorer Prompt: Tells the "grader" (judge model) how to grade the exam and what scoring standards to apply.

Scenario-based practice cases

Custom Evaluation:Model Evaluation - Numeric evaluation dimension

Applicable scenarios: Situations requiring granular scoring, such as QA quality, content generation quality, or customer service dialogue quality.

Recommendations:

Make scoring criteria specific—provide clear descriptions for each score level.
Regularly spot-check high-scoring and low-scoring samples to verify accuracy.
Avoid having all samples clustered in one score range.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Common-sense QA accuracy
Description	Evaluate model accuracy in answering common-sense questions
Type	Select Model Evaluation - Numeric
Judge Model	Select Qwen-Max
Scorer Template	Select Semantic Similarity
Scoring Range	Keep default 0~5
Pass Threshold	3.0

Related notes:

For Scorer Template selection, the interface displays three radio buttons—select one. For more information, see Prompt variable documentation and System Prompt differences.

Template	Description	Applicable scenarios	Need custom Prompt?
Comprehensive Evaluation (default)	A system preset template that scores across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement.	General conversations and customer service scenarios	No—use directly or modify as needed
Semantic Similarity	A system preset template that evaluates the semantic closeness between the output and the reference answer.	QA systems and knowledge retrieval	No—use directly or modify as needed
Custom Evaluation	Write your own prompt.	Specific business scenarios	Yes—write according to your requirements

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	QA test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_QA Test Data.xlsx (CMS attachment, file_id=151014, download online).

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	QA accuracy test
Evaluation Object	Select Qwen-Plus.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Semantic Similarity, and choose the previously created “QA test data” dataset.
Scoring Range	Select the previously created “Common-sense QA accuracy.”
Pass Threshold	This test won't participate.

4. Check task status

Find the "QA accuracy test" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Corresponds to the evaluation dimension name—for example, “Common-sense QA accuracy”—and displays scoring results.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	100
Pass Rate	Percentage of samples meeting the threshold (≥3 points)	90%
Total Evaluation Sets	Total number of test data items	10 items
Completed	Number of completed evaluations	10 items
Remaining	Number of incomplete evaluations	0 items

Custom evaluation: Large Model Evaluation - Classification evaluation dimension

Applicable scenarios: Situations requiring binary judgments like "yes/no" or "pass/fail", such as content moderation or answer correctness verification.

Recommendations:

Clearly specify judgment conditions for each label in your Prompt.
Explicitly require the judge model to output only label names.
Regularly spot-check samples to ensure accuracy.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Sentiment analysis
Description	Detect customer sentiment
Type	Select Auto
Judge Model	Select Qwen-Max
Scorer Template	Select Sentiment Analysis
Scoring Range	No need to modify—use default content

Related notes:

For Scorer Template selection, the interface displays three radio buttons—select one. For more information, see Prompt variable documentation and System Prompt differences.

Template	Description	Applicable scenarios	Need custom prompt?
Standard Match (default)	A system preset template that checks whether the model output matches the reference answer.	Classification labels, function names, and fixed answers	No—use directly or modify as needed.
Sentiment Analysis	A system preset template that analyzes text sentiment (positive, neutral, or negative).	User reviews, customer service dialogues, and public opinion monitoring	No—use directly or modify as needed.
Custom Scorer	Write your own prompt.	Custom scenarios such as style judgment, compliance checks, and logic verification	Required. Write as needed.

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	Sentiment analysis test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_Sentiment Analysis.xlsx (ICMS attachment, file_id=151061, download online).

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	Sentiment analysis test
Evaluation Dimension	Select Sentiment Analysis.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Semantic Similarity, and choose the previously created "Sentiment analysis test data" dataset.
Scoring Range	Select the previously created "Sentiment Analysis".
Pass Threshold	You are not participating in this test.

4. Check task status

Find the "Sentiment analysis test" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Corresponds to the evaluation dimension name—for example, "Sentiment Analysis"—and displays the analysis results.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	40
Pass Rate	Percentage of passed samples	40%
Score Details - Data Item Distribution	Distribution of classification evaluation results
Total Evaluation Sets	Total number of test data items	10 items
Completed	Number of completed evaluations	10 items
Remaining	Number of incomplete evaluations	0 items

Custom evaluation: Large Model Evaluation - Numerical evaluation dimension

Applicable scenarios: Situations requiring multi-dimensional comprehensive scoring, such as answer quality assessment, content safety scoring, or translation quality scoring.

Recommendations:

Set appropriate score ranges and pass thresholds—test small-scale first before batch evaluation.
Choose suitable scorer templates: Comprehensive Evaluation for general scenarios, Semantic Similarity for scenarios with reference answers.
When writing custom scorer Prompts, clearly define scoring criteria and value ranges.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Comprehensive quality assessment
Description	Evaluate comprehensive quality of model responses
Type	Select Model Evaluation - Numeric
Judge Model	Select Qwen-Max
Scorer Template	Select Comprehensive Evaluation (default)
Prompt	No need to modify—use default content
Scoring Range	Use default values 0-5
Pass Threshold	Use default value 3.0—samples scoring ≥3.0 are marked Pass

Related notes:

For Scorer Template selection, the interface displays three radio buttons—select one. For more information, see Prompt variable documentation and System Prompt differences.

Template	Description	Applicable scenarios	Need custom Prompt?
Comprehensive Evaluation (default)	A system preset template that scores responses across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement.	General answer quality assessment	No—use directly or modify as needed
Semantic Similarity	A system preset template that evaluates the semantic similarity between the model output and a reference answer.	Scenarios with clear reference answers, such as knowledge-based QA or translation	No—use directly or modify as needed
Custom Evaluation	Write your own prompt—specify scoring criteria and value ranges in your prompt.	Custom scenarios, such as professional domain assessment or scoring based on specific metrics	Required. Write as needed.

About score range and pass threshold:

Score range: Set minimum and maximum integer values for judge model scoring (default 0–5). The judge model will score each evaluation item within this range.
Pass threshold: Scores ≥ threshold are marked Pass, scores below are marked Fail. Threshold range is constrained by score range and supports decimals (step size 0.1), default is 3.0.

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	Comprehensive quality assessment test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload a test data file in JSONL format, with each line containing prompt and completion fields.

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	Comprehensive quality assessment test
Evaluate Model	Select Qwen-Max.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Comprehensive Evaluation and choose the previously created “Comprehensive quality assessment test data” dataset.
Scoring Range	Select the previously created “Comprehensive quality assessment.”
Pass Threshold	You are not participating in this test.

4. Check task status

Find the "Comprehensive quality assessment test" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Matches the evaluation dimension name—for example, “Comprehensive Quality Assessment”—and shows the judge model’s numerical score.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions (for numerical dimensions, this is the arithmetic mean of all sample scores)	3.8
Pass Rate	Percentage of samples with scores greater than or equal to the pass threshold	70%
Total Evaluation Sets	Total number of test data items	10 items
Completed	Number of completed evaluations	10 items
Remaining	Number of incomplete evaluations	0 items

Custom evaluation: Rule-based Evaluation - String Match evaluation dimension

Applicable scenarios: Fixed answer formats requiring exact matching, such as Function Calling or NL2SQL.

Recommendations:

Normalize preprocessing: standardize case, trim whitespace.
For synonyms, consider using text similarity scorers instead.
Test matching rules with small samples first to verify expectations.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Function Calling validation
Description	Check if model correctly invoked specified function
Type	Select Rule Evaluation - String Matching
Compare Text	Three-column row: left text box enter `${output}` (model output), middle operator dropdown select Include, right text box enter `${completion}` (reference answer). Type `/` + Enter in text boxes to quickly insert `${prompt}` / `${output}` / `${completion}` variables. Enter model output variable in left text box, reference answer variable in right text box

How to choose matching rules:

Rule	Behavior Description	Typical Application
Equal	Exactly identical (case-insensitive)	City name validation, fixed answers
Not Equal	Completely different (case-insensitive)	Prohibited word screening
Contains	Contains the specified string	Keyword checking, essential information validation
Starts With	Starts with the specified string	Prefix validation, format checking
Ends With	Ends with the specified string	Suffix validation, terminator checking

Evaluation data example:

{"prompt": "What's the weather like in Beijing today?","completion": "query_weather"}

If matching rule is "Contains", when model output contains query_weather, it's marked Pass.

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	Function Calling test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_Function_Calling.xlsx (ICMS attachment, file_id=151064, download online).

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	Function Calling test
Evaluation Set	Select Function Calling test data.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Comprehensive Evaluation and choose the previously created "Function Calling test data" dataset.
Scoring Range	Select the previously created "Function Calling validation".
Pass Threshold	You are not participating in this test.

4. Check task status

Find the "Function Calling test" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Matches the evaluation dimension name—for example, “Function Calling validation”—and shows analysis results.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	100
Pass Rate	Percentage of passed samples	100%
Total Evaluation Sets	Total number of test data items	2 items
Completed	Number of completed evaluations	2 items
Remaining	Number of incomplete evaluations	0 items

Custom evaluation: Rule-based Evaluation - Text Similarity evaluation dimension

Applicable scenarios: Situations where answers vary in wording but share the same meaning, such as translation, summarization, or rewriting.

Recommendations:

Test with small samples first—observe pass rate distribution before adjusting thresholds.
Focus on word order and exact matching → choose BLEU.
Focus on semantic understanding → choose Cosine.
Need error tolerance → choose Fuzzy Match.
Focus on key information coverage → choose ROUGE-L.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Translation quality
Description	Evaluate Chinese-to-English translation quality
Type	Select Rule Evaluation - Text Similarity
Similarity algorithm	Select `BLEU_4`
Scorer Template	Left: `${output}` Right: `${completion}` Enter model output variable in left text box, reference answer variable in right text box
Pass Threshold	Set to 0.4—samples with similarity ≥0.4 are marked Pass (range 0~1)

How to choose similarity algorithms:

Algorithm	Characteristics	Best scenarios	Threshold recommendation
ROUGE-L	Based on the longest common subsequence	Text summarization	0.4–0.6
BLEU	Based on n-gram precision matching and sensitive to word order	Machine translation	0.3–0.5
Cosine	Measures vector space similarity and captures semantics	Open QA and semantic understanding	0.6–0.8
Fuzzy Match	Based on edit distance and offers high error tolerance	OCR and speech recognition	0.7–0.9
Accuracy	Requires an exact match	Precise answer verification	1.0

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	Chinese-to-English test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_Chinese-to-English.xlsx (ICMS attachment, file_id=151065, download online).

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	Translation quality evaluation
Evaluate Model	Select Qwen-Max.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Comprehensive Evaluation and choose the previously created "Chinese-to-English test data" dataset.
Scoring Range	Select the previously created "Translation quality".
Pass Threshold	Not included in this test

4. Check task status

Find the "Translation quality evaluation" task and check its Evaluation Status column:

Pending: During peak request periods, evaluation tasks queue for execution.
In Progress: System is evaluating—please wait patiently.
Completed: Evaluation finished—you can view results.
Evaluation Failed: Evaluation task execution failed.
Stop Evaluation: Evaluation task manually stopped by user.

Click the refresh button () in the top-right corner. When the task status changes to Completed, click the Task Name column to enter the details page.

Select the Data Details tab to view detailed scoring for each data point.

Column name	Description
Status	Evaluation status: Pending, In Progress, Evaluation Complete, Evaluation Failed, Evaluation Terminated
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	This column displays the analysis results for the "Translation Quality" evaluation dimension.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	100
Pass Rate	Percentage of passed samples	100%
Total Evaluation Sets	Total number of test data items	3 items
Completed	Number of completed evaluations	3 items
Remaining	Number of incomplete evaluations	0 items

Custom evaluation: Manual Evaluation - Classification evaluation dimension

Applicable scenarios: Situations requiring human subjective judgment, such as creativity assessment or professional quality review.

Recommendations:

Use manual evaluation when model output quality can't be precisely judged by algorithms or judge models alone.
Document clear judgment criteria for each label in an external Evaluation Guide and train annotators for consistent understanding.
Design labels to cover all possible output scenarios using concise, clear classification terms.
Have multiple evaluators cross-annotate some samples to verify consistency.
Avoid annotating too much data at once—process in batches to reduce fatigue effects.

Procedure:

1. Create evaluation dimension

Interface location: Go to the Model Evaluation page, select the Evaluation Dimension tab, then click the Create Evaluation Dimension button in the top-right corner.

Fill out the form and click the Save button.

Item	Content
Dimension Name	Content creativity
Description	Evaluates content creativity.
Type	Select Manual Evaluation - Classification.
Pass	Create two labels: "Excellent" and "Good."
Fail	Create two labels: "Average" and "Poor."

2. Prepare test data

Interface location: Go to the Data Management page, then click the Add Dataset button in the top-right corner.

Fill out the form and click the Confirm button.

Item	Content
Dimension Name	Content creativity test data
Description	Select Evaluation Set type. ❗ Model evaluation only supports datasets of the Evaluation Set type—not training sets.
Type	Upload Sample Data_Content Creativity.xlsx (ICMS attachment, file_id=151066, download online). Leave Completion blank—let the model generate responses.

3. Create evaluation task

Interface location: Go to the Model Evaluation page, select the Evaluation Task tab, then click the Create Evaluation Task button in the top-right corner.

Fill out the form and click the Evaluate button.

Item	Content
Dimension Name	Content creativity evaluation
Evaluate Model	Select Qwen-Plus.
Judge Model	Select Qwen-Max.
System Prompt	Leave blank—this test doesn't require it. For more information, see System Prompt vs. Scorer Prompt differences.
Scorer Template	Select Custom Evaluation and choose the previously created "Content creativity test data" dataset.
Scoring Range	Select the previously created "Content creativity".
Pass Threshold	This is not included in the test.

4. Check task status

Find the "Content creativity evaluation" task and check its Evaluation Status column—it shows In Progress status. Manual evaluation requires manual annotation—the system won't score automatically. Click the Task Name column to enter the details page.

Select the Data Details tab, then click the Actions column's Annotate button to perform manual annotation. After labeling all data, the task's Evaluation Status will change to Completed.

Column name	Description
Status	Evaluation status: In progress
No.	Sample ID
Prompt	Original question
Completion	Reference answer
Output	Model-generated response
[Evaluation dimension name]	Corresponds to the evaluation dimension name—for example, "Content creativity evaluation"—and displays the analysis results.
Actions	Available user actions—for example, click Details to view the full question, answer, and evaluation results.

Select the Metric Statistics tab to view evaluation statistics.

Metric name	Description	Example value
Overall Score	Average score across evaluation dimensions	100
Score Details - Data Item Distribution	Distribution of classification evaluation results
Pass Rate	Percentage of passed samples	100%
Total Evaluation Sets	Total number of test data items	3 items
Completed	Number of completed evaluations	3 items
Remaining	Number of incomplete evaluations	0 items

Compare multiple models using leaderboards

Applicable scenarios: Compare multiple models' performance on the same evaluation dimension and visually see model rankings through leaderboards.

Recommendations:

Compare different specifications within the same model series (e.g., qwen-max, qwen-plus, qwen-turbo).
Compare models from different vendors (e.g., Qwen vs DeepSeek).
Compare different versions or configurations of the same model.

Prerequisites: You've created a custom evaluation dimension.

Procedure:

1. Create leaderboard

Interface location: Go to the Model Evaluation page, select the Leaderboard tab, then click the Create Leaderboard button in the top-right corner.

Fill out the form and click the OK button.

Item	Content
Leaderboard Name	Common-sense QA accuracy leaderboard
Evaluation Dimension	Select an existing evaluation dimension, such as Common-sense QA accuracy.
Associate Optional Tasks	You can select existing evaluation tasks now or add them later.

Notes:

Each leaderboard supports up to 50 evaluation tasks.
After creating a leaderboard, you can't modify its evaluation dimension.
All tasks in the same leaderboard use the same evaluation dimension to ensure fair comparison.

2. Add evaluation tasks to leaderboard

Interface location: Go to the Model Evaluation page, select the Leaderboard tab, then click the leaderboard name to enter its details page.

The leaderboard supports two ways to Add Evaluation Task:

Method 1: Create New Task

In the leaderboard details page, click the Add Evaluation Task button.
Click Create New Task.
The system automatically redirects to the task creation page with leaderboard pre-linked:
- Include in Leaderboard: Automatically checked and locked—can't be unchecked.
- Leaderboard: Automatically filled with current leaderboard name and locked.
- Evaluation Dimension: Automatically filled with leaderboard's linked dimension and locked.
Fill other required fields:
- Task Name: Enter a task name—include model info for easy identification.
- Evaluate Model: Select the model to evaluate.
- Data Source: Select evaluation dataset.
Click the Evaluate button.

Method 2: Select Existing Task

In the leaderboard details page, click the Add Evaluation Task button.
Click Select Existing Task.
Select completed evaluation tasks to add to the leaderboard.
Click the OK button.

Note:

Only completed tasks using the same evaluation dimension can join the leaderboard.
Use the same evaluation dataset for accurate comparison results.

3. View leaderboard results

Interface location: Go to the Model Evaluation page, select the Leaderboard tab, then click the leaderboard name to enter its details page.

The leaderboard details page shows:

Column name	Description
Rank	Automatically calculated ranking based on the leaderboard score—higher scores receive higher rankings.
Task Name	Name of the evaluation task.
Evaluate Model	Model name used in the task.
Leaderboard Score	Comprehensive score for this task on the current evaluation dimension (range: 0–100).
[Evaluation dimension name]	Shows the specific score for this dimension—for example, the "Semantic Similarity" dimension displays the average similarity score.
Actions	Delete task (removes the task from the leaderboard without affecting the task itself).

Notes:

Leaderboard Score updates in real-time as task evaluation progresses.
Tasks in progress show "-" in the score column.
Click Task Name to view detailed evaluation data.

Advanced techniques

Best practices for evaluation dimension design

Single responsibility:

Each evaluation dimension should focus on one evaluation goal.
Avoid mixing multiple standards.
Example: Don't assess both "accuracy" and "fluency" in the same dimension.

Quantifiable standards:

Use objective, quantifiable standards whenever possible.
Reduce subjectivity.
Provide clear criteria for each score level.

Iterative optimization:

Continuously optimize Prompts based on evaluation feedback.
Regularly spot-check to verify evaluation accuracy.
Compare human and AI scoring to adjust evaluation standards.

Prompt writing techniques

See Prompt Best Practices and Plugins for Alibaba Cloud Model Studio to learn how to maximize the model's performance without model fine-tuning.

Cost optimization recommendations

Choose evaluation methods wisely:

Rule-based evaluation has the lowest cost—ideal for formatted output validation.
Large model evaluation costs more—use for scenarios requiring semantic understanding.
Manual evaluation is most expensive—reserve for cases requiring human judgment.

Use inference result sets:

For frequently evaluated baseline models, save outputs as inference result sets.
Avoid repeated model calls to reduce inference costs.

Phased evaluation:

First validate configurations with small datasets (50–100 items).
After confirming correct setup, scale to 200–500 items.

Batch evaluation:

Evaluate multiple models simultaneously to improve efficiency.
Select multiple models in the same evaluation task.

Evaluation result analysis methods

Monitor score distribution:

Check sample counts at each score level.
Identify problematic clustering (e.g., all scores at 3 points).
Adjust scoring standards or thresholds.

Analyze typical samples:

Review high-scoring samples: understand model strengths.
Review low-scoring samples: identify model weaknesses.
Review borderline samples: refine evaluation standards.

Compare different models:

Don't just look at total scores—examine performance across different scenarios.
Identify each model's strengths and weaknesses.
Select appropriate models based on business priorities.

Track model iterations:

Establish regular evaluation cycles.
Record evaluation results after each fine-tuning.
Analyze whether fine-tuning directions are correct.

Troubleshooting common issues

Evaluation results don't meet expectations:

Verify test data represents your business scenarios.
Review if evaluation dimension Prompts are clear.
Try adjusting score ranges and thresholds.
Consider changing judge models or scorer templates.

Scores are overly clustered:

Scoring criteria too broad → refine descriptions for each score level.
Test data lacks diversity → add edge and abnormal samples.
Judge model too conservative → switch to a more capable inference model.

Results conflict across evaluation dimensions:

This is normal—models perform differently across dimensions.
Weight dimensions according to business priorities.
Don't rely on single dimensions—make decisions holistically.

If model outputs frequently show these defects, consider adding a knowledge base:

Model outputs completely unrelated to reference answers (model clearly lacks relevant background knowledge).
Model outputs partially related to reference answers (model has background knowledge but it's outdated).

Billing information

Model evaluation costs come from two parts: evaluated model inference costs and judge model scoring costs.

Evaluated model inference costs

Model type	Billable?	Billing details
Preset models (such as qwen-max, qwen-plus)	Yes	Billed based on token consumption
Independently deployed models (fine-tuned and deployed)	No	No additional charges

Evaluated model billing formula:

Input tokens include:

System Prompt (if configured)
Prompts from evaluation data (user questions)

Output tokens include:

Model-generated responses (Output)

Judge model scoring costs (free for limited time)

Only large model evaluation (numerical, classification) involves judge model usage.

Complete cost calculation example

Scenario: Evaluate 100 data items using qwen-plus with Model Evaluation - Numeric, using qwen-max as judge model.

Assumptions: Average 50 tokens/Prompt, 200 tokens/Output, 500 tokens/scorer Prompt, 5 tokens/judge model output.

Cost calculation (qwen-plus input $0.26/1M tokens, output $0.78/1M tokens; judge model billed per Model Studio console pricing):

Evaluated model: 5,000/1,000,000 × 0.26 + 20,000/1,000,000 × 0.78 = $0.0013 + $0.0156 = $0.017
Judge model: Billed per current Model Studio console pricing.
Total cost ≈ $0.017 (excluding judge model costs).

Pricing reference

Latest pricing: See Model Studio Console.

Note:

Pricing may change—always refer to console display.
Some models may have limited-time promotional offers.

Next steps

If you're satisfied with the evaluated model's performance, start model invocation and integrate it into your business workflow.
If you're dissatisfied with the evaluated model's performance, select another model for re-evaluation. You might also consider learning and obtaining the ACA Generative AI Engineer Certification. Its free accompanying courses help you systematically understand large model capabilities, application scenarios, and optimization techniques.