Evaluation dimensions_Alibaba Cloud Model Studio(Model Studio) - Alibaba Cloud Model Studio

This topic describes how to create and manage evaluation dimension templates.

Evaluation dimensions overview

Evaluation dimensions define the scoring rules for model evaluation tasks. Each dimension is configured with one scorer. After being created as a template, it can be referenced by multiple evaluation tasks, enabling unified management and reuse of evaluation criteria.

The Model Studio platform provides the following five scorer types, covering three major evaluation paradigms: LLM-based automatic evaluation, rule matching, and human review:

LLM evaluation - Categorical: The judge model classifies model outputs by user-defined labels (Pass/Fail). It is suitable for scenarios that require semantic understanding, such as content safety checks and answer correctness judgment.
LLM evaluation - Numerical: The judge model assigns continuous numerical scores to model outputs (for example, 0~5). It is suitable for scenarios that require fine-grained quantification, such as Q&A quality assessment and content generation quality scoring.
Rule evaluation - String matching: Programmatically determines the exact matching relationship (equal/not equal/contains) between the model output and the reference text. It is suitable for scenarios with deterministic standard answers, such as Function Calling and NL2SQL.
Rule evaluation - Text similarity: Uses algorithms to calculate the similarity between the model output and the reference text (BLEU/ROUGE/Cosine, and more). It is suitable for quantifying the similarity of open-ended responses, such as translation, summarization, and rewriting.
Human evaluation - Categorical: Evaluators manually classify model outputs by label. It is suitable for scenarios that require subjective judgment, such as tone and style, and compliance review.

Dimension type	Evaluation method	Applicable scenarios	Reference answer required	Level of automation
LLM evaluation - Categorical	Judge model classifies by label	Content safety, answer correctness	No	Fully automated
LLM evaluation - Numerical	Judge model scores by range	Q&A quality, generation quality	No	Fully automated
Rule evaluation - String matching	Exact/partial string matching	Function Calling, NL2SQL	Yes	Fully automated
Rule evaluation - Text similarity	BLEU/ROUGE/Cosine and other algorithms	Translation, summarization, rewriting	Yes	Fully automated
Human evaluation - Categorical	Manual review by label	Tone and style, compliance review	No	Manual

When selecting a scorer type, you can refer to the following decision path:

Standard answers with a fixed format: prefer rule evaluation - string matching, which offers the lowest cost and fastest speed.
Standard answers with varied wording: use rule evaluation - text similarity, which accommodates wording variations through algorithms.
No standard answer, semantic understanding required: use LLM evaluation (categorical or numerical) and let the judge model assess at the semantic level.
Subjective judgment or professional review required: use human evaluation - categorical, with evaluators reviewing each item.

Model Studio provides preset scorer templates for LLM evaluation types (such as Standard Matching, Sentiment Analysis, and Comprehensive Evaluation). After you select one, the scorer prompt and default label configuration are automatically populated, which is suitable for quickly starting common scenarios. To customize judging criteria for specific business logic, you can choose a custom scorer and manually write the prompt.

After you create a dimension template, you can reference it as the judging criteria in evaluation tasks. For more information about creating and running evaluation tasks, see Model evaluation.

Create an evaluation dimension template

Log on to the Model Studio console. In the left-side navigation pane, choose Model Evaluation > Evaluation Dimensions, and click Create Evaluation Dimension to enter the creation page.

When creating an evaluation dimension template, you must first fill in the following common fields:

Dimension name: required, no more than 20 characters.
Description: optional, no more than 100 characters. It is used to supplement the judging objective of the dimension.
Type: required. Select one of the five scorer types. The scorer type cannot be changed after creation. If you select the wrong one, you can only delete and recreate it.

The following table lists the common and type-specific parameters for creating an evaluation dimension template.

Parameter	Description	Required	Value description
Dimension name	Name of the evaluation dimension template	Yes	No more than 20 characters
Description	Supplementary explanation of the dimension	No	No more than 100 characters
Type	Evaluation method of the scorer, which cannot be changed after creation	Yes	Select one of the 5 types
Judge model	The large language model that performs the evaluation	Required for LLM evaluation types	Select from the drop-down list
Scorer template	The prompt template that guides the judge model	Required for LLM evaluation types	Preset template or custom
Score range	Minimum and maximum values of numerical scoring	Required for the numerical type	Integer, default 0~5
Pass threshold	Minimum score or similarity for a pass judgment	Required for numerical/similarity types	Numeric

After you select a type, the page displays the type-specific configuration items. The complete creation process for each type is described below.

Important The scorer type cannot be changed after creation. If you select the wrong one, you can only delete and recreate it.

LLM evaluation - Categorical

LLM evaluation - Categorical uses the judge model to classify model outputs by label. Complete creation steps:

Select a judge model: select the large language model used for evaluation from the drop-down list.
Select a scorer template: select a preset template (Standard Matching or Sentiment Analysis) or a custom scorer. After you select a preset template, the system automatically populates the prompt and default labels. For more information, see Configure scorer prompts.
Write the scorer prompt: the prompt must contain at least one variable (${prompt}, ${output}, ${completion}) and be no more than 50000 characters. The prompt defines the judging criteria and output format of the judge model.
Configure Pass labels and Fail labels: define the classification labels for pass and fail, each no more than 20 characters. Pass and Fail labels cannot be duplicated.
Click Save.

LLM evaluation - Numerical

LLM evaluation - Numerical uses the judge model to assign scores within a score range. Complete creation steps:

Select a judge model.
Select a scorer template: select a preset template (Comprehensive Evaluation or Semantic Similarity) or a custom scorer.
Write the scorer prompt: at least one variable must be included.
Set the score range: configure the minimum and maximum values, both integers. The minimum value must be ≥ 0, the maximum value must be ≥ 1, and the default is 0~5.
Set the pass threshold: a score at or above this value is judged as Pass, with a step of 0.1. The threshold automatically follows the score range.
Click Save.

Rule evaluation - String matching

Rule evaluation - String matching uses programmatic rules to judge model outputs. Complete creation steps:

Configure the evaluation input (left side): enter the reference text. You can use variables to reference evaluation data.
Select a comparison operator: Equal (passes when the output exactly matches the reference), Not equal (passes when they do not match), or Contains (passes when the output contains the reference text).
Configure the model output (right side): enter the text to match. You can use variables. At least one side of the evaluation input and model output must contain a variable.
Click Save.

Rule evaluation - Text similarity

Rule evaluation - Text similarity uses algorithms to calculate the similarity between the output and the reference text. Complete creation steps:

Configure the evaluation input: enter the reference text. You can use variables.
Select an evaluation metric: seven algorithms are supported—Fuzzy Matching, BLEU, Cosine Similarity, ROUGE-1, ROUGE-2, ROUGE-L, and Accuracy.
Configure the model output: enter the text to evaluate. You can use variables. At least one side of the evaluation input and model output must contain a variable.
Set the pass threshold: the range is 0~1 with a step of 0.01, and a similarity at or above this value is judged as Pass.
Click Save.

Human evaluation - Categorical

Human evaluation - Categorical relies on manual review by evaluators. Complete creation steps:

Configure the Pass label: define the meaning of the label for pass judgments.
Configure the Fail label: define the meaning of the label for fail judgments. Pass and Fail labels cannot be duplicated.
Click Save.

Configure scorer prompts

The scorer prompt is the instruction that guides the judge model in evaluating output quality. It applies only to LLM evaluation types (categorical and numerical). By writing the prompt, you define the judging criteria and output format of the judge model.

Prompt variables

Scorer prompts support the following three variables, which the system automatically replaces with actual values during evaluation execution:

${prompt}: the question or instruction entered by the user, that is, the original prompt sent to the model under test.
${output}: the output of the model under test, that is, the model's reply to the user input.
${completion}: the reference standard answer, that is, the expected output preset in your evaluation dataset.

Use the ${variable name} syntax in the prompt to reference variables. The prompt must contain at least one variable; otherwise, it cannot be submitted.

Preset templates

The system provides preset scorer templates for LLM evaluation types. After you select one, the prompt content and default label configuration are automatically populated. The template list is dynamically retrieved from the system configuration.

Preset templates for the categorical type:

Standard Matching (default): the judge model determines whether the output meets the expected standard.
Sentiment Analysis: the judge model determines the sentiment tendency of the output.
Custom Scorer: use the custom prompt you write.

Preset templates for the numerical type:

Comprehensive Evaluation (default): the judge model scores comprehensively across multiple quality dimensions.
Semantic Similarity: the judge model evaluates how semantically close the output is to the reference answer.
Custom Scorer: use the custom prompt you write.

Write a custom prompt

When you select the custom template, write the scoring guidelines in the prompt editing area. Key points:

Use the ${prompt}, ${output}, and ${completion} variables to reference evaluation data.
Clearly define the scoring criteria. For the categorical type, define the judgment conditions for each label; for the numerical type, define the quality standard corresponding to each score.
Specify output format requirements to ensure that the judge model's evaluation results can be correctly parsed by the system.

The following is an example custom prompt for LLM evaluation - Categorical:

Evaluate the quality of the following model response. User question: ${prompt}, model response: ${output}, reference answer: ${completion}. Scoring criteria: if the response contains the core elements of the reference answer and has no factual errors, judge it as Pass; if the response omits key information or contains factual errors, judge it as Fail. Output only Pass or Fail.

You can switch scorer templates or click the Restore Default Prompt button to restore the prompt to the template preset content. Switching templates overwrites the current prompt content. Back up manually edited content before switching.

Manage evaluation dimension templates

On the evaluation dimension list page, you can view, modify, and delete the dimension templates you created.

View dimensions

The dimension list page displays all created dimension templates and supports the following operations:

Search by name: enter a dimension name keyword in the search box for a fuzzy query.
Filter by type: click the filter icon in the Type column header to filter by selecting multiple scorer types.
Sort by creation time: click Sort by creation time to toggle between ascending and descending order.

Click the name of a target dimension in the dimension list to go to the details page and view the complete configuration of the dimension (including scorer type, prompt, labels, score range, and more).

Modify dimensions

On the dimension details page, you can modify the dimension name, description, scorer prompt, labels, and other configuration items. The scorer type cannot be changed after creation. Changes take effect immediately after saving.

Delete dimensions

Click the Delete button in the operation column of the target dimension in the dimension list. The deletion operation is irreversible. If the dimension is referenced by an evaluation task, it cannot be deleted. You must remove the associated evaluation tasks before deleting the dimension.

Evaluation dimension design recommendations

Designing high-quality evaluation dimensions helps accurately measure model performance. The following are key recommendations from practice.

Type selection and iteration

Select an appropriate type based on the evaluation scenario: prefer rule evaluation for scenarios with standard answers (low cost and fast speed), use LLM evaluation for scenarios that require semantic understanding, and use human evaluation for subjective quality judgments. For a detailed comparison of types, see Evaluation dimensions overview.

We recommend that you first create dimensions with preset templates and run an evaluation on a small sample (10~20 data entries) to verify the effect. Based on the cases with large deviations in the evaluation results, adjust the judging criteria and score definitions of the prompt in a targeted manner, and scale up the evaluation after verification. Gradually transition from preset templates to custom prompts.

Parameter configuration recommendations

Pass and Fail labels should be mutually exclusive and collectively exhaustive, that is, all possible outputs can be classified into one of them. Binary classification is sufficient for simple scenarios; when multiple labels are required for complex scenarios, keep the number of labels low to reduce judgment ambiguity.

For LLM evaluation - Numerical, we recommend a score range of no more than 10 (an excessively large range reduces LLM scoring consistency). The default 0~5 range with a pass threshold of 3 is a common configuration. Set the pass threshold based on the business tolerance for quality: the higher the threshold, the stricter the pass standard.

Select an appropriate similarity algorithm based on the evaluation task type:

Translation tasks: BLEU (measures n-gram exact matching).
Summarization tasks: ROUGE-L (measures the longest common subsequence).
Semantic relevance: Cosine similarity (measures semantic vector distance).
Classification tasks: Accuracy (measures the exact match rate).

Naming conventions

We recommend the naming pattern of "assessment aspect + evaluation method", such as "Answer accuracy - LLM scoring", "Sentiment tendency - label classification", and "Format compliance - string matching". Clear naming helps you quickly identify and select dimensions in evaluation tasks.

FAQ

The following are common issues and solutions when you use the evaluation dimension feature.

Issue: After creating an evaluation dimension template, I found that the wrong scorer type was selected.

The scorer type cannot be changed after creation. You can only delete the dimension template and recreate it. If the dimension is referenced by an evaluation task, you must remove the associated evaluation tasks before deleting it.

Issue: A message appears when I delete a dimension template, indicating that it cannot be deleted.

A dimension template cannot be deleted when it is referenced by evaluation tasks. Delete or modify the evaluation tasks that reference the dimension in the evaluation task list first, and then delete the dimension template.

Issue: When I submit a scorer prompt, the system prompts that variables are missing.

The scorer prompt must contain at least one variable (${prompt}, ${output}, or ${completion}). Check the prompt content and make sure that variables are correctly referenced in the ${variable name} format.

Issue: The submission fails after I configure Pass/Fail labels.

Check whether the Pass and Fail labels are duplicated. The same label cannot appear in both Pass and Fail, and labels within Pass or within Fail cannot be duplicated either.

Issue: After I switch scorer templates, the prompt content I manually edited disappears.

This is expected behavior. Switching templates replaces the prompt with the template preset content. If you have important custom prompt content, back it up before switching templates.

Issue: Does using evaluation dimensions of the LLM evaluation type incur fees?

Creating and managing evaluation dimension templates is free of charge. However, during evaluation task execution, the LLM evaluation type calls the judge model for inference, and the resulting inference fees are charged according to the billing standard of the judge model.