AgentLoop:Evaluation result fields

Key concepts

• Evaluation task: An evaluation configuration that you create. It contains definitions such as the data source, evaluators, and scheduling policy.

• Run: An actual execution of an evaluation task. A run can be manually triggered or automatically scheduled within a continuous evaluation window.

• Evaluator: A unit that scores or judges data. A task can contain multiple evaluators, and each evaluator produces one result log entry.

• Evaluation result: The score, judgment, and explanation that a single evaluator produces for a single piece of data. Each evaluation result is one log entry in the evaluation_detail Logstore.

Common fields

The following fields are common to every evaluation result log entry. Some fields have a value only in specific modes.

Task and run information

The following fields describe the task and run context of the evaluation result.

Evaluation result

The following fields describe the score, judgment, and explanation that a single evaluator produces.

Error information

When the evaluation status is failed, the following fields record the cause of the failure.

Data association (data_link)

data_link is a JSON field that records the association information of the evaluated data. You can use this field to trace the raw data source of an evaluation result.

Evaluator metadata (eval_meta)

eval_meta is a JSON field that records the invocation metadata generated during evaluator execution.

Evaluator metrics (eval_metrics)

eval_metrics is a JSON field that records the token consumption and duration of the evaluator execution.

Evaluator input (eval_info)

eval_info is a JSON field that records the actual variable values that are passed to the evaluator. Each key corresponds to a variable name defined in the evaluator's variableMapping, and each value is the actual value of that variable.

For example, if the evaluator is configured with the following variable mapping:

{
  "variableMapping": {
    "input": "question",
    "expected": "label"
  }
}

eval_info contains the following content:

{
  "input": "The actual value of the question field",
  "expected": "The actual value of the label field"
}

Custom outputs (custom_outputs)

custom_outputs is a JSON field that stores custom outputs in addition to the standard results of the evaluator. The structure of this field is defined by the evaluator and has no fixed schema.