The experiment feature in AgentLoop allows you to use datasets to repeatedly test LLMs or AI Agents. You can verify optimization results, run regression tests, and discover issues through structured experiment plans.
LLM experiments and Agent experiments
Experiments are classified by the object being tested into LLM experiments and Agent experiments.
LLM experiments
LLM experiments test your model services. Common scenarios include:
Model comparison testing — Use the same prompts to test different models, then evaluate output metrics such as hallucination and correctness to select the best model for your specific scenario.
Prompt tuning — For the same set of questions, test models with different prompts and evaluate related metrics to find the optimal prompt.
Model degradation detection — Use a set of challenging questions to periodically test the model service provided by your model service provider and evaluate the correctness of model outputs. A significant drop in related metrics often indicates an issue with the model.
Agent experiments
Agent experiments test your deployed AI Agents. Common scenarios include:
Agent regression testing — After each Agent change, use the same dataset to run a round of regression tests and evaluate whether Agent performance on key metrics (such as accuracy, hallucination, and evidence support) has significantly declined.
Agent tuning verification — After targeted optimization for current bad cases, use those bad cases to run a round of regression tests on the optimized Agent and verify whether Agent performance on key metrics (such as accuracy, hallucination, and evidence support) has improved.
Online experiments and offline experiments
During experiments, HTTP requests are sent to your LLM or Agent service. This involves network isolation issues — your LLM service or Agent service may not be reachable from the AgentLoop server. However, your environment must be able to access the AgentLoop server through the public network or internal network. Therefore, experiments are distinguished into offline experiments and online experiments.
The core difference between offline and online experiments is whether the experiment is initiated from the AgentLoop server or from your own environment. The overall differences are as follows:
| Item | Online experiment | Offline experiment |
| Initiation environment | AgentLoop server | Your local environment |
| Network requirement | AgentLoop server must be able to reach your service | Your environment must be able to reach the AgentLoop server |
| Experiment group configuration | Required | Not required |
| Execution method | Configured and run on the platform | Copy generated code to run locally |
Trace propagation during experiments
When experiments are initiated through HTTP calls, regardless of whether they are online or offline experiments, a traceparent request header that carries traceId information in compliance with the W3C specification is added to the HTTP request headers. For example: traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.
If your AI Agent has completed onboarding in accordance with the AgentLoop onboarding guide, the traceId of the experiment request and the traceId of the Agent processing request are consistent, and the evaluators configured in the experiment plan can correctly obtain the Agent execution trajectory.
Prerequisites
Before you create an experiment, ensure that the following conditions are met:
Your AI Agent has completed onboarding in accordance with the AgentLoop onboarding guide. This is required for evaluators to correctly obtain the Agent execution trajectory.
Your environment can access the AgentLoop server through the public network or internal network.
For online experiments, the AgentLoop server must be able to reach your LLM or Agent service.
Create an online experiment
(Optional) Step 1: Register a model or Agent service
If you want to test a custom model service or Agent service rather than a built-in model service in AgentLoop, register it first.
Go to to register your model service or Agent service.
If you plan to use a built-in model service in AgentLoop, you can skip this step.
Step 2: Create a dataset
In Data Center, create a dataset and ensure it contains at least one data record. If you already have a dataset created through other methods, you can skip this step.
Step 3: Create an online experiment plan
On the Experiments page, click Create Experiment Plan.
Enter the plan name and plan description, select Online Experiment, and select the dataset to associate with this experiment.
Click Create Plan.
After you click Create Plan, the experiment configuration page appears. This page is divided into three main sections:
Experiment configuration section
This section displays experiment configurations in a card format. Each experiment configuration contains two phases:
Select the endpoint for this experiment. This can be a built-in model service in AgentLoop, or a model service or Agent service that you registered in the previous steps.
Fill in the prompt template to send to the model, or the HTTP request body template to send to the Agent service. In the template, you can reference dataset column names using double curly braces.
Evaluation metrics section
In the evaluation metrics section, click the configuration button on the right to select zero or more evaluators. After you select evaluators, check the variable mapping section of the evaluators. The following variables are available for configuring variable mappings in evaluators:
| Variable | Description |
experiment_input | The input of a single experiment run, which is the content of the prompt template or request body template after variable substitution. |
experiment_output | The output of a single experiment run, which is the response content from the model or your Agent. |
experiment_agent_trajectory | The execution trajectory of your Agent in this experiment. This variable can only be replaced with the actual Agent trajectory during evaluation if your Agent application has completed onboarding according to the AgentLoop onboarding guide. |
dataset.xxx | A column in the dataset. Replace xxx with the column name of the dataset. |
Dataset display section
The leftmost column displays the current dataset data. After the experiment completes, experiment results are displayed in columns on the right side, with one column for each experiment group.
Step 4: Run the experiment
Click the experiment button in the upper-right corner to start the experiment. After the experiment completes, results are displayed in the dataset display section. Each experiment group shows its results in a separate column alongside the dataset data.
Create an offline experiment
The prerequisite steps for offline experiments are the same as for online experiments. However, on the offline experiment configuration page, you do not need to configure experiment groups. Instead, a Python code snippet for local execution is generated based on the current experiment configuration. You can copy and adjust the code, then run it locally.
To create and run an offline experiment:
Create an experiment plan and select Offline Experiment as the experiment type.
Configure the evaluators for your experiment.
Click the button in the upper-right corner to obtain the local execution code. Select one of the following three modes based on how you want to initiate the experiment locally:
Initiate the experiment through an HTTP request.
Initiate the experiment by executing a bash command.
Other cases where you fully customize how the experiment is executed and what content it returns.
Copy the generated code and run it in your local environment.
For more information about reporting offline experiment results, see Report offline experiment results.
View experiment results
Regardless of whether it is an offline or online experiment, after each experiment run completes, an experiment record appears on the Experiment Records page.
You can select any two experiment records (typically two records from the same experiment plan) to compare, and view the response result differences and scoring differences for each data entry between the two experiment runs.