AgentLoop:Run offline experiments and report results with the AgentLoop SDK

Prerequisites

• You have created an agent space.

• You have created a dataset and imported data. The dataset must contain at least one record. The field names of the record are used as the keys of the experiment input.

• You have created an offline experiment plan through an evaluation task and associated the experiment plan with the dataset above. The experiment plan type must be offline (experiment_type is offline).

• (Optional) You have configured an evaluator for the experiment plan. You can also skip the platform configuration and specify the evaluator through EvaluatorConfig in the SDK code, as described in Step 5: Configure evaluators.

• You have obtained the Alibaba Cloud AccessKey ID and AccessKey secret for SDK authentication.

Step 1: Install the SDK

Install agentloop-sdk with pip:

pip install agentloop-sdk

Step 2: Configure experiment parameters

Use the AgentLoopConfig class to configure the core parameters of the experiment. The dataset is resolved automatically from the experiment plan, so you only need to identify the agent space, the experiment plan, and the region.

from agentloop_sdk import AgentLoopConfig

config = AgentLoopConfig(
    agent_space="my-agent-space",
    experiment_plan_id="my_plan_id",
    region_id="cn-hangzhou",
)

AgentLoopConfig supports the following parameters:

Step 3: Configure credentials

The SDK uses the Alibaba Cloud AccessKey to authenticate your identity. The following two methods are supported.

export AGENTLOOP_AK="your-access-key-id"
export AGENTLOOP_SK="your-access-key-secret"

from agentloop_sdk import AgentLoopConfig

config = AgentLoopConfig(
    agent_space="my-agent-space",
    experiment_plan_id="my_plan_id",
    region_id="cn-hangzhou",
    access_key_id="your-ak",
    access_key_secret="your-sk",
)

Step 4: Define the solution function

The solution function defines how to call the agent and obtain results. The SDK provides two methods: HTTP calls and command-line calls. HTTP calls apply to remote agents accessed through an API. Command-line calls apply to locally deployed agents.

The task.input of the solution function is a dict that corresponds to one record in the dataset, where the keys are the dataset column names.

from agentloop_sdk import http_solution_with

solution = http_solution_with(
    url="https://your-agent-endpoint.com/invoke",
    headers={"key": "your-api-key"},
    body_builder=lambda task: {
        "input": task.input.get("input", ""),
        "stream": False,
    },
    timeout=60.0,
)

solution = http_solution_with(
    url="https://your-agent-endpoint.com/invoke",
    headers={"key": "your-api-key"},
    body_builder=lambda task: {
        "input": task.input.get("input", ""),
        "stream": True,
    },
    timeout=120.0,
)

solution = http_solution_with(
    url="https://your-agent-endpoint.com/invoke",
    headers={"key": "your-api-key"},
    body_builder=lambda task: {
        "input": task.input.get("input", ""),
        "stream": True,
    },
    timeout=120.0,
    chunk_extractor=lambda chunk: chunk.get("output", ""),
)

from agentloop_sdk import http_solution, HttpRequestSpec

def build_request(task):
    return HttpRequestSpec(
        url="https://your-agent-endpoint.com/invoke",
        headers={"key": "your-api-key"},
        json={"input": task.input.get("input", "")},
    )

solution = http_solution(build_request, default_timeout=60.0)

from agentloop_sdk import command_solution_with

# Static command
solution = command_solution_with(["python", "agent.py", "--input", "hello"])

# Build the command dynamically from the Task
solution = command_solution_with(
    lambda task: ["python", "agent.py", "--input", task.input.get("input", "")],
    default_timeout=120.0,
)

from agentloop_sdk import command_solution

def build_command(task):
    return ["python", "agent.py", "--input", task.input.get("input", "")]

solution = command_solution(build_command, default_timeout=120.0, cwd="/path/to/agent")

Step 5: Configure evaluators (optional)

To have the AgentLoop platform evaluate the experiment automatically, configure an evaluator. You can configure the evaluator on the platform when you create the experiment plan, or specify it in code with EvaluatorConfig. When you set evaluators on AgentLoopConfig, platform-side evaluation is triggered automatically after the experiment completes.

Pass a list of EvaluatorConfig objects to the evaluators parameter of AgentLoopConfig:

from agentloop_sdk import AgentLoopConfig, EvaluatorConfig

config = AgentLoopConfig(
    agent_space="my-agent-space",
    experiment_plan_id="my_plan_id",
    region_id="cn-hangzhou",
    evaluators=[
        EvaluatorConfig(
            evaluator_ref="Builtin.agent_correctness",
            result_type="score",
            variable_mapping={
                "input": "experiment_input",
                "output": "experiment_output",
                "expected_output": "dataset.expected_output",
            },
        ),
    ],
)

EvaluatorConfig uses the following fields:

Step 6: Run the experiment

After the configuration is complete, call the run function provided by the SDK to run the experiment. The SDK supports sequential and parallel execution.

import asyncio
from agentloop_sdk import run_experiment

asyncio.run(run_experiment(
    config=config,
    solution_fn=solution,
    result_dir="./results",
    n_repeat=1,
))

import asyncio
from agentloop_sdk import run_experiment_parallel

asyncio.run(run_experiment_parallel(
    config=config,
    solution_fn=solution,
    result_dir="./results",
    n_repeat=1,
    n_workers=4,
))

Step 7: Verify the experiment results

After the experiment finishes, confirm that the run completed and that the results reached the AgentLoop platform.

1. Check the local results. The SDK saves results to the directory specified by result_dir (./results by default). Confirm that the directory contains the generated result files.

2. Confirm platform reporting. On the AgentLoop platform, confirm that a new experiment record appears for your experiment plan. The record name is generated from the plan name plus a timestamp.

3. Confirm evaluation. If an evaluator is configured—either on the platform or through EvaluatorConfig—platform-side evaluation is triggered automatically after the experiment completes. Confirm that the evaluation results appear on the platform.

Clean up

The tutorial writes experiment results to the local result_dir directory and reports experiment records to the AgentLoop platform. To reclaim local disk space, delete the results directory when you no longer need the local copies.

Trace correlation

When http_solution and http_solution_with send HTTP requests, they automatically inject a traceparent header that complies with the W3C Trace Context specification into the request header. They also extract eagleeye-traceId or traceId from the response header to associate the experiment requests with platform-side trace correlation. You do not need to handle the trace headers manually.

Complete example

The following example shows a complete offline experiment workflow: configuring the experiment parameters, defining an evaluator, calling an agent over HTTP, and running with 4 parallel workers.

import asyncio
from agentloop_sdk import (
    AgentLoopConfig,
    EvaluatorConfig,
    http_solution_with,
    run_experiment_parallel,
)

async def main():
    config = AgentLoopConfig(
        agent_space="default-cn-hangzhou",
        experiment_plan_id="cc_plan_id",
        region_id="cn-hangzhou",
        experiment_name="Offline-Experiment-LocalAgent",
        experiment_config={"agent_name": "LocalAgent"},
        evaluators=[
            EvaluatorConfig(
                evaluator_ref="Builtin.agent_correctness",
                result_type="score",
                variable_mapping={
                    "input": "experiment_input",
                    "output": "experiment_output",
                    "expected_output": "dataset.expected_output",
                },
            ),
        ],
    )

    solution = http_solution_with(
        url="https://your-agent-endpoint.com/invoke",
        headers={"key": "your-api-key"},
        body_builder=lambda task: {
            "input": task.input.get("input", ""),
            "stream": True,
        },
        timeout=120.0,
        chunk_extractor=lambda chunk: chunk.get("output", ""),
    )

    await run_experiment_parallel(
        config=config,
        solution_fn=solution,
        n_workers=4,
    )

if __name__ == "__main__":
    asyncio.run(main())