AgentLoop:AgentLoop quickstart: end-to-end guide

Procedure

1. Log on to the AgentLoop console.

2. In the Agent Spaces list on the console homepage, click Create Space in the upper-right corner.

3. In the Create Agent Space dialog box, configure the following parameters.

   Parameter	Description
   Region	Select cn-hangzhou, cn-shanghai, or cn-hongkong. The region cannot be changed after the space is created.
   Agent Space (AgentSpace)	The space name. The name must be 3 to 63 characters in length and can contain lowercase letters, digits, hyphens (-), and underscores (_). The name must be globally unique across all users. The name cannot be changed after the space is created.
   Description	A brief description of the space purpose (optional).
   CMS 2.0 Workspace	Auto-create: The system creates a new workspace named agentloop-<32-character code>. Select existing: Select an existing CMS 2.0 workspace in the same region under your account from the drop-down list for reuse, which allows you to unify the monitoring view with your existing application monitoring. The binding cannot be changed after the space is created.
   MSE Namespace	The system automatically creates a namespace named agentloop-<32-character code> to host agent assets such as Prompts and Skills.
   SLS Project	The system automatically creates a project named agentloop-<32-character code> to host audit and evaluation logs.

4. Click Create Space. The system automatically creates AgentLoop resources and associated cloud service resources, and establishes the binding relationships. If a step fails, the resources that have been created are not rolled back. You can rebuild the resources that were not created on the space management page.

Verification

Go to System Management > Space Management. In the Basic Information and Resource Bindings sections, verify that the AgentSpace name, region, associated CMS 2.0 workspace, MSE namespace, and SLS project are all ready.

1. Select an ingestion method

Go to the Integration Center and select an ingestion method based on your agent tech stack.

AgentTeams and AgentRun ingest observability data through CMS 2.0 by default. You do not need to select an ingestion method in the Integration Center, and you do not need to install an ARMS agent. The CMS 2.0 workspace that you bind when you create an AgentSpace in Step 1 completes the integration. You can go directly to 3. Verify the integration in this step to view the data.

2. Complete SDK integration

The following example uses a LangChain/LangGraph application:

# Install the Alibaba Cloud agent
pip install aliyun-bootstrap
aliyun-bootstrap -a install

# Configure environment variables
export ARMS_APP_NAME=my-agent-app
export ARMS_REGION_ID=cn-hangzhou
export ARMS_LICENSE_KEY=<Obtain from the Integration Center>
export ARMS_WORKSPACE=<CMS 2.0 workspace name associated with the AgentSpace>

# Start the application with the agent
aliyun-instrument python app.py

No changes to your business code are required. The agent automatically collects trace data for LLM calls, tool calls, Retrievers, LangGraph nodes, and more.

3. Verify the integration

Go to AI Agent Observability and check the following on the Tracing tab:

• The line charts for trace count, average latency, and token count increase with the ingested traffic.

• The Trace List displays the latest trace records. Click a Trace ID to go to the call chain details, where you can view model inputs and outputs, tool parameters, latency, and token consumption for each span.

4. View key metrics in the Agent Overview

Go to Agent Overview. On the Agent Insights page, you can view key metrics such as sessions, conversations, users, token usage, and average latency. You can also view the key metrics for each agent by agent dimension in the Trend Charts and the Agent List.

1. Use built-in evaluators

Go to Evaluation > Evaluators. The left-side panel lists the built-in evaluators grouped by category:

Click an evaluator name to go to the details page. On the Prompt & Capabilities tab, view the evaluator's Prompt and capability bindings (such as Skills and MCP). Built-in evaluators are ready to use out of the box and require no additional configuration.

2. Create a custom evaluator

For business-specific evaluation scenarios (such as complex task planning rationality or multi-turn conversation coherence), you can create custom evaluators for precise evaluation.

1. In the upper-right corner of the evaluator list page, click Create Evaluator.

2. Enter the evaluator's basic information, such as name, description, and tags.

3. Configure the Prompt: Write evaluation instructions that define the evaluation objectives, dimensions, and scoring criteria. For example, a Skill completeness evaluator:

   You are a rigorous and strict AI Agent evaluation expert. 
   For the AgentTrace to be evaluated (i.e., the complete execution trajectory of an Agent), 
   score it on the following dimensions (0–1)
   1. Accuracy of task understanding
   2. Rationality of tool invocation
   3. Correctness of multi-step reasoning
   Completeness of final output

4. View Extracted Variables: For example, input, output, agent_trajectory, tool_context, and rag_context. These variables are automatically bound to the corresponding mapping fields when the evaluation task runs.

5. Configure Capability Extensions (optional): Bind Skills or MCP to the evaluator agent so that it can invoke external retrieval, rule validation, knowledge base queries, and other capabilities during evaluation. This enables the Agent-as-a-Judge evaluation paradigm.

6. View the output definition and confirm whether to enable evaluation process tracing.

7. Click Save in the upper-right corner.

Create an evaluation task

1. Go to Evaluation > Evaluation Tasks and click Create Task in the upper-right corner. Follow the two-step wizard: Data Configuration and then Select Evaluators.

2. Configure data settings by referring to the following table:

   Parameter	Description
   Task Name	The business identifier for the evaluation task.
   Task Description	A brief description of the purpose (optional).
   Data Source	Select one of the following: Traces, Logs, or Datasets.
   Agent	Select the target agent application, such as ai-coding-agent-claude-code.
   Execution Policy	Continuous evaluation on new data: Scores newly reported data in real time. Evaluation on historical data: Performs backtesting on historical data within a specified time window.
   Time Range	The time range for historical data evaluation (for example, the last 24 hours).
   Sampling Ratio	Controls the evaluation data sampling rate (1%–100%). Use this parameter together with Maximum Evaluation Count to control the number of items evaluated per task run.

3. Select evaluators:

   • Select built-in or custom evaluators from the left-side panel. You can select multiple evaluators to run in parallel.

   • In the Configuration section on the right side, bind variable mapping fields as required by the evaluator. Common mappings include input ← trace.input, output ← trace.output, and agent_trajectory ← trace.agent_trajectory.

   • After the configuration is complete, click Update and Run to save the task and trigger execution immediately, or click Test Run for debugging.

Interpret evaluation results

After the task runs, view the evaluation result details and distribution on the Evaluation Task Details page.

1. Import Trace data

Go to Data Center > Datasets and click Create Dataset:

1. Enter the dataset name and description.

2. For data source, select Process from Trace Data.

3. In the data filter section, select an Agent Application from the drop-down list.

4. In the data processing section, select the desired processing template, such as ot_trace_qa_extract (Trace QA pair extraction), and specify the time range.

5. In the data preview section, click Generate Preview on the right to view the preview data.

6. After you confirm the data, click Create Dataset in the lower-right corner. The system creates the dataset in the background and runs the associated pipeline task to process and import the data.

2. Manual annotation

Go to the dataset details page, click the Annotation tab, and then click Create Annotation Template:

• Configure the Data Annotation Interface and Annotation Controls on the left side. You can preview the result on the right side.

• After the configuration is complete, click Start Annotation in the upper-right corner to annotate items one by one. You can skip items.

• You can view the annotation results in the agentloop_annotations field of the dataset.

3. Build Golden Sets and BadCase Sets

Based on evaluation results and manual annotation reviews, you can split datasets into the following types:

1. Create a Prompt asset

Go to Agent Assets > Prompts and click Create Prompt:

1. Enter the basic information: name (for example, my-prompt) and description.

2. In the Prompt editor, write the Prompt content and use placeholders such as {{input}} to declare variables.

3. Click Save Draft or Publish Version. Each published edit automatically increments the version number (v0.0.1, v0.0.2, v0.0.3, and so on).

2. Version management

The Version Information section on the right side of the asset details page displays the version history:

• Each version displays the version number, committer, change description, and timestamp.

• Click Compare Versions to compare the differences between the current version and a target version.

• Click Usage Guide to view the client integration method for referencing this asset in an agent.

3. Canary release and collaboration

Agent assets can be used with the MSE AI Governance Center for traffic splitting. When multiple people edit assets, version locks and commit messages help prevent conflicts. The Skills asset management workflow is similar to that of Prompts.