:What is AgentLoop

Agent full-stack observability

AgentLoop uses proprietary Alibaba Cloud probes, open-source probes, and deep integrations with cloud services to enable non-intrusive, end-to-end collection of observability data, including traces, metrics, and logs. Based on UModel, it automatically discovers topological relationships between entities like Agent → Tool → Model to build a full-stack Agent Ontology topology view. Combined with the STAROps intelligent diagnosis engine, it quickly pinpoints performance bottlenecks and token consumption hotspots.

A trace records the complete execution process of a single agent request from start to finish. This includes user input, model calls, tool calling, retrieval augmentation, memory access, and the final response. With traces, developers can reconstruct an agent's reasoning and execution path to locate slow requests, errors, abnormal tool calls, and bad cases.

Agent behavior auditing

AgentLoop builds a tamper-resistant evidence chain for every tool call and decision branch an agent makes, achieving 100% action traceability. The platform's built-in abnormal behavior detection engine can identify risk patterns such as unauthorized operations and data exfiltration in real time, issuing risk alerts within minutes.

Agent evaluation

AgentLoop introduces the Agent-as-a-Judge evaluation paradigm, where the evaluator itself is an agent capable of complex task planning, tool use, and multi-step reasoning. It performs deep evaluations based on execution trajectories, delivering results much closer to human expert assessment than traditional LLM-as-a-Judge methods.

AgentLoop supports the evaluation of model outputs, RAG retrieval results, tool calling, and agent execution trajectories. The data source for evaluators can be traces, logs, or a dataset. Evaluator types include pre-built evaluators (for metrics like hallucination, correctness, and task completion) and custom evaluators. Evaluation results can be used to discover bad cases, monitor quality in production, and verify whether version changes cause regressions.

Agent experimentation

AgentLoop provides a Playground for online comparative experiments. It supports both agent and LLM experiment types, suitable for validating real enterprise agents or testing in a sandbox environment. This feature is scheduled for official release before June 30, 2026.

Data center

An AgentLoop dataset is a data store purpose-built for AI scenarios. Beyond traditional log storage, it provides full CRUD operations, a flexible schema, vector search, and multi-dimensional analysis, upgrading AI runtime data from read-only logs to manageable assets.

The built-in pipeline data processing engine transforms massive volumes of online trace data into high-quality, reusable data assets, such as a golden dataset or a bad case dataset. It supports a multi-step orchestration process: data source ingestion → data reduction (filtering, deduplication, and sampling) → feature extraction → writing to the target dataset. This entire process runs automatically, saving over 90% of manual data processing effort.

Memory

AgentLoop Memory is a persistent memory layer purpose-built for AI agents that provides the following capabilities:

• Cross-session coherence: Retain key information such as conversation history, task status, and decision rationale for the long term. Through efficient retrieval and context injection mechanisms, it dynamically provides relevant background for the model in new interactions.

• Highly adaptive personalization: Systematically record user preferences, such as formatting requirements and communication style, and remember historical behavior patterns and long-term goals, enabling the model to generate highly customized outputs.

• Deep reasoning based on historical information: By empowering an AI agent with the ability to remember, learn, and evolve, it significantly improves interaction continuity and intelligence.

Memory policies include four types: Facts, Episodic, Summary, and Custom. Each type uses a different retrieval and organization method, covering the spectrum from specific events to abstract summaries.

Agent assets

AgentLoop asset management provides centralized management, version control, and collaborative iteration for an enterprise agent's prompts and skills. Agent assets are the core units that define an agent's behavior logic, and an agent's performance largely depends on the correctness and consistency of this logic. AgentLoop lets teams manage agent behavior like code: every change is traceable, its effect is measurable and comparable, the release cadence is controllable, and it supports canary releases and collaborative editing.

Agent-level, not LLM-level, observability

Traditional LLMOps tools observe only single model calls, failing to understand the causal relationships and dependencies between steps in a multi-step reasoning chain. AgentLoop treats the complete agent execution trajectory as the object of observation. It automatically parses the Agent → Tool → Model topology, supporting analysis from both a business perspective (such as the end-to-end latency of a user request) and a technical perspective (such as a tool call failing and retrying three times).

Agent-as-a-Judge evaluation paradigm

In AgentLoop, the evaluator itself is an agent, equipped with the ability to use tools, retrieve external knowledge, and perform multi-step reasoning. Compared to LLM-as-a-Judge, which makes a single-turn judgment on a model's output using only a prompt template, Agent-as-a-Judge can replay the entire execution trajectory, call verification tools to check the correctness of results, and compare against historical baseline data. This delivers an evaluation much closer to that of a human expert.

Trace2Dataset automated data flywheel

The AgentLoop pipeline engine automatically transforms real-time trace data from online environments into structured datasets, simplifying the data cleaning process. Through processing steps such as filtering, deduplication, sampling, and clustering, raw operational data is continuously processed into golden datasets, bad case datasets, or post-training datasets. This creates a data flywheel effect where "the more data runs, the higher its quality becomes."

Zero-intrusion and framework-agnostic integration

AgentLoop offers a zero-intrusion integration method based on proprietary Alibaba Cloud probes. You can complete data collection by modifying only configurations, without changing your business code. It also supports integration through the OpenTelemetry standard protocol and is compatible with mainstream agent frameworks like LangChain/LangGraph, AgentScope, Dify, OpenClaw, and the Claude Agent SDK, as well as various self-developed agent systems.

Enterprise-grade security and compliance

AgentLoop provides audit capabilities with 100% action traceability, creating a tamper-resistant evidence chain for every tool call and decision branch. Its built-in abnormal behavior detection engine can identify risk patterns like unauthorized operations and sensitive data leaks in real time, helping enterprises meet the compliance traceability and regulatory review requirements of industries such as finance, government, and healthcare.

Closed-loop optimization with context engineering

AgentLoop provides agents with long-term context capabilities through memory and the experience base (in private preview). Memory allows an agent to understand user background and preferences across sessions, while the experience base enables the agent to learn best practices from historical executions. By combining both, an agent can automatically retrieve and inject relevant context into the runtime during each interaction, achieving a self-evolving effect of "getting smarter with use" without needing to retrain the model.

Improve intelligent customer service quality

An enterprise's intelligent customer service agent handles a large volume of real user conversations, and response quality directly affects customer satisfaction. AgentLoop can evaluate the quality of each conversation turn, automatically identifying bad cases such as irrelevant answers, hallucinations, and information omission. It then compiles high-frequency issues into a dataset for targeted optimization. By using memory to maintain the continuity of user preferences and the experience base to capture best-response patterns for complex problems, you can continuously improve the resolution rate and user satisfaction of your customer service agent.

Safeguarding coding agent quality

Coding agents, such as AI code assistants and automated programming tools, produce code of varying quality. Traditional code reviews cannot cover the reasoning quality within an agent's decision-making process. AgentLoop supports the observation and evaluation of a coding agent's entire execution trajectory across multiple dimensions, including the rationality of its task planning, the accuracy of its tool calls, and the correctness of its code output. This helps teams verify the effects of version changes through experiments before the agent goes live and intercept quality regressions through continuous evaluation in production.

Agent version quality gates

Enterprise agent applications frequently undergo changes such as prompt tuning, model upgrades, and the addition of new skills. AgentLoop experimentation capabilities allow teams to conduct A/B tests based on historical datasets before implementing changes, quantifying the new version's performance differences in accuracy, latency, and cost. By establishing an automated quality gate with the evaluation system, you can ensure that only changes that pass regression testing are released to the production environment, blocking change-related failures before they go live.

Agent cost governance (FinOps)

When an enterprise runs multiple agent applications simultaneously, token consumption and model call costs can grow rapidly and become difficult to attribute. AgentLoop's full-stack observability capabilities precisely track the token consumption and latency for each agent, tool call, and model inference. This helps enterprises identify cost hotspots, such as a tool that frequently times out and triggers retries, or a prompt template with abnormal token consumption. By using experiments to compare the cost-effectiveness of different models and configurations, you can implement fine-grained FinOps governance for your agents.

Compliance auditing for finance and government

Highly regulated industries such as finance and government require every decision-making step of an AI system to be traceable and auditable. AgentLoop's behavior auditing capability provides a complete evidence chain for the agent's execution process, supporting audit replays organized by time, user, and agent application. The built-in abnormal behavior detection engine can provide real-time warnings for unauthorized operations and sensitive data access, helping enterprises meet regulatory compliance requirements.

Progressive onboarding path

AgentLoop recommends the following progressive onboarding path:

1. Start with observability: Use proprietary probes to ingest full-stack agent data. You can see trace data in the console within five minutes to quickly establish an observability baseline.

2. Turn data into assets: Use pipelines to automatically convert online traces into structured datasets. Combine this with labeling to build ground truth, shifting from passively accumulating logs to actively building data assets.

3. Establish an evaluation system: Build or select suitable evaluators to perform real-time quality assessments for your online business. At the same time, use datasets and backtesting to establish a quality measurement baseline for your agent.

4. Create a continuous optimization loop: Create a data flywheel by tuning agent assets or using memory and the experience base to improve the agent's long-term performance.

Building an evaluation system

The evaluation system is the core driver of the AgentLoop data flywheel. We recommend building it with the following approach:

• Start with pre-built evaluators: AgentLoop provides out-of-the-box pre-built evaluators for metrics like hallucination, correctness, and task completion, which are ideal for quickly establishing a baseline.

• Build business-specific custom evaluators (Agent-as-a-Judge): For complex scenarios, such as verifying the correctness of multi-step reasoning or the rationality of tool calling, write custom prompts and skills to achieve evaluation results closer to those of a human expert.

• Calibrate with human feedback: Periodically verify the accuracy of the evaluator agent through manual labeling to continuously calibrate the evaluation standards.

• Establish version regression testing: Before each agent change, run experimental regression tests on historical evaluation datasets to quantify the impact of the change.

Dataset management

Datasets are the foundation for evaluation, experimentation, and optimization. We recommend organizing them by purpose:

• Evaluation benchmark set: A set of high-quality samples reviewed by humans, serving as the gold standard for evaluation.

• Bad case set: A collection of failure cases identified through evaluation, used for targeted optimization and regression testing.

• Online sampling set: A dataset automatically sampled from online traces by pipelines, used for continuous monitoring and trend analysis.

• Experiment dataset: A dataset prepared for a specific experiment plan to ensure the comparability and reproducibility of results.

Agent asset version management

AgentLoop recommends managing agent assets like code:

• Create a new version for each change to a prompt or skill, and maintain a complete change history.

• Before a change goes live, validate its impact on performance through experiments to quantify benefits or risks.

• Use the canary release capability to gradually ramp up traffic. After confirming that online metrics are stable, roll out to all traffic.

• For collaborative editing, clearly define change owners and review processes to prevent conflicts.