What Is the Newly Released AgentLoop by Alibaba Cloud?

This article introduces Alibaba Cloud's AgentLoop, a one-stop platform for observing, evaluating, and continuously optimizing the performance and reasoning trajectories of enterprise AI agents.

The evolution of software engineering in every era is defined by how it manages increasing uncertainty.

In the era of traditional software, engineers used logs, metrics, and distributed tracing to observe the runtime status of distributed systems. In the cloud-native era, containers and microservices increased system complexity, requiring richer and more fine-grained observation metrics. Now, in the Agent era, a new uncertainty has emerged: an intelligent agent no longer just executes a deterministic code path. Instead, it operates through non-deterministic reasoning, autonomous planning,,, tool calling, knowledge retrieval, and dynamic memory access. The focus of observability has shifted from standard infrastructure metrics to the full reasoning trajectory of the Agent.

1. Agent Optimization: Why Must Observation Come First?

You can’t optimize what you can’t observe.

Online issues in traditional applications are relatively deterministic, such as a slow query, a memory leak, or a timed-out RPC call. Engineers can locate the specific line of code using APM tools and fix it for redeployment, which usually resolves the issue. This "issue detection, root cause location, fix, and verification" workflow is built on the premise that system behavior is predictable.

Agent applications break this premise. After a user request reaches an Agent, the system goes through an uncertain execution progress: the model might choose to call Tool A instead of Tool B, generate a completely different response due to decoded context differences, or fall into an infinite loop during multi-round reasoning. Given the exact same prompt and model, the behavioral path can vary every time an input is processed.

Therefore, Agent observation is often not about whether a specific metric reports an error, but rather focusing on the outcome quality, including the accuracy and richness of the answer, the rationality of reasoning and action planning, and even the token efficiency.

Without observation, these issues remains a black box. When we cannot observe how the Agent is performing, we cannot talk about how to optimize it. Therefore, we must first completely record the reasoning path (Trajectory) of the Agent to form an analyzable, traceable data foundation before we can discuss evaluation and optimization.

2. What makes Agent application observation different?

In the previous section, we discussed Agent applications, where the object of observation changes, and introduced a new term: Trajectory (the reasoning path of the Agent).

From receiving a task to completing it, the thinking steps, the tools called, the knowledge retrieved, and the decisions made by the Agent form the Trajectory. This trajectory is not pre-orchestrated but is autonomously determined by the Agent at runtime.

Consider this analogy: Traditional app observability is like debugging a vending machine: inserting coins, selecting products, and yield predictable outputs. The problems are deterministic, root causes can be located, and fixes can be verified. Agent observability is like analyzing a chess game. Even against the same opponent, two games unfold differently. You can’t blame a single move without understanding the strategic context established by previous moves.

Observability Dimensions Have Shifted.

Traditional observation focuses on standardized dimensions, such as latency, error rates, throughput, and resource utilization. Agent observation needs to focus on more diverse dimensions, including at least: the logic of reasoning steps (whether the Agent's planning is correct), the accuracy of tool calls (whether the right tool was called with the correct parameters), the relevance of retrieved context (whether retrieved documents are actually useful), the quality of model outputs (whether answers are accurate and hallucination-free), as well as token consumption and cost-effectiveness. All of these will serve as evaluators.

Alibaba Cloud AgentLoop features a variety of built-in, out-of-the-box evaluators

Among these dimensions, some can be directly measured (such as token count and latency), while others require more complex evaluation methods (such as output quality and reasoning rationality). For instance, Alibaba Cloud AgentLoop evaluates by introducing evaluation Agents, which is known as Agent as a Judge.

The granularity of observability data has also changed.

The granularity of traditional distributed tracing is typically at the application, service, and interface level. The granularity of Agent observation is a Session ID, Trace ID, and Request ID levels. Each session contains multiple end-to-end calls, each end-to-end call contains multiple requests, and each request involves multiple environmental variables, including knowledge bases, tools, context, memory, etc. To organize this fine-grained data into a meaningful observation view, a completely new data modeling method is required. Leveraging UModel, AgentLoop automatically maps dependencies among upstream and downstream entity topological relationships such as Agents, Tools, and Models, constructing a full-stack Agent Ontology to visualize to recreate the complete Agent reasoning path (Trajectory).

3. Are LLM optimization and Agent optimization the same methodology?

LLM optimization follows a mature methodology: preparing training data, fine-tuning or RLHF, and evaluating results on benchmarks. This methodology assumes that model capability is the sole variable determining system performance. However, as "Agent = Model + Harness" gains widespread acceptance, this assumption breaks down in Agent scenarios. (Previously, in "Constraint Infrastructure Growing on Alibaba Cloud," we shared the composition of Harness. Learn more.

The ultimate performance of an Agent is the cumulative result of multiple factors such as model capability, prompt design, tool definitions, knowledge base quality, orchestration logic, and memory strategies. The same model under different Agent configurations can result in vastly different performances. Conversely, a well-designed Agent system, even when using a slightly weaker model, can perform exceptionally well in actual business scenarios.

The behavior of an Agent exhibits three main characteristics:

First, Agent tasks are typically open-ended, with no single correct answer. If a customer service Agent answers a user's question with accurate information, appropriate expression, and successfully resolves the issue, it is considered qualified. But what is the "standard answer"? This is difficult to define in advance.
Second, the execution path of an Agent is dynamic, and the same task can yield completely different reasoning trajectories. You cannot cover all possible path combinations with a fixed set of test cases.
Third, Agent performance is highly context-dependent. A configuration that works well in Scenario A may fail completely in Scenario B. This means evaluation must be conducted in specific business scenarios and cannot be assessed separate from the context.

These differences dictate that Agent optimization requires an independent methodology rather than simply copying the benchmark ideas of model optimization. Alibaba Cloud AgentLoop proposes a methodology better suited for Agent evaluation: the MVP closed-loop, which drives the continuous evolution of Agents (to be discussed in detail in Chapter 5 of this article).

AgentLoop Product Manager Yahai sharing the MVP closed-loop at the 2026 China AI Agent Conference

4. What are the challenges of Agent optimization?

In the first three sections, we have already mentioned some challenges of Agent optimization. Let's summarize them here.

The first level: Where does the data come from?

Optimization requires data, and we need to know in which scenarios the Agent performs well, in which scenarios it performs poorly, and where the specific discrepancies lie. However, in production environments, this data does not automatically appear. Traditional APM data (latency, error rate) can only tell you if the system is experiencing issues, not how good the Agent's response quality is. To obtain Agent behavioral data, a complete collection and structured storage of the reasoning process must be conducted, which is technically far from simple.

The volume of Agent reasoning trajectory data is large, its structure is complex, and its semantics are rich. A typical Agent task execution can generate tens of KB or even MBs of trajectory data, including model inputs and outputs, tool calling parameters and returns, retrieval results, and intermediate reasoning steps. How to efficiently collect, store, and retrieve this data, and form high-quality evaluation datasets to increase the level of automation, is also a technical challenge.

The second level: How to evaluate?

Even with data, evaluating Agent performance remains an open problem. Currently, common practices in the industry include: manual evaluation (high cost, unsustainable), rules-based evaluation (limited coverage), and model evaluation (LLM-as-a-Judge, which is not suitable for Agents with Harness engineering).

To evaluate the quality of such behavioral sequences, we need an evaluator that likewise possesses Agent capabilities: one that can understand task goals, analyze the rationality of reasoning paths, judge whether tool calls are appropriate, and assess whether the final outcome truly resolves the user's issue. This is the core concept of Agent-as-a-Judge: using an Agent to evaluate another Agent.

The third level: How to modify?

After identifying and evaluating issues, we must ultimately return to how to make improvements. However, the parameter space of an Agent system is extremely vast: how to write prompts, define tools, organize the knowledge base, design coordination logic, and choose the model. Each dimension has massive room for adjustment, and complex coupling relationships exist among these dimensions. Adjusting prompts might affect the accuracy of tool calls, modifying the knowledge base might affect the completeness of answers, and changing models might affect the entire reasoning trajectory.

This high-dimensional, highly coupled optimization space makes it difficult for Agent optimization to rely on guesswork; it requires systematic experimental methods, including controlled variables, comparison experiments, quantitative evaluations, and iterative optimizations.

These three levels of challenges correspond to the three core modules of AgentLoop's product design: full-stack observation and auditing, evaluation and experimentation, and asset management and continuous optimization.

5. How does AgentLoop optimize intelligent agents?

AgentLoop is a one-stop platform for Agent self-evolution launched by Alibaba Cloud for enterprise-grade intelligent agents. Its positioning is very clear: it is neither an Agent development framework nor a model training platform, but an engineering platform that helps enterprises upgrade their Agents from "usable" to "exceptional."

From the perspective of product capability, AgentLoop achieves intelligent agent evolution through the five rings of MVP:

The First Ring: Agent Observation and Auditing. This is the starting point for all optimizations.

AgentLoop supports mainstream development frameworks such as Dify, LangChain/LangGraph, and AgentScope, as well as clients like OpenClaw, Hermes, Qoder, Claude Code, Codex, and Cursor. It uses non-intrusive data collection methods to obtain the Trace and Log data of Agents in production environments without modifying Agent code or embedding extra SDKs. The collected data is not just user input and final Agent output, but also includes detailed records of model calls, tool executions, knowledge retrievals, and token consumption.

Trajectory collected by Alibaba Cloud AgentLoop

The Second Ring: Agent Trajectory Analysis. With the raw data, the next step is to understand it.

AgentLoop uses ATIF (Agent Trajectory Interchange Format) as the standard format to conduct structured analysis on the Agent's reasoning trajectory. Trajectory analysis is not simply viewing logs, but a semantic interpretation of the Agent's reasoning path: in which stage did the Agent make a decision? What was the basis for that decision? Were the tool calling parameters reasonable? Were the retrieval hit documents relevant? The core engineering capability of this ring is the Pipeline data processing pipeline. Users can specify a dataset and a cleaning template, such as "Trace QA Q&A pair extraction," configure the data window and trigger conditions, and the Pipeline will automatically process trajectory data in batches, transforming unstructured reasoning trajectories into structured evaluation samples.

The AgentLoop Pipeline supports rolling execution on an hourly data window, processing thousands of trajectory data rows per execution, with an average duration in seconds and a success rate close to 100%. This means that the massive amount of trajectory data generated daily in production environments can be automatically cleaned, transformed, and stored via Pipelines, continuously supplying high-quality evaluation samples to baseline and test sets. Meanwhile, trajectory analysis also supports online continuous evaluation, which monitors the real-time performance of online Agents through intelligent sampling (without Ground Truth).

Pipeline capabilities of Alibaba Cloud AgentLoop

The Third Ring: Agent Performance Evaluation.

AgentLoop adopts the Agent-as-a-Judge mechanism and offers 20+ out-of-the-box evaluator Agents tested through engineering practice in mainstream business scenarios to perform in-depth judgments based on complete reasoning trajectories. Performance evaluation covers two scenarios: online evaluation (without GT) to perform continuous quality monitoring on Agents running in production environments, discovering anomalies and low-score samples through intelligent sampling; and experimental evaluation (with GT) to score Agents quantitatively in a controlled environment using annotated test sets. Low-score samples automatically flow back to the baseline set, completing the data loop. Evaluation dimensions cover task completion, reasoning path rationality, tool calling success rate, retrieval relevance, hallucination detection, etc.

Alibaba Cloud AgentLoop has pre-installed 20+ out-of-the-box evaluators

The Fourth Ring: Agent Experimental Backtesting. Once problems are evaluated and located and a new version is optimized, the improvements must be verified.

AgentLoop's experimental backtesting module provides two testing modes: CI/CD baseline testing, which automatically runs regression tests whenever Agent assets change to ensure new changes do not introduce regression; and scenario-based testing, which creates test cases for specific business scenarios to verify how the Agent performs in critical situations. Experimental samples are input from the baseline set, and experimental results are output as multi-dimensional metric analysis reports, supporting comparison across multiple versions.

The Fifth Ring: Agent Continuous Optimization. This is the end of the flywheel and the starting point for a new loop.

AgentLoop's optimization feature currently focuses on the optimization of Prompts and Skills for CLI and Claw-style intelligent agents, on runtime optimization of high-code ReAct intelligent agents such as AgentScope and LangChain, and on cost optimization for Coding intelligent agents.

AgentLoop provides two types of tuning paradigms. First, based on evaluation and experimental results, it performs multidimensional tuning on an Agent's Prompts, Skills, and model configurations. Second, it automatically extracts from the experience base. Alibaba Cloud has accumulated extensive evolutionary experience in mainstream scenarios like customer service Agents, Coding Agents, and Data Agents, and continues to co-create with corporate clients. These intelligent agent assets under FDE scenarios directly feed back into AgentLoop's experience database. Users can automatically extract successful patterns from high-quality Trajectories to build reusable experience snippets and dynamically inject them into the Agent's context.

We will continue to output best practices around AgentLoop's MVP five rings in the future, so stay tuned.

6. Evolution is not a silver bullet, but it is a practice Agent Builders won't want to miss

Models will hallucinate, reasoning will drift, and tools will be miscalled. These issues cannot be fully enumerated before deployment and can only be discovered and corrected continuously during runtime. Of course, evolution is not a cure-all; in the long run, evolution is essentially an engineering placeholder when model capabilities do not yet fully cover the requirements, much like Harness.

When models become strong enough, this layer of external scaffolding will eventually be internalized. But that day is still far off. Especially in non-generalized scenarios close to business operations—such as approval processes unique to each enterprise, special compliance requirements within every industry, and domain experiences accumulated by individual companies—these represent long-tail areas that general models cannot and will not enumerate. It is precisely in these areas that the engineering framework of Agent evolution has its most irreplaceable value: it allows enterprises to use their own production data to continuously train Agent or Skill capabilities that belong uniquely to them.

Models are advancing, but business scenarios continue to emerge in endlessly diverse layers. Before the gap between the two truly disappears, whoever runs the flywheel of execution, observation, and evolution first will secure the admission ticket to the next generation of enterprise software.

AgentLoop DingTalk User Group: https://qr.dingtalk.com/action/joingroup?code=v1,k1,Jsobu+cckwu0kOSNIm12ZGMnWvxi6vGZIRcWP42Jprs=&_dt_no_comment=1&origin=11

Community

What Is the Newly Released AgentLoop by Alibaba Cloud?

1. Agent Optimization: Why Must Observation Come First?

2. What makes Agent application observation different?

3. Are LLM optimization and Agent optimization the same methodology?

4. What are the challenges of Agent optimization?

5. How does AgentLoop optimize intelligent agents?

6. Evolution is not a silver bullet, but it is a practice Agent Builders won't want to miss

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Alibaba Cloud Model Studio

Managed Service for Prometheus

Qwen

Alibaba Cloud for Generative AI