AgentLoop builds its product capabilities around the data flywheel. By continuously collecting agent runtime data and converting it into reusable assets, AgentLoop drives the iterative optimization of agents.
Data flywheel
The data flywheel is the core operating mechanism of AgentLoop and the key to understanding the product's value.
When an agent runs in production, it continuously generates traces, conversation records, and log data. A pipeline cleans this raw data into trajectories that capture the complete process and status metrics, including user input/output, model inference, tool calls, and retrieval augmentation. Deep processing of trajectories produces two types of core data assets: datasets and experiences. These data assets in turn drive agent evaluation and experimentation, unsafe-behavior detection, prompt and skill tuning, and dynamic context optimization, improving the performance of production agents—which then generates higher-quality runtime data, forming a positive reinforcement loop.
The key to the flywheel is this: data is collected once to generate complete trajectories, which are then reused across the four scenarios of observability, audit, evaluation, and optimization, with their value growing continuously over time.
AgentSpace
An AgentSpace is the top-level workspace for organizing and managing resources in AgentLoop.
An AgentSpace defines the complete resource boundary for a team, business line, or standalone project, containing all resources such as Agent Applications, datasets, evaluation tasks, experiment plans, memory stores, and experience stores. AgentSpace provides resource isolation, centralized management, and access control, making it the fundamental unit for enterprise-grade multi-tenant governance.
A typical approach is to create a separate AgentSpace for each business domain (such as customer service, operations, or development) to achieve clean isolation of resources and permissions.
Agent
An agent is an autonomous entity capable of independent decision-making and multi-step execution. Unlike traditional LLM applications that only complete a single input-output exchange, an agent can plan based on a goal, autonomously select and call tools, maintain state across multiple steps, and dynamically adjust its strategy to ultimately complete complex tasks.
In the context of AgentLoop, "agent" is a general concept that refers to any autonomous entity with the execution characteristics described above—regardless of the framework it is built on or the platform it runs on.
Agent Application
An Agent Application is an agent instance connected to AgentLoop for observability, evaluation, and optimization. It can be a customer service bot, an operations assistant, a coding assistant, an enterprise copilot, or a custom-built agent system.
Difference between an agent and an Agent Application: An agent is a business concept that describes an AI system with autonomous execution capabilities; an Agent Application is a resource concept that refers to a concrete instance onboarded to AgentLoop. A single Agent Application may contain one or more collaborating agents, but AgentLoop treats it as one unified unit for observation and management.
After onboarding, AgentLoop continuously collects data such as traces and audit logs generated while the Agent Application runs, helping developers understand how the agent performs in real environments and driving subsequent evaluation and optimization workflows.
Trajectory
A trajectory records the complete execution path of a single agent request from receipt to completion. A trajectory includes all steps—user input, model inference, tool calls, retrieval augmentation, memory reads, decision branches, and the final response—along with the latency, token consumption, and status information of each step.
The trajectory is the fundamental data unit of the AgentLoop data flywheel: observability relies on it to reconstruct execution paths and locate issues; evaluation uses it to judge output quality and the reasonableness of the trajectory; audit uses it to identify unsafe-behavior attack chains; and experiences are also extracted and distilled from it.
Dataset
A dataset is the core carrier through which AgentLoop manages agent runtime data, used to accumulate high-quality, structured data assets that are queryable, evaluable, and available for experimentation.
You can accumulate information such as conversation records, trajectories, and expert annotations into a dataset. Each dataset supports custom schemas (with field types including text, long, double, and json), version management, full-text retrieval, semantic search, and SQL analysis. Datasets play multiple roles in the product: as the input data source for evaluation tasks, the test set for experiment comparisons, the regression baseline set for CI/CD gates, and the data foundation for continuous optimization.
Datasets are automatically accumulated and updated through pipelines as the business runs, with update frequencies configurable as real-time, daily, weekly, or manual. The platform will also provide a dataset marketplace with preset templates for various scenarios to help you quickly build evaluation baselines.
Pipeline
A pipeline is the automated data-processing engine of AgentLoop, responsible for converting raw runtime data into trajectories or high-quality datasets.
Pipelines support multi-step orchestration. A typical processing chain is: data source ingestion (trace data or SLS logs) → data reduction (filtering, deduplication, sampling) → feature extraction (step rounds, tool-call sets, and so on) → AI data processing (topics, intents, annotations, and so on) → writing to the target data store (a dataset or log store). The entire process runs automatically and continuously, saving more than 90% of manual data-processing costs.
Evaluation
Evaluation is used to quantify the quality of agent outputs and execution processes, turning the subjective question of "is it good?" into objective metrics that are trackable and comparable.
AgentLoop supports multi-dimensional evaluation of model outputs, RAG retrieval results, tool calls, and agent execution trajectories. Evaluators include preset evaluators (covering general purpose, RAG, tool use, agent trajectories, and more) and custom evaluators. Evaluation methods include Agent-as-a-Judge, LLM-as-a-Judge, and human feedback. Evaluation data sources include trajectories, traces, logs, and datasets. Evaluation tasks support two run strategies: continuous evaluation based on new data (for real-time monitoring of quality changes) and batch evaluation based on historical data (for offline regression validation).
Evaluation results can be used to discover bad cases, monitor production quality trends, verify whether version changes introduce regressions, and integrate with CI/CD to implement automated quality gates.
Experiment
Experiments are used to compare the performance differences of different configurations under controlled conditions, helping teams validate the effect of changes before release.
When you create an experiment plan, you select a target dataset and a combination of evaluators, define multiple sets of experiment variables (agent instances, model versions, prompt versions, parameter configurations, tool configurations, and so on), and run them in batches to compare each group's performance across dimensions such as accuracy, quality scores, latency, token consumption, and cost. Experiment results provide statistical significance analysis to help teams make data-driven decisions and reduce the production risks introduced by prompt adjustments, model upgrades, and agent version iterations.
Playground
The Playground is an interactive experimentation environment for rapid debugging and validation.
In the Playground, you can instantly adjust experiment objects such as agents, prompts, model parameters, and input samples, view agent and model outputs in real time, and attach evaluators to compare evaluation results. It is suitable for single-instance or multi-version agent debugging, rapid prompt experimentation, performance troubleshooting, and feasibility validation before formally creating an experiment plan. The Playground lowers the barrier to experimentation, letting developers iterate on ideas quickly without configuring a complete experiment workflow.
Agent assets
Agent assets are the core manageable units that make up an agent's behavioral logic, mainly including prompts and skills.
A prompt defines the agent's role setup, reasoning instructions, and output specifications; a skill encapsulates the tool-calling logic and process orchestration the agent uses to complete a specific task. In real-world engineering, an agent's performance largely depends on the correctness and consistency of the behavioral logic in these assets.
AgentLoop provides centralized hosting, version control, and change auditing for agent assets: every modification is traceable, its effect can be quantified and compared through associated evaluations, and changes to a prompt or skill can automatically trigger evaluation validation to ensure that the change does not introduce quality regression. Teams can manage agent behavior the way they manage code: changes are recorded, effects are measured, and releases are gated.
Memory
Memory is the long-term context capability that AgentLoop provides for agents, used to store user preferences, important facts, and reusable information. In subsequent interactions, an application retrieves relevant memories and adds them to the context of the current request, helping the agent deliver more coherent and personalized service.
A memory store is the resource that manages these memories. An application uses the API key bound to the memory store to read and write memories through the mem0-compatible interface, and can also retrieve long-term memories in the console. For more information, see Memory Store overview.
Experience
An experience is reusable operational knowledge automatically extracted from agent execution trajectories, recording effective handling methods, decision paths, success patterns, or failure lessons for a certain class of tasks.
Unlike memory, which focuses on "remembering user and environment information," experience focuses on "how the agent itself can better complete tasks." For example, the best sequence of steps for a certain type of troubleshooting, the correct parameter combination for a certain type of tool call, or an effective fix strategy for a certain type of bad case can all be automatically distilled into experience fragments.
Experiences support organization-level sharing: multiple Agent Applications within the same enterprise can share the same experience store, allowing the successful experiences of a single agent to be reused across applications and accelerating overall capability improvement.