As AI becomes easier to access, the enterprise challenge is no longer simply how to build an agent. It is how to make agents complete tasks reliably, fit into business workflows, and deliver outcomes people can trust. Alibaba Cloud's open-source AI Agent Handbook explores the engineering and organizational work needed to close that gap.
The AI Agent Handbook has surpassed 3,000 stars on GitHub. Its focus is a practical question: as intelligent capabilities become more widely available, how can organizations turn them into reliable results?
Advances in models are making understanding, reasoning, planning, and generation available through model calls. Capabilities that once depended entirely on human time and expertise are becoming more accessible. But greater access to AI does not automatically translate into greater productivity.
An agent may generate a plan, call a tool, and carry out an operation. That does not mean a business can safely hand it an important task. In a real workflow, the system must also track progress, know which actions have already taken effect, decide when to continue or stop, and identify who should approve the final result.
Moving from answering questions to delivering outcomes requires more than a better model. It changes how applications are organized and where engineering responsibilities lie.
Initiated by Alibaba Cloud, the handbook brings together architecture, agent construction, runtime infrastructure, governance, optimization, and enterprise case studies. It shifts the discussion from building agents quickly to making them useful and dependable in production.
As models and agents evolve, enterprise attention is shifting toward three connected challenges.
The first is production engineering. Model-generated judgments are probabilistic, while business deliverables need explicit acceptance criteria. A compelling response is not enough. A system must keep work moving when information is incomplete, tools fail, or task requirements change. It must also provide verifiable evidence that the final result meets the requirements.
The second is scale. As adoption expands from a few experiments to many users, teams, and concurrent tasks, trade-offs between reliability, security, performance, and cost become harder to ignore. A few extra rounds of inference may be acceptable in a trial. Repeated across a large workload, the same overhead can change the economics of the service.
The third is organizational integration. An enterprise agent operates within departmental responsibilities, cross-system data flows, and real approval and authorization rules. Connecting several agents does not, by itself, create effective collaboration. Being able to perform an action does not mean being authorized to perform it.
The handbook's 2026 Agent Developer Survey offers a useful snapshot. It collected 1,906 valid responses, primarily from enterprise technology decision-makers, architects, engineers, and product professionals. Within this sample, 18% reported production deployment. Manual spot checks remained the most common evaluation method, at 55%, while only 6% reported automated evaluation based on execution trajectories.
These figures are not a census of the industry. They do, however, highlight two areas that deserve attention: moving agents into production and building the mechanisms to keep evaluating them after deployment.
That is why the white paper behind the handbook needed to evolve. Teams need more than ways to assemble an agent. They need to understand how to advance a task reliably, detect a failure promptly, and turn something useful to one person into a capability shared across an organization.
More capable AI expands what is possible. Mature engineering determines how much of that potential becomes a usable result.
The handbook's opening essay, “Putting Intelligence to Work: From Creating Intelligence to Using It Well,” makes a central argument: as cognitive capabilities become more abundant and affordable, the next challenge is to organize and constrain them so they can support reliable delivery at scale.
Two foundations matter here: multimodality and full-stack engineering.
Multimodality concerns how AI understands the information involved in a task. Business workflows span text, audio, images, video, code, and structured data. A service record may come from a voice conversation; an equipment condition may be captured in an image; the resulting action may take place in a business system. When a task depends on these inputs, the corresponding ability to interpret them is part of reliable execution.
Full-stack engineering concerns how AI can be supplied and operated sustainably. Models, chips, networks, storage, inference services, runtimes, and security systems together determine how long agents can operate, whether costs are affordable, and whether actions remain within controlled boundaries. Model capability cannot replace those supporting systems.
At the application level, the handbook places particular emphasis on the harness. In its broad sense, a harness is the engineering system outside the model that organizes, constrains, and supports model-driven task execution. It includes runtime support as well as orchestration and control.
Within that system, orchestration determines what information the model receives, which actions are allowed, when work should continue, and how completion is verified. The runtime manages agent instances, resources, and recovery. The model supplies intelligent capabilities; the harness organizes them into work that can keep moving.
The handbook's Building section is organized around tasks, information, and actions. Tasks must be broken down and advanced. Information must be selected and maintained. Actions must be constrained and verified. The Runtime section then covers runtimes, sandboxes, state storage, traffic management, asynchronous tasks, and multi-agent collaboration—the infrastructure that keeps these mechanisms operating.
Consider an illustrative refund scenario. An agent understands the customer's request and sends a valid refund instruction, but the external system times out before returning a result. The immediate question is not how to generate a better explanation. It is whether the refund has already happened. Retrying without checking could issue a duplicate refund; ending the task could leave the request unresolved.
The system needs to preserve task state, identify actions that have already taken effect, query the actual outcome, and escalate to a person when necessary. This may not require more sophisticated reasoning. It does require better execution and recovery mechanisms.
Tastien's intelligent operations case study provides a concrete example of the supporting engineering. Alerts from four monitoring systems feed into a unified event center, where they are grouped into incidents with separate identifiers and records of ownership, handling, and recovery. When an on-call engineer asks a question, the system supplies the model with alert timelines, call-chain snapshots, and historical root-cause records. Data-collection steps and diagnostic logic are packaged as reusable skills and scheduled around business operating hours.
The model can then analyze a specific incident using relevant context and defined procedures, rather than relying on a few disconnected fragments of information. Incident management, context delivery, and execution orchestration make this possible. Experienced engineers' troubleshooting knowledge becomes a capability the system can invoke, rather than remaining only in individual memory.
An agent architecture therefore needs to be designed around successful task delivery, not just a single model call. Context, tools, execution environments, and verification feedback must all serve that goal.
A smarter model improves capability. A system that knows when to act, when to stop, and how to verify the result builds the ability to deliver.
Once agents move from making suggestions to taking actions, a plausible-looking answer is no longer a sufficient basis for trust.
A tool call may access customer information. Code execution may change files. A business operation may incur a real charge. Organizations need to know what an agent did, what evidence supported its decisions, whether it had authorization, and how to investigate and address a problem.
The handbook's Governance section addresses four requirements: observable execution, bounded behavior, managed assets, and behavior that can be validated before deployment.
Observability is about more than checking server health. It must help reconstruct task execution: which models and tools were called, where something went wrong, and where time and cost were spent. Boundaries likewise require more than a safety instruction. Identity checks, authorization checks for individual actions, and additional approval for high-risk operations must be enforced by the system.
When prompts, skills, Model Context Protocol (MCP) services, and agents are shared across teams, their versions, dependencies, and change impact need to be managed. If a tool interface changes, teams should be able to determine which tasks are affected and which evaluations need to run again. Pre-deployment simulation and validation create opportunities to test behavior without causing real business consequences.
These mechanisms support security, but they also produce the evidence needed for optimization. Without a traceable execution record, a team may know that a task failed without knowing why.
The Optimization section consequently emphasizes attribution. First, rule out execution-environment problems. Next, examine whether the harness supplied sufficient information and reliable control. Only then assess whether the model itself has a capability gap. Missing retrieval results, ambiguous tool parameters, or lost state after an interruption should not automatically be treated as weaknesses in the model.
Execution records can be organized into task trajectories and then into reusable samples and curated “golden” evaluation datasets. Continuous evaluation identifies problems. Verified fixes can then be applied to context, skills, tools, or runtime mechanisms, allowing experience from one execution to improve the next.
Alibaba Cloud's STAROps evaluation and optimization practice turns evidence quality into an explicit gate. The team found a reference answer claiming an error rate of approximately 25%. Re-running queries across five consecutive time windows returned zero errors. The team then required each numerical claim in a reference answer to include an executable query, its time window, and the resource or service being measured. A program re-runs the query to verify the result. Numbers that cannot be verified cannot enter the reference answer.
For problems found in production, the team first preserves execution trajectories, call traces, and the incident time window. It then reconstructs the failure mechanism and reproduces it in the evaluation environment. Independent review and final human approval govern admission to the evaluation dataset. During optimization, the team distinguishes invalid evaluation data from actual agent defects; after a fix, it runs regression evaluation on data that remains valid.
Optimization is therefore not simply a matter of switching models and trying again. It turns problems found during execution into verifiable test cases, then uses those cases to determine whether a change actually helps.
For the same reason, self-improvement should not mean that a system can change itself without constraints. Proposed improvements still need validation, a defined scope of impact, and a controlled release process.
Trust is not an optional layer added to an intelligent system. It is a prerequisite for organizations to keep entrusting that system with work.
Technology creates value only when it contributes to the work a business needs to complete.
An agent that writes a report, analyzes data, or generates code may have completed only one step. Whether it can receive work from an upstream process, pass a usable result to a downstream system, find the right person when something goes wrong, and follow organizational rules determines whether it can become part of a business workflow.
This is why enterprise practice is a major part of the handbook. Its case studies cover software development, design engineering, operations, security, enterprise IT, and customer, sales, and business operations. They place architecture and engineering mechanisms back into specific working environments.
In the PolarDB-X case study, an agent starts from an alert or ticket and proceeds through diagnosis, reliable reproduction, code changes, and testing. Producing code is not the definition of completion. The reproduction test must trigger the original problem before the fix and pass afterward. Regression tests must also pass before the change is submitted for engineering review. Engineers review the reproduction evidence first and the code change second. The agent delivers not just a proposed change, but evidence that the change addresses the problem.
TOPSPORTS' enterprise agent practice connects capabilities for store operations, customer membership, products, and marketing. Agents identify customers who need follow-up, match preferences with available products, prepare communication suggestions, and assemble actionable daily tasks. Store managers confirm those tasks and personally follow up. Contact and transaction records feed into the next round of customer engagement. The system reduces cross-system analysis and preparation—not the manager's responsibility for business decisions and execution.
Scaling those capabilities across stores also requires clear boundaries around data and maintenance. TOPSPORTS emphasizes that reusing capabilities does not mean sharing all data: store managers can access only their stores' authorized data. Reusable skills need versions, owners, and evaluation standards. Business teams define acceptance criteria; IT teams handle integration, permissions, and reliability; AI teams handle agent engineering and optimization. A successful application becomes a sustainable organizational capability only when responsibilities and maintenance practices scale with it.
Different tasks require different degrees of autonomy. Drafting a document and performing a high-risk operation cannot follow the same authorization policy. A reversible change and an action that is difficult to undo cannot share the same acceptance criteria.
Adopting agents is not a competition to maximize autonomy. A fixed workflow may be appropriate when the process is well defined. An agent may need more decision-making freedom when the task requires exploration. For high-risk outcomes, people still need to authorize, review, and take responsibility when intervention is necessary.
The right choice is the simplest architecture sufficient to meet the business goal within acceptable cost and risk, not the architecture with the most components or the highest apparent sophistication. A shared vocabulary and clear responsibilities do not require every team to use the same technology stack. They help teams explain their choices and hand work over reliably.
The human role evolves as well. Models can generate more candidate answers. People must articulate business intent, define completion, decide which outcomes are acceptable, and take responsibility for those choices.
The value of an agent is not how many actions it performs on someone's behalf. It is whether those actions add up to a reliable, acceptable business outcome.
The handbook's preface asks a direct question: if AI can quickly generate large amounts of reasonably good text, why invest human effort in another white paper?
The answer is not that human-written content is inherently better. It is that, as the cost of producing text falls, the scarce sources of value shift elsewhere.
One is conceptual consistency. Terms such as harness, runtime, agent, and workflow can refer to different things in different teams. Without alignment, even a well-written proposal can lead to misunderstandings over responsibilities and delivery. A shared language is a foundation for collaboration.
Another is judgment. Public material can explain which techniques are available. It cannot automatically decide what a particular enterprise should prioritize, what it should defer, or whether an added layer of complexity is worth the cost. Those decisions need to be made within real business constraints.
A third is experience and lessons from failure. Where does a real system repeatedly break down? Which dependency is easy to overlook? Why does an approach work in a demonstration but become expensive when usage expands? These insights come from first-hand practice. Recording them gives readers not just information, but opportunities to avoid unnecessary trial and error.
This is the value of maintaining the handbook as an open-source project. It is not a document frozen after publication. It is an evolving body of knowledge that can be extended, challenged, and corrected. The project adds new practices weekly and updates existing material periodically. Developers and enterprise teams can contribute through Issues, Pull Requests, and sanitized case studies and postmortems.
Openness does not automatically make every conclusion correct. It does allow conclusions to be tested against more situations. New experience can lead to revised guidance, while recurring problems across teams can turn scattered knowledge into reusable methods.
The project also offers online reading and interactive challenges, and invites universities and other educational institutions to co-develop enterprise agent courses. Reading builds understanding; hands-on verification builds judgment; production experience supplies the next set of questions and answers.
“Putting intelligence to work” captures the handbook's central idea. Intelligent capabilities are the starting point. Reliably delivered task outcomes are the destination. Architecture, engineering, and organizational practices connect the two.
AI Agent Handbook aims to strengthen that connection: helping organizations do more than demonstrate intelligence, by organizing, constraining, verifying, and applying it to real work.
Intelligent capabilities are becoming easier to access. Reliable outcomes still need to be built with care. That is the next stage for agents—and a reason to keep reading, testing, and improving this open handbook together.
Explore the project on GitHub, read the English overview, or visit the online handbook.
The handbook's main chapters and case studies are currently in Chinese. The English README provides an overview of the existing content.
DeepSeek Harness: A Practical Guide to Comprehensive Observability
828 posts | 61 followers
FollowAlibaba Cloud Native Community - November 11, 2026
Justin See - March 19, 2026
Alibaba Cloud Native Community - October 28, 2026
Alibaba Cloud Native Community - October 26, 2026
ApsaraDB - June 26, 2026
Xi Ning Wang - August 30, 2018
828 posts | 61 followers
Follow
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn More
AgentCore
AgentCore unifies agent construction, governance, and execution capabilities so enterprises can deploy diverse agents with confidence.
Learn More
Qwen
Full-range, open-source, multimodal, and multi-functional
Learn More
Token Plan
Build more, spend less. One plan, every modality.
Learn MoreMore Posts by Alibaba Cloud Native Community
