In recent years, AI's ability to write code has advanced quickly. Hand a requirement to a Coding Agent, and it can already read code, change code, run tests, and even get a PR ready. But in real R&D we run into a question: code is being written faster, so why does one requirement still take so long to go from being raised to going live?
Around this question, we tried using AgentCore to organize 15 Agents with different divisions of labor into an R&D squad, and had them take part in the complete delivery process; we then used AgentLoop to analyze how they ran, and to keep improving the squad's performance. This article is about how this approach works, and the problems we ran into in practice.
When we first used Chatbots, we would usually ask one question and wait for the answer. They could help us organize knowledge, write copy, and produce summaries, but on complex problems the reliability of the answers was not stable enough.
Later, reasoning models represented by DeepSeek R1 and others began to handle complex problems through more thorough thinking. Further on, models gained tool-calling ability: they could read external information, run commands, and modify files, and so they started to genuinely take part in tasks.
With these capabilities in place, the next step is to let Agents work continuously: observe the environment, take an action, check the result, adjust the method when a problem appears, and then keep pushing forward.
In Coding scenarios this progress is already quite visible. The relevant benchmark metric has gone from about 33% to over 70%. At the same time, the tasks AI can take on keep getting more complex. If we measure by the time a human expert needs to finish a task, that time span roughly doubles every seven months.
For an R&D team, this brings a new possibility: we can start to consider letting Agents take part in longer workflows, and give them more of the tasks that have to be pushed forward continuously.
Take a rough estimate of the typical flow for a complex Feature on a team: from the requirement being raised to going live it takes about 20 working days, and writing the code is only two of them. Suppose a Coding Agent makes coding ten times faster — those two days shrink to 0.2 of a day. Set that inside the complete flow, though, and the delivery cycle only goes from 20 days to 18.2 days.
Where does the rest of the time go?
Requirements have to be discussed until they are clear, the Spec has to be confirmed again and again, and the approach has to be worked out against the existing architecture. After the code is written there is still testing, Review, cross-module integration, and release preparation. And if a problem turns up along the way, we may have to go back to an earlier step and adjust again. So once coding speeds up, the steps of confirming requirements, collaborating, verifying, and waiting become all the more visible.
What we want to do next is to let Agents take part in this work too, and join the whole process together. Context can be passed downstream along with the task, verification results can directly drive the next step, and people can put more of their energy into goals, architecture, and judging risk.
Our approach is to break the work carried by different roles in the R&D process into different Agents, and then organize them through AgentCore.
These 15 Agents form a capability pool that a Leader schedules according to what the task needs. Each Worker has its own Skill, Tool, and context, and also a clear responsibility.
| Role | Main Responsibilities | Key Deliverables |
|---|---|---|
| Researcher | Understand requirements and the codebase; build a problem evidence map | Relevant modules, constraints, and historical changes |
| Explorer | Find isomorphic implementations and form candidate solutions | Reference Anchor and solution comparison |
| Spec | Structure goals, boundaries, and acceptance criteria | Executable Spec |
| Code Worker | Implement candidate solutions in independent branches and sessions | Code changes |
| Test / Harness | Compile, test, and perform end-to-end verification | Reproducible test evidence |
| Judge / Verify | Independently select the best candidate, perform semantic acceptance, and adversarial review | Review conclusions and failure feedback |
| Deploy | Prepare the environment and execute release verification | Release and rollback evidence |
Once a task comes in, the Leader has to judge: what to do first, who to hand it to, how to check it when it is done, and where to go back to if a problem appears.
For example, if a test does not pass, go back to the original Code Session and keep fixing; if the direction of the approach turns out to be wrong, redo Research and Explore with the evidence of the failure; if repeated attempts still do not converge, trigger a Human Gate and ask a person to supply judgment.
What matters here is keeping the task state. What the previous Agent did, why it did it that way, and what was found during verification — all of that should still be usable by the Agents that follow.
A task can start from an Issue, and information can be added through Chat. What is finally delivered includes, besides the code and the PR, the test report, the review evidence, progress, blocker information, and the complete run trace.
Our goal is for this squad to be able to push from a requirement all the way through to a verified delivery, and then extend toward automated deployment and release.
Organizing the Agents and running one task through is only the first step. Once we were inside real R&D processes, we gradually met several kinds of persistent problems.
First, a requirement may be stated clearly, but architectural constraints are not necessarily carried through.
For instance, we want to extend the capability of an existing API; if we do not tell the Agent about that constraint, it may choose to add a new API instead. Looking only at the current task, that way may be easier to finish; set it in the system as a whole, and it does not meet the team's design requirements.
This kind of information usually lives in code, in documents, and in people's experience. It has to be sorted out, and kept updated as the business changes.
Second, real scenarios always bring problems nobody anticipated.
Traditional software can design tests around known functional paths. The inputs of an AI application are far more open: a user may phrase the question differently, or raise something never considered at design time.
For example, asking how to write a Python program inside a food-delivery app is already outside the original business scope. We can hardly cover all of these cases on day one; we can only keep discovering and adding while the system runs.
Third, the same problem may show up again and again.
Getting one task through does not mean the ones after it will go smoothly. Code, business rules, and runtime environments are all changing. If experience is not accumulated continuously, an Agent may fall into the same hole over and over, burn time and Tokens, and in the end still need a person to deal with it.
Fourth, fixing one problem may affect other tasks.
Change one Skill and the current case passes, while another case may regress. So after every optimization we need a comparatively stable Benchmark and regression check, to tell us whether overall performance really improved.
These problems also made us see that Agents need a mechanism to keep running, observing, and adjusting.
We use AgentCore as the running plane and AgentLoop as the optimization plane.
AgentCore is responsible for leading the squad to finish the task. AgentLoop records and analyzes how the task was finished: which models and tools were called, where it failed, how many rounds of rework happened, and whether the final result met the requirements.
These Trace, Session, and Trajectory data, run through Pipeline, Dataset, Evaluation, and Experiment, can become reproducible samples, comparable metrics, and experiments that can be verified again and again.
In this way we can start from one concrete problem, find the cause, adjust the Skill, the Tool, or the collaboration flow, run it once more, and see whether it really improved. Each new run produces new traces, which feed into the next round of tuning.
In this process we used three approaches, which by convention we call manual, semi-automatic, and automatic.
Manual is led by the domain expert. The expert first defines clearly what a good result is, which metrics to look at, and which Rubric to evaluate with. After the evaluation, Bad Cases are found and put into the dataset, and different approaches are compared through experiments.
Semi-automatic means the system supplies optimization suggestions: which tasks consumed too many Tokens, which steps were inefficient, which tool kept reporting errors. The system sorts these into a report, and the domain expert then makes targeted adjustments from it.
Automatic works by continuously accumulating and reusing experience through an experience repository. Following the RAC approach, the system extracts experience from the traces and injects it into Agents dynamically. Sedimenting and injecting experience can be done automatically: experts do not have to organize it item by item, and Agents can use the experience already accumulated on later tasks.
The three can be used together: experts define the goal, the system assists the analysis, and the experience repository takes care of continuous accumulation.
We mainly did two things: build an offline baseline from expert implementations, and continuously accumulate experience from real runs.
First, the offline experiments.
From the codebase we selected 20 human-expert Commits of good quality, then generated the corresponding requirements from that code to build a Benchmark.
Then we had the Agent Team complete these tasks and put its results side by side with the expert implementations: whether the approach had drifted, what problems there were in the code, whether verification was sufficient, and whether the division of labor among the Agents was right. Once we found problems, we adjusted the system and ran another round of experiments and regression tests. This way every adjustment has a comparatively fixed set of questions to be checked against, and it also becomes clearer where progress was made and where we still have to keep changing things.
Then, online experience accumulation.
Real tasks keep producing new traces. From them the experience repository can distill SOPs for this class of problem, good execution paths, and anti-patterns in tool calls and tool routing. For example, if a certain way of calling keeps failing, we record that pattern so the Agent can avoid it next time it meets something similar. If a class of problem has a good resolution path, we can also turn it into experience to reuse on later tasks.
Offline experiments let us check against the expert baseline, while online experience lets the system keep absorbing feedback from real tasks.
On how multiple Agents collaborate, we adopted a continuously iterating Loop: research first, then explore approaches, and after the implementation is done, verify and compare. If the acceptance conditions are not met, carry the feedback into the next round; once they are met, move on to delivery. This is similar to the observe–act–get-feedback idea in ReAct, except that we applied it to collaboration among several roles.
In practice, two things turned out to matter. One is to have implementation and verification carried out by different Agents, so that verification has an independent viewpoint. The other is to keep the necessary context across iterations, so the next round can move forward on the evidence and the attempts already made.
Coding tasks are highly open: the same requirement can have many ways to implement it. So we first bound the task scope and the acceptance conditions clearly. At present, samples from two scenarios have gone live.
One of these scenarios comes from the Trace to Trajectory service. That service converts Trace data into trajectory data, and in production it keeps meeting new parsing exceptions.
When an unparseable Bad Case appears, we write the relevant information into the logs, an alert triggers a new Issue, and then the R&D squad is started up to locate the problem, fix the code, and verify it — ending in a verified delivery.
For the final verification we picked 19 questions. About 10 of them produced implementations roughly equivalent to the human expert's solution, and another 5 came close to expert level. In these scoped, constrained scenarios the delivery cycle also went from about one month to under a week.
For us, what is most valuable in this practice is that requirements, implementation, verification, and feedback have started to be joined into one complete process. AgentCore lets several Agents collaborate like a team, and AgentLoop helps that team remember what it has done, accumulate experience from every delivery, and gradually do the tasks that come after it better.
804 posts | 61 followers
FollowAlibaba Cloud Native Community - October 28, 2026
Alibaba Cloud Native Community - July 14, 2026
Alibaba Cloud Native Community - November 6, 2026
Alibaba Cloud Community - August 12, 2026
Alibaba Cloud Native Community - October 12, 2026
Alibaba Clouder - April 20, 2020
804 posts | 61 followers
Follow
Token Plan
Build more, spend less. One plan, every modality.
Learn More
QwenWork
QwenWork is dedicated to helping employees strengthen their professional competitiveness in the AI era and to enabling enterprises to improve organizational effectiveness.
Learn More
AgentLoop
An Agent self-evolution platform designed to make your Agents progressively smarter and more attuned to users.
Learn More
DeepSeek
DeepSeek-V4.1-Flash is the lightweight flagship of DeepSeek's new architecture family, packing 552B total MoE parameters to deliver flagship-surpassing intelligence across key benchmarks.
Learn MoreMore Posts by Alibaba Cloud Native Community
