By Ren Yi
Agentic SOC is a security operations product built around multi-agent collaboration. During its development, we gradually established a closed loop for product improvement comprising real-world issues, unified evaluation, change validation, and production feedback. As Agents took on more work, the team also began adjusting how it collaborated, enabling product, security, engineering, and testing to advance together toward the same customer outcome.
Drawing on this experience, this article explains how evaluation drives continuous product evolution, how to define the boundaries of human-machine collaboration, and how teams can reduce cross-functional handoffs and communication overhead by aligning goals, outcome criteria, and responsibilities—turning technical capabilities into tangible delivery results.
This article has been included in the open-source Alibaba AI Agent Handbook. For an enhanced reading experience, visit: https://aliyun.github.io/ai-agent-handbook/cases/chapter-25/agentic-soc-organization.html
Agentic SOC is a security operations product built around multi-agent collaboration. Faced with alerts scattered across multiple clouds and products, it first consolidates the information into security incidents that can be investigated and handled. Different Agents then divide the work of investigation, analysis, response, and containment, while also supporting threat hunting and routine security operations.
For users, seeing an analysis report is only part of the process. More important questions are: What happened? Where is the evidence? Which entities were affected? What action should be taken next? The report, timeline, and remediation recommendations must answer these questions together.
Figure 1: Agentic SOC consolidates alerts from multiple sources into security incidents, with multiple Agents collaborating on investigation and response.
Product improvement must therefore return to real-world tasks. What problems surfaced in production? Which capability fell short? Did the change resolve the problem? Did it disrupt scenarios that previously worked correctly? Each question requires verifiable evidence.
We gradually established a continuous iteration pipeline: production issues become failure samples, evaluation helps identify capability gaps, an Agent submits candidate changes, and validation and approval determine whether those changes take effect. New production outcomes then enter the next improvement cycle.
Whether this pipeline works depends on two foundations: a stable set of criteria for determining whether the product has improved, and someone who remains accountable for the final outcome.
In one investigation, A had attacked B, but the report reversed their roles. Swapping the attacker and victim affects the reader's understanding of the entire incident. Worse, if several modules make independent judgments, then even after the roles in the report are corrected, the timeline or remediation recommendations may still follow a different conclusion.
The investigation found that the three modules lacked a shared determination. Based on this root cause, the fix expanded from “correcting a passage in the report” into a structured redesign: generate unified fields for the attacker, victim, and determination result, then have the report, timeline, and remediation recommendations reuse those fields.
Figure 2: From error identification and root-cause analysis through the actual change, retesting, deployment, and continuous regression, a single defect becomes validation evidence for subsequent releases.
During retesting, we reran the original scenario to check whether A was identified as the attacker and B as the victim, while also verifying that the outputs of all three modules were consistent. Only then could we confirm that the change addressed the mechanism behind the error rather than merely correcting one piece of wording.
The role fields and retrieval logic subsequently entered the production release, and the corresponding regression case was added to the evaluation baseline. From then on, any change to the model, prompts, or rules had to keep using this sample to verify role determination and its consistency across modules.
One fix produced two outcomes: an effective product capability and a validation sample that could be run repeatedly. The latter meant the team no longer had to rely on someone remembering the issue every time, while also giving subsequent improvements a baseline for comparison.
This is what we mean by product self-evolution: real-world problems continuously enter the improvement process; changes take effect after evaluation and risk assessment; and lessons from failures become constraints for the next cycle. This requires traceable changes, reliable validation, and clear release boundaries.
Fixing one case is not enough to show that the release has improved overall. A change may resolve a role-identification issue while degrading capabilities in other scenarios. Candidate changes must therefore be compared against the same baseline.
We built SOCBench around security investigation tasks. At the stage documented in this presentation, it contained 211 real-world investigation tasks covering 16 incident categories, evaluating five investigation stages across nine dimensions. The question set, execution configuration, and scoring specifications were all version-controlled to make results reproducible and releases comparable.
Figure 3: Evaluation covers different incident types and investigation stages, with the question set, configuration, and scoring specifications managed under unified version control.
A security investigation is a long-horizon task, and checking only the final conclusion misses many problems. We also need to assess whether the environment was identified accurately, the entry point was confirmed, the main attack chain was reconstructed, the scope of impact was examined, and the final conclusion was supported by evidence.
For capability assessment, in addition to report completeness, we must examine log coverage, temporal accuracy, entry-point reconstruction, process-chain reconstruction, impact expansion, evidence quality, reasoning and severity classification, as well as efficiency and stability. This allows the evaluation to identify “which stage needs improvement” rather than merely producing an overall score.
The effectiveness of long-horizon tasks must also be understood in conjunction with query volume and execution time. More queries do not necessarily mean a more thorough investigation, and less time does not mean the task has been completed.
Figure 4: The left side compares capability profiles across releases; the right side examines query calls and stage scores in a long-horizon task.
In one long-horizon task presented in the talk, Agentic SOC-Latest scored 86, made 43 queries, and took 28 minutes. One comparison result scored 93, made 95 queries, and took 48 minutes. This case helped us examine the relationship among quality, query overhead, and time; it represents a single case and cannot be treated directly as the average performance across all tasks.
From a candidate change through release, we focus on whether the original issue passes retesting, whether any regression appears within the agreed evaluation scope, and whether the risk is acceptable. Beyond retesting the defect sample, independent test-set evaluation, replay of production tasks, and canary validation are also required. Changes that fail are returned for adjustment, while newly exposed failure samples continue to accumulate in the regression suite.
Unified evaluation therefore serves three purposes: locating weaknesses, validating improvements, and supporting continuous regression. Every change should leave behind results that can be explained, reviewed, and reused.
As Agents become capable of handling more steps, the way people participate must also change. Guiding every operation step by step can reduce people to intermediaries who dispatch tasks and shuttle results; granting complete freedom, however, cannot meet the requirements that security products impose on production changes.
Our approach is to define four types of boundaries before a task begins, allowing the Agent to complete analysis, modification, and validation within them.
Figure 5: People define goals, quality, permissions, and stop conditions in advance, and the Agent advances the task within the agreed scope.
The clearer these constraints are, the clearer the scope within which an Agent can advance autonomously. Human effort can then focus on goal setting, critical judgments, and risk handling. Our expected path of evolution is from assisted execution, to autonomous completion of single-step tasks, and ultimately to completing the self-evolution loop within defined boundaries.
Once the product began iterating continuously around task outcomes, another way of working within the organization became more conspicuous: product completed the requirements, security completed the rules, engineering completed the implementation, and testing completed acceptance. Each stage produced a deliverable, but the customer's problem was not considered solved until the entire chain had run its course.
At that point, speeding up execution in any one role could only reduce the total time so much. Goals were interpreted repeatedly during handoffs, different functions queued separately, acceptance was concentrated at the end, and problems could then travel back along the same route.
A schedule comparison for one requirement illustrates the difference. Under the sequential handoff model, request clarification, design, security criteria, back-end development, front-end development, and centralized acceptance proceeded one after another, totaling 51 days. The alternative first established 12 acceptance criteria, appointed one person accountable for the outcome throughout the process, and organized product experience, engineering delivery, and security effectiveness as parallel tasks, producing a 14-day cycle.
Figure 6: A 51-day sequential schedule versus a 14-day parallel cycle.
What merits attention in this comparison is the placement of acceptance and accountability. Criteria that had previously been assessed collectively only at the end were moved to the start of the task. Accountability, previously transferred stage by stage with each deliverable, remained with one Owner—the person accountable for the outcome—from goal confirmation through outcome validation.
Organizational collaboration under this model can be summarized in four requirements.
First, align on the customer outcome. Product, security, and engineering need a shared understanding of what the work is intended to improve and which real-world usage outcomes will ultimately determine completion.
Second, align on outcome criteria. Metrics and evidentiary standards are established in advance, preventing each function from completing its own checklist only to discover at the end that their goals do not align.
Third, maintain end-to-end accountability for the outcome. The accountable person continuously follows up on issues, dependencies, and validation results to drive the task toward closure. Parallel work may be undertaken by multiple people and multiple Agents, but someone must still be accountable for the result.
Finally, involve experts according to risk. High-risk and irreversible changes must be reviewed by specialists; other stages involve them as the task requires, reducing unnecessary waiting.
These requirements also explain why team consolidation and role integration became an important step. A shared goal and definition of done must be reflected in the everyday division of work.
In this initiative, role changes mainly took the form of broader responsibility: understanding the problem further upstream, following delivery further downstream, and using Agents to perform more execution work.
Product managers extended beyond PRDs and prototypes into implementation, delivering parts of the front end and runnable prototypes, and validating product judgments through runtime results. Security engineers expanded from a focus on security capabilities into product design and implementation, translating specialist experience into scenarios, decision criteria, and effectiveness validation.
Engineering expanded in two directions. First, engineers moved toward end-to-end full-stack delivery, reducing the waiting caused by separate front-end and back-end scheduling and integration. Second, they deepened their understanding of customers and the business, including the roadmap, value, and commercial impact behind each requirement.
Figure 7: Product, security, and engineering broaden their scope of responsibility; whether a person or an Agent performs a specific task depends on how clear the criteria are and how much risk is involved.
Here, we must distinguish between “being able to do more kinds of work” and “everyone taking on every kind of work.” Tasks with explicit criteria that can be evaluated and rolled back may be executed by an Agent and approved by a person. For cross-domain work with manageable risk, the person accountable for the outcome can use Agents to complete the closed loop. High-risk, highly ambiguous, or irreversible work must still be led by specialists.
Specialized expertise therefore remains important, but it is increasingly expressed through defining problems, assessing solutions, validating outcomes, and accepting accountability. Security engineers need to encode their experience into rules and code that systems can execute and inspect; product managers need to judge whether a runnable result solves the real problem; and engineering must balance implementation quality with production reliability.
All three roles also need to strengthen two common capabilities: understanding business and value, and orchestrating AI to perform sophisticated work. How a task is decomposed, which parts can run in parallel, what information must be provided, and which results require review all affect the quality of the final delivery.
Interim statistics in the presentation show that the average cycle time from requirement creation to completion fell from 50.9 days to 13.8 days—a reduction of approximately 72.9% based on those two figures—while throughput within the measurement window rose from 63.9% to 98.9%.
Figure 8: Interim results and directions for subsequent validation.
The next questions to validate are equally clear: Can the same mechanism be replicated from individual projects across more modules? Can practices dependent on individual initiative be institutionalized as stable team capabilities? Can shared outcome criteria become part of routine evaluation rather than working only during a particular project?
For teams wishing to try a similar approach, the starting point can be one real-world problem: define the customer outcome and acceptance evidence to be delivered, appoint one person who remains accountable, run the process end to end, and then examine where the time was spent, which handoffs can be reduced, and which risks still require expert judgment.
As Agents continuously evolve around task outcomes, teams must also work together toward the same customer outcome. This is the direction we are pursuing through Agentic SOC—and one that still requires further validation.
LiteLLM Poisoned Twice in Half a Year: Where Should an LLM's 'Keychain' Be Kept?
820 posts | 61 followers
FollowAlibaba Cloud Native Community - November 6, 2026
Alibaba Cloud Native Community - July 14, 2026
Alibaba Cloud Community - July 17, 2026
Alibaba Cloud Native Community - April 22, 2026
Alibaba Cloud Native Community - April 3, 2026
Alibaba Cloud Native Community - April 10, 2026
820 posts | 61 followers
Follow
Qoder
From Assisted to Autonomous — Adapting to Every Programming Need
Learn More
Token Plan
Build more, spend less. One plan, every modality.
Learn More
QwenWork
QwenWork is dedicated to helping employees strengthen their professional competitiveness in the AI era and to enabling enterprises to improve organizational effectiveness.
Learn More
DeepSeek
DeepSeek-V4.1-Flash is the lightweight flagship of DeepSeek's new architecture family, packing 552B total MoE parameters to deliver flagship-surpassing intelligence across key benchmarks.
Learn MoreMore Posts by Alibaba Cloud Native Community
