From a Single Alert Card to One-Click RCA: How Tastien Built a Closed-Loop AIOps Practice Across 10,000+ Outlets

This article introduces how Tastien built a closed-loop AIOps system to unify alerts, enable AI-driven root cause analysis, and cut MTTR by 65% across its 10,000+ outlets.

Founded in 2012, Fuzhou Tastien Catering Management Co., Ltd. is a restaurant chain built around the "hand-rolled, freshly baked" Chinese burger. Drawing on traditional Chinese pastry craft, Tastien developed a distinctively textured hand-rolled, freshly baked burger bun and created an entirely new category: the Chinese burger. Strong R&D, consistently high product quality, and a convenient, attentive service experience have made Tastien one of China's leading restaurant brands. Today it operates more than 10,000 outlets across over 300 cities, with a large and loyal consumer following.

At a chain of this size, operational systems span dozens of service chains — outlet POS, supply chain, Customer Relations Management, online ordering, and more. In the AI era, as the business grows and iterates at speed, monitoring covers ever more ground and teams configure ever more monitoring and alerting — which brings problems of its own. Alerts arrive from many sources: CloudMonitor, ARMS, Simple Log Service, custom business alerts, and certificate alerts. Because that alert data sits in separate systems with different data formats, notification methods, and handling flows, the cost of configuring alerts, delivering notifications, responding and investigating, capturing knowledge, and closing the loop keeps climbing.

The Business Challenge: What Happens After the Alert Determines O&M Efficiency

In a multi-business cloud environment, companies rarely lack monitoring tools. What they lack is the ability to turn an alert into a management event that teams can work on together, track, and reuse. One fault can fire alert after alert until the important details are buried in the flood. On-call engineers switch between platforms and stitch the context together from experience. AI returns a diagnosis, but with no human feedback, no one can tell which analyses were right and which need work.

Traditional monitoring solved the problem of detecting faults. Alert management, and the collaboration that follows an alert, still runs into the same recurring problems:

Alerts scattered across sources:Alerts come from many places, and the data lives in four or five different systems. Each monitoring system needs its own contacts, notification methods, and delivery policies — expensive to maintain and easy to get wrong. Worse, alert content and status are expressed differently in each system, so on-call engineers struggle to read the alert status quickly, locate the problem, or notify the right owner in time.
Heavy duplicate noise: A single alert rule usually covers many instances, sometimes dozens. When one fault trips them all at once, a flood of duplicate Lark messages and phone calls follows within minutes — an alert storm. Duplicates add to the judgment and handling load on on-call engineers and drown out what actually matters, so critical alerts get missed.
Diagnosis that depends on individual experience:When an alert fires, the root cause may lie in a metric, a Simple Log Service log, an application trace, or an operational system. Investigating means switching back and forth across platforms and manually correlating monitoring, application, container, and log context — slow work that rests heavily on how much the on-call engineer happens to know.
No closed loop, no accumulated knowledge:Alert recovery is not the same as closing the loop. Without structured capture of root causes and solutions, past experience is hard to retrieve and reuse, and it never feeds back into the next diagnosis.

In our view, what happens after the alert is what decides O&M efficiency.Solving these problems calls for a single intelligent AIOps platform that ingests alerts from every source, standardizes them, and orchestrates them — delivering each alert to the right O&M engineer, helping them locate and fix problems fast, limiting the impact, and capturing what was learned so the alert loop keeps improving. The goal was never to add one more alert page, but to give every alert a complete lifecycle.

The Solution: One Alert Entry Point, an AI Feedback Loop, and a Digital Worker Matrix

To tackle scattered sources, duplicate noise, slow root cause work, and lost knowledge, Tastien built on Alibaba Cloud CloudMonitor 2.0 (CMS 2.0) and the full-stack AIOps platform STAROps. The Unified Alert Center in CMS 2.0 became the single entry point, a low-code workflow was added to the existing O&M platform as the alert orchestration layer, and together they close the loop:

CMS 2.0 multi-source alert ingestion → low-code workflow orchestration → automatic convergence → Lark collaboration → STAROps AI-assisted Analysis → root cause information added → AI rated → knowledge captured

1787740649568-dc9b264b-a8a4-41b2-a4c2-29

(1) Unified Alert Management in CloudMonitor 2.0 + Workflow Orchestration: One Entry Point for Every Alert Source via Event Integration and Subscription

Tastien routes ARMS, CMS 1.0, Simple Log Service alerts, Prometheus, and custom alerts into the Event Center through the event integration capability of CMS 2.0. A workflow then orchestrates the rules that normalize them into management events:

Webhook ingestion → source detection → field standardization → convergence decision → group routing and on-call scheduling → card generation or update → write to data store → tiered escalation and feedback.

The workflow picks a parse branch by alert source and transforms the title, level, resource, status, and other information into one common context. It then invokes alert convergence and group routing to decide whether to send a new card, update an existing one, or close it out on recovery, and it applies the matching notification and escalation policy for P1, P2, or P3. Claims, status updates, and root cause feedback raised on the Lark card flow onward through the same orchestration chain. This keeps the visual orchestration flexible while keeping long-lived state, consistency, and historical facts out of the workflow. Adding a new alert source usually means adding a parse branch and reusing the shared downstream stages, with no need to rebuild the whole handling chain.

(2) Lark Alert Cards + AI Root Cause Analysis: Claim, Ask, and Track End-to-End on One Card

The platform keeps real-time status, the collaboration entry point, and historical facts apart: real-time status drives alert convergence and card updates, Lark carries the live collaboration, and historical data preserves confirmed root causes, feedback, and handling records.
On-call engineers claim an alert, view its status, and reference the card to ask the AI what caused it — all from the card itself. The system pulls in the current alert context automatically, so no one has to restate the situation or search across platforms.
Typical scenarios delivered on the Lark alert card:
Ask about O&M alerts: describe the query in natural language, such as today's alert volume, unclaimed alerts, or top alert rules
Create Alert Rules: create cloud alerts and custom alert rules in natural language
Ask follow-up questions from an alert card: reference the card, skip the recap, and let the AI pull in the context
One-click root cause analysis: reference the card to ask the AI, which queries historical root causes and invokes STAROps root cause analysis to reach a conclusion, streaming the diagnosis in real time so the process stays transparent
Human-AI collaboration: interactive execution, with manual review for high-privilege commands

This turns the alert card from a notification into a handling entry point: who owns the problem, how far the analysis has progressed, and what the root cause turned out to be all accumulate around the same management event.

(3) The AI Feedback Loop: Every Diagnosis Captured as a Triplet of Context, Human-Confirmed Root Cause, and Rating

Whether AI root cause analysis is reliable cannot be settled by impressions on the product side; it takes data. Tastien embedded a minimal but complete feedback mechanism into the handling chain:

Step 1 · AI produces a diagnosis: after an on-call engineer references the alert card and asks a question, the AI produces a diagnosis based on the injected management event context and the historical root cause repository for that service.

Step 2 · The engineer fills in the actual root cause: once the management event has recovered (or once the root cause is confirmed during handling), the on-call engineer describes the actual root cause through the "Fill in Root Cause" entry on the card — free text, but it must state clearly what actually happened.

Step 3 · Rate AI accuracy: after the root cause is filled in, the system opens a rating entry with only two options — "AI is accurate" or "AI is inaccurate". If the engineer selects "AI is inaccurate", they can add an optional one-line note (for example, "AI attributed the root cause downstream; it was actually an upstream traffic problem").

As these records build up, they deliver two direct benefits:

A historical root cause repository: the next time a similar alert hits the same service, the AI reads that service's historical root cause records first in its Step 1 diagnosis, drawing on real past root causes rather than reasoning from current metrics alone. The more it is used, the closer its conclusions track the customer's actual business.
Diagnosis accuracy statistics and scenario-based improvement: the team can measure AI accuracy by service and alert_type, then target improvements at specific scenarios

(4) The STAROps Digital Worker Matrix: Skill Encapsulation, Mission Orchestration, and One-Click RCA

On top of the alert loop, Tastien built a second layer of capability with STAROps: encoding the troubleshooting steps held in senior SREs' heads as Agent capabilities the platform can schedule.

Skill Encapsulation — Turning Troubleshooting SOPs into Reusable Capability Units

Each Skill defines the complete troubleshooting path for one specific fault scenario: trigger conditions (which alert or metric combination should invoke the Skill), data collection steps (which metrics, logs, and traces to query, and in what order), judgment logic (which metric combination points to which root cause), and output format (a structured diagnosis plus recommended actions).

Mission Orchestration — Running Skills in Parallel to Form an Inspection Matrix

A single Skill handles diagnosis for a single scenario, but daily inspection needs dozens of Skills running in parallel under line-of-business and time-window policies. Using the Mission (long-running task) capability in STAROps, Tastien orchestrates multiple inspection Skills into continuously running digital workers. After each inspection round, the Agent outputs a structured inspection report: normal items collapsed, abnormal items highlighted with a diagnosis and recommended actions. The SRE's day shifts from "watching monitoring dashboards in shifts" to "reading the inspection report each morning and handling the items flagged in red".

Taken together, what Tastien built is not one more alert page. It turns alerts from messages into management events, AI output into data that can be rated, and a one-off incident response into a reusable O&M asset.

Only when every alert is grouped correctly, handled promptly, reviewed in full, and fed back into the next diagnosis does an O&M platform move from a notification tool to an engineering system that keeps learning.

Results:From an Engineering Loop to Measurable Gains

Significant reduction in alert noise:Through unified alert convergence and AI-based noise reduction in CloudMonitor 2.0, the alert compression rate reached 95%, and the invalid alert rate decreased by 15%.
Substantial improvement in incident resolution efficiency:With AI-assisted root cause analysis and end-to-end collaboration through Lark, the Mean Time to Recovery (MTTR) was reduced by 65%.
Effective accumulation of O&M expertise:A historical root cause library covering more than 40 business-specific scenarios has been established, enabling sustainable reuse of operational experience and continuous evolution of AI models.

Community

From a Single Alert Card to One-Click RCA: How Tastien Built a Closed-Loop AIOps Practice Across 10,000+ Outlets

The Business Challenge: What Happens After the Alert Determines O&M Efficiency

The Solution: One Alert Entry Point, an AI Feedback Loop, and a Digital Worker Matrix

(1) Unified Alert Management in CloudMonitor 2.0 + Workflow Orchestration: One Entry Point for Every Alert Source via Event Integration and Subscription

(2) Lark Alert Cards + AI Root Cause Analysis: Claim, Ask, and Track End-to-End on One Card

(3) The AI Feedback Loop: Every Diagnosis Captured as a Triplet of Context, Human-Confirmed Root Cause, and Rating

(4) The STAROps Digital Worker Matrix: Skill Encapsulation, Mission Orchestration, and One-Click RCA

Results:From an Engineering Loop to Measurable Gains

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Bastionhost

Managed Service for Grafana

CloudMonitor

Cloud-Native Applications Management Solution