Well-Architected Framework:Stability pillar

Rationale

Any single model service can experience unavailability or performance degradation. By configuring a three-tier fallback strategy—primary model, backup model, and cache—you can ensure business continuity when the primary service fails or degrades.

Implementation

Design a three-tier fallback chain: Primary model → Backup model → Cache/Rule-based system. Under normal conditions, the primary model handles 100% of the traffic. When the primary model experiences issues, such as timeouts, an error rate exceeding a threshold, or a drop in quality score, traffic automatically switches to the backup model. The backup model is typically a lower-cost model that can still support basic business functions, such as downgrading from a large-parameter model to a medium-parameter one. If the backup model also becomes unavailable, the system degrades to a cache that returns pre-calculated answers to common questions or a rule-based system that generates standardized responses from templates.

Use the circuit breaker pattern to manage model switching. A circuit breaker maintains three states: Closed (use the primary model), Open (switch to the backup model), and Half-Open (probe the primary model for recovery). The circuit breaker trips to the Open state when conditions are met, such as N consecutive failed requests, an error rate exceeding a threshold in a sliding window, or average latency surpassing the SLO limit. After a set period, the circuit breaker enters the Half-Open state and allows a small number of requests through to test whether the primary model has recovered.

Ensure the output formats of the primary and backup models are compatible. If the primary model is a GPT-4 class model and the backup is a GPT-3.5 class model, their output schemas must be consistent so that business logic does not need to be adjusted to parse the output when a switch occurs.

Anti-patterns: relying on a single model service with no fallback configuration, which leads to a complete business outage if the service fails; or configuring a fallback without compatibility testing, causing downstream parsing errors due to schema mismatches when a switch occurs.

Alibaba Cloud practice: AI Gateway allows you to configure multiple model endpoints and implement primary/backup switching with custom routing logic. PAI-EAS can deploy multiple model service instances, and the health check mechanism of Application Load Balancer (ALB) can automatically remove unhealthy backend instances.

Rationale

API quotas and rate limiting are inherent constraints of cloud AI services. Poorly managed traffic patterns can lead to critical requests being rejected due to quota exhaustion, while non-critical requests consume valuable quota. Build client-side quota management and traffic monitoring mechanisms to handle upstream rate limiting gracefully.

Implementation

Implement client-side rate limiting using the token bucket algorithm. Based on a known API quota (e.g., 100K tokens per minute), maintain a token bucket on the client side. Each request acquires tokens based on its estimated consumption. If the bucket is empty, requests are placed in a waiting queue instead of failing immediately. This approach transforms the API's hard rate limiting (rejection) into a soft client-side limit (queuing), providing a smoother user experience.

Use a backpressure mechanism. When the request queue backlog exceeds a threshold, send a "system busy" signal to the upstream caller, such as an end user or another service. The caller can then decide whether to wait, retry, or use a fallback. The backpressure signal should include an estimated waiting time to help the caller make an informed decision.

Set request priorities for different types of requests. Critical business requests, such as real-time customer service chats, should get priority access to the quota. Non-real-time requests, like batch data processing or report generation, can be executed during off-peak hours. A priority queue ensures that critical requests are not crowded out by non-critical ones when the quota is tight.

Warm up your quota. Before predictable traffic peaks, such as e-commerce promotions or marketing events, coordinate with your cloud provider to increase your quota or pre-deploy additional inference instances. Use historical traffic patterns to forecast demand and prepare resources before the peak arrives.

Anti-pattern: sending requests at an uncontrolled rate without awareness of upstream quota limits. This leads to a high number of rejections, and subsequent retries can further exacerbate congestion, creating a retry storm.

Alibaba Cloud practice: AI Gateway supports an API Rate Limiting Policy designed for large model services and high-concurrency scenarios. It provides dynamic control based on multiple dimensions, including token consumption, request count, and concurrency. You can configure rate limiting rules based on consumer identity, request headers, query parameters, cookies, client IP, or model name. It also offers global API-level rate limiting. This multi-dimensional approach precisely matches the resource consumption characteristics of large models, effectively preventing system overload, API abuse, and malicious calls while ensuring the stability of core business operations in complex scenarios.

Rationale

Deploying in a single availability zone exposes your service to the risk of a data center-level failure. Although such failures are rare, their blast radius is large and recovery can be lengthy. A cross-AZ deployment for AI inference services ensures business continuity even if one availability zone fails.

Implementation

Deploy your inference service in at least two availability zones within the same region. Each availability zone should have a complete, independent deployment of the inference service, including model loading and GPU resources, rather than a simple active-standby setup. Under normal conditions, both AZs handle traffic. If one AZ fails, the other takes over the entire load.

Use a load balancer for health checks and traffic routing. Health checks must validate more than just process liveness (TCP/HTTP probes). They should also include model availability verification by periodically sending a lightweight inference request to confirm the model is loaded correctly and can respond. A traditional TCP health check cannot detect a problem where the process is running but the model has failed to load, for example, due to a GPU driver issue.

Plan cross-AZ capacity with redundancy: each availability zone must be able to support the full traffic load independently (an N+1 capacity model), rather than assuming all AZs are always available. Otherwise, if one AZ fails, the remaining AZ will be overloaded, leading to a cascading failure.

Anti-patterns: deploying all inference instances in a single availability zone; or deploying across AZs but with insufficient capacity in any single AZ to handle the full load.

Alibaba Cloud practice: PAI-EAS supports cross-AZ deployment for inference services; we recommend deploying in at least two availability zones. Model Studio supports PrivateLink, which enables multi-AZ network access to ensure cross-AZ disaster recovery. Combine this with Application Load Balancer (ALB) for traffic distribution and health check-based routing. ALB supports custom health check paths, which can be configured to target a model service's inference probe endpoint.

Rationale

When an AI's output must be consumed programmatically by downstream systems, such as a JSON-formatted API response or a structured decision result, the reliability of the output format directly impacts system stability. Large language models can produce structured content that is incomplete, has missing fields, or contains incorrect data types, causing downstream parsing to fail.

Implementation

Use a JSON Schema to define the expected output format, including required fields, data types, enum values, and nested structures. When calling the model, use Function Calling or a response format parameter to specify the output schema and constrain the model to generate output that complies with the schema.

Perform schema validation on the model's output before sending it downstream. If validation fails, initiate a retry flow. Send the validation error back to the model with more explicit format requirements and regenerate the output. Set a retry limit, typically 2-3 attempts. If retries fail, execute fallback logic, such as returning a default value or notifying the user that the request cannot be processed.

Handle streaming outputs carefully. A streaming generation can be interrupted midway due to token limits or network issues, resulting in an incomplete JSON output. Validate the completeness of the streaming output after it is fully received. Treat incomplete outputs as failures and trigger a retry.

For critical business scenarios, consider using constrained decoding techniques. This ensures that the generated output strictly adheres to a specified grammar, such as JSON syntax or specific enum values, eliminating format errors at the source.

Anti-pattern: parsing the model's output directly using string concatenation or regular expressions without format validation. A minor change in the output format, like an extra space or a different field order, can cause the parser to crash.

Alibaba Cloud practice: The Function Calling feature in Model Studio allows you to define the return format with a schema. The DashScope API supports the response_format parameter to specify JSON output. In your application code, you can implement a validation-and-retry loop using a JSON Schema validation library.

Rationale

For scenarios where the output directly influences high-impact decisions, such as risk scoring, compliance judgments, or medical assistance recommendations, a single model inference is not reliable enough. By performing multiple inferences and comparing the results for consistency, you can identify cases where the model's uncertainty is high and avoid making critical decisions based on a single, potentially unstable output.

Implementation

Use self-consistency voting: run N inferences for the same input (N is usually 3 to 5) and compare the results. If a majority of the results are consistent, use the majority result. If the results are diverse with no clear majority, flag the output as "model uncertain" and route it to a human for a decision or request more information from the user.

Use cross-model verification. Run inference on the same input using two different models and compare their results. Different models have different training data and bias patterns, so a consistent result from both is more trustworthy. If the results are inconsistent, trigger a manual review or use a third model as a tie-breaker.

Create a consistency score. Quantify consistency as a score (for example, a score of 0.8 if four out of five inferences are consistent). Set different consistency thresholds for different business scenarios. Results below the threshold should not be executed automatically but should instead include an uncertainty disclaimer or be escalated.

Control costs. Multiple inferences increase the cost. You should enable self-consistency checks only for critical decision-making scenarios. For low-risk use cases like information queries or casual chat, a single inference is sufficient. Control the validation strategy through business-level tiering.

Anti-patterns: enabling multi-inference checks for all requests indiscriminately, which leads to uncontrollable costs; or using only a single inference for a critical decision and executing it directly, which creates unacceptable risk.

Alibaba Cloud practice: AI Gateway allows you to configure multiple model endpoints. You can use Log Service (SLS) to record the raw results of each inference for post-hoc consistency analysis and model quality assessment.

Rationale

Models are sensitive to prompt wording—different phrasings of the same semantic meaning can lead to outputs of varying quality. When an inference fails (due to poor quality or incorrect format), retrying with an adjusted prompt can often produce a better result. This approach leverages the model's sensitivity to prompts.

Implementation

When to trigger variant retries: trigger this only when output validation fails (due to format errors, incomplete content, or a failed safety check), not proactively for all requests. Use the primary prompt for the first attempt and switch to a predefined variant for retries.

Design variant strategies: predefine 2-3 semantically equivalent but differently phrased prompt variants. Variations can include changes in instruction wording ("Please generate" vs. "Please output"), format emphasis (adding format examples or using XML tags to wrap the desired format), chain-of-thought guidance (adding "Let's think step by step"), or reordering constraints. Design variants based on empirical evidence of what works best for a specific model.

Set a cost budget. For each request, set a maximum number of retries and a maximum token consumption budget. If the cumulative token consumption approaches the budget during retries, stop and execute fallback logic. This prevents runaway costs from infinite retries.

Retry interval: you do not need a wait time between variant retries (unlike network retries), because the goal is not to wait for service recovery but to try a different phrasing. However, if the failure is caused by upstream rate limiting, you must follow a backoff strategy.

Anti-patterns: retrying indefinitely with the exact same prompt, which will likely produce the same failed result; or using prompt variants with significant semantic differences, which can cause the function of the request to shift.

Alibaba Cloud practice: Configure multiple prompt templates in your Model Studio application. The application code implements the logic for validation, variant switching, and retries. Use Application Real-Time Monitoring Service (ARMS) to monitor the retry rate and final success rate. A high retry rate can trigger an alert for prompt optimization.

Rationale

The knowledge base is the foundation of a RAG system's accuracy. Outdated content in a knowledge base is a gradual problem—the system will not suddenly crash, but its answers will become less and less accurate. Without proactive freshness monitoring, you might only discover the problem after a large number of users have received incorrect information.

Implementation

Define a freshness Service Level Objective (SLO) for each knowledge base, which is the maximum acceptable age of its content. Differentiate SLOs based on content type. For example, the SLO for pricing information could be 24 hours (must be updated daily), the SLO for legal policies could be 48 hours (must be ingested within two business days of a new regulation's release), and the SLO for product documentation could be 7 days.

Monitor knowledge base content updates. Record the last update time for each document and the modification time of its source document. Use a scheduled task to scan the knowledge base and check if any document has exceeded its SLO. Out-of-date documents should trigger an alert to the responsible owner for an update.

Detect source document changes. Periodically crawl or use webhooks to monitor the sources of your knowledge base content, such as official websites, internal wikis, or policy release sites. When a change is detected, automatically trigger a knowledge base update process or at least a manual review process.

Mark outdated content. For documents that have exceeded their SLO but have not yet been updated, add an "information may be outdated" tag to the retrieval results. This allows the model to include an uncertainty disclaimer in its generation. This prevents the system from presenting potentially outdated information with high confidence.

Anti-patterns: ingesting the knowledge base once and never maintaining it; or relying on human memory to decide what needs to be updated.

Alibaba Cloud practice: The data quality monitoring capabilities of DataWorks can be used to monitor the freshness of knowledge base content. Model Studio knowledge bases support API-driven document updates, which can be integrated with a CI/CD pipeline for automated updates. Log Service (SLS) can record knowledge base change events, and Application Real-Time Monitoring Service (ARMS) can be configured with freshness alert rules.

Rationale

The vector database is the retrieval engine of a RAG system. Its unavailability prevents the AI system from answering questions based on the knowledge base, forcing it to rely on the model's own knowledge, which may be inaccurate or outdated. A high-availability design for a vector database must balance data consistency and service continuity.

Implementation

Use a multi-replica architecture. Maintain at least two replicas of the vector database data, distributed across different availability zones. Read requests can be load-balanced across replicas, enabling read scaling and AZ-level disaster recovery. Replicate write operations synchronously or asynchronously to all replicas, depending on your consistency requirements.

Deploy across AZs. Distribute the vector database nodes across multiple availability zones in the same region. If one availability zone fails, the replicas in the other zones can continue to provide service. Use a load balancer with health checks for automatic failover.

Implement data backup and recovery. Regularly create full snapshot backups of the vector database for quick recovery. The snapshot frequency should be based on the data change rate—hourly snapshots for frequently updated knowledge bases and daily snapshots for stable ones. You should validate your snapshot recovery process and measure its duration.

Maintain index rebuilding capability. Retain the original data source for the vector database (the non-vectorized text data). This ensures that you can completely rebuild the index from the original data if the vector database suffers unrecoverable corruption. The rebuilding process should be automated and have a predictable timeline.

Anti-patterns: deploying the vector database as a single instance in a single availability zone with no replicas or backups; or having backups but never validating the recovery process.

Alibaba Cloud practice: Alibaba Cloud Vector Search supports multi-replica and cross-AZ deployments. You can use Log Service (SLS) to monitor the query latency and availability metrics of the vector database. For data backup, you can use Object Storage Service (OSS) as the storage target and run periodic snapshot tasks.

Rationale

The retrieval step in a RAG system can fail in several ways: the vector database may time out, the retrieval may yield no results if the knowledge base has no relevant content, or the retrieved results may be of low quality with all similarity scores below a threshold. You need to design clear degradation strategies for these situations rather than letting the model "improvise" without valid retrieval results.

Implementation

Degrade when the vector database is unavailable. If the vector database times out or returns an error, choose a degradation strategy based on the request type. For simple, common questions, switch to a pre-computed FAQ cache and return the answer directly. For complex questions, honestly inform the user that the "knowledge base is temporarily unavailable, so the answer may be incomplete" and generate a response based on the model's own knowledge while including a low-confidence disclaimer.

Handle empty retrieval results. When no relevant information is found in the knowledge base, clearly inform the user, for example, "No relevant information was found in the current knowledge base," instead of letting the model invent an answer. You can optionally provide suggestions, such as "You can try rephrasing your question with these keywords" or "This question may require contacting a human agent."

Handle low-quality retrieval results. If all retrieved results have similarity scores below a predefined threshold, judge the results as unreliable. When generating a response, explicitly state that "The following information is for reference only, as no highly matching knowledge was found," and mark the response with a high level of uncertainty.

Make degradation observable. Log and monitor all degradation events. A sudden spike in the degradation rate could indicate a problem with the knowledge base (such as a corrupted index or outdated data) or an issue with the vector database service, requiring timely investigation.

Anti-pattern: silently ignoring retrieval failures and letting the model generate a response without any retrieved context. The user cannot distinguish whether the response is supported by knowledge or is a model hallucination.

Alibaba Cloud practice: In the Model Studio knowledge base configuration, you can set a similarity score threshold for retrieval results. Implement the degradation logic in your application code. Log retrieval quality metrics, such as similarity scores and the number of returned items, to Log Service (SLS). Configure degradation rate alerts in Application Real-Time Monitoring Service (ARMS).

Rationale

Complex tasks executed by an agent can involve multiple steps and take several minutes or even hours to complete. A failure at any intermediate step should not require the entire task to be restarted from the beginning, especially if the successfully completed steps have external side effects, such as an email that has been sent or data that has been committed. Checkpointing ensures that a task can resume from the last successful checkpoint.

Implementation

Persist task state. Periodically save the execution state of the task to a persistent storage medium like a database or object storage. The state should include a list of completed steps, the output of each step, the currently executing step, and the input parameters for the next step. Persist the state immediately after each step is completed.

Design checkpoints. Set checkpoints between the steps of a task. Each checkpoint should record the complete execution state and intermediate results up to that point. When recovering from a failure, load the state from the most recent checkpoint and continue execution, skipping the steps that have already been successfully completed.

Ensure idempotency. When execution resumes, the step that failed might be re-executed. Therefore, each step must be designed to be idempotent, meaning that re-executing it does not produce additional side effects. For operations that are not naturally idempotent, such as sending an email, check an execution log to determine if the step has already been successfully completed to prevent duplicates.

Provide a manual recovery mechanism. Offer a UI or API that allows operators to manually trigger a task recovery. They should be able to choose to resume from a specific checkpoint or skip a failed step and continue with the subsequent steps. For scenarios that cannot be automatically recovered, such as those requiring human intervention to correct an intermediate result, support the injection of corrected data before resuming execution.

Anti-patterns: having a long-running task with no state persistence, forcing a full restart upon failure; or having checkpoints but with buggy recovery logic that causes completed steps to be re-executed, leading to data duplication.

Alibaba Cloud practice: Use Function Compute (FC) or Cloud Workflows for task orchestration and state persistence. Model Studio workflows support step-level execution state recording. Log the task execution trajectory to Log Service (SLS) for fault diagnosis and recovery validation.

Rationale

External tools that an agent relies on, such as APIs, databases, or third-party services, may become unavailable or slow to respond. Without proper controls, the agent might repeatedly retry a failed tool call, wasting resources and increasing user wait time. The circuit breaker pattern can promptly cut off calls to a failing tool, preventing resource waste and speeding up failure recovery.

Implementation

Configure a separate circuit breaker instance for each external tool. The circuit breaker should record the success, failure, and timeout status of tool calls and calculate the failure rate over a sliding window.

Set circuit breaker trigger conditions. Within a sliding window (e.g., the last 30 seconds), if the tool call failure rate exceeds a threshold (e.g., 50%) or the average latency surpasses the SLO (e.g., twice the timeout value), trip the circuit breaker to the Open state.

Define behavior during an open circuit. When the circuit breaker is Open, subsequent calls to that tool are not actually executed. Instead, they immediately return a predefined fallback result. The fallback result should be informative, telling the model that "the tool is currently unavailable," rather than returning a false success result. Upon receiving this information, the model should be able to adjust its strategy, such as skipping the step, using an alternative, or informing the user that the task cannot be completed.

Implement a recovery strategy. After a set period (e.g., 30 seconds), the circuit breaker should enter the Half-Open state and allow a small number of requests to test if the tool has recovered. If the test requests succeed, it returns to the Closed state and resumes normal calls. If they still fail, it remains Open.

Classify tool dependencies. Distinguish between "required tools" and "optional tools." If a required tool's circuit breaker opens, the entire task should enter a waiting or degraded state. If an optional tool's circuit breaker opens, that step can be skipped, and the subsequent steps can continue.

Anti-patterns: having no timeout, retry limit, or circuit breaker for tool calls, allowing a single failing tool to bog down the entire agent's response; or having a circuit breaker that is too coarse-grained, where a failure of one endpoint of a tool causes all calls to that tool to be blocked.

Alibaba Cloud practice: Application Real-Time Monitoring Service (ARMS) provides health monitoring for service calls. In your application code, you can use a circuit breaker component like Sentinel to implement tool-level trip strategies. Log the result and latency of each tool call to Log Service (SLS) and configure failure rate-based alert rules in ARMS.

Rationale

An agent’s "quality of service" cannot be measured solely by traditional availability metrics like 99.9% uptime. An agent can be continuously online (100% availability) but have a task completion rate of only 60% and an average completion time far exceeding user expectations. You need to define AI-specific Service Level Objectives (SLOs) to measure an agent's service quality comprehensively.

Implementation

Define core Service Level Indicators (SLIs), including at least the following:

Task Completion Rate: The percentage of accepted tasks that the agent successfully completes. You must clearly define "completion"—does it mean the model returned a result, or that the result passed a quality validation? We recommend the latter.

End-to-end Latency: The total time from when a user initiates a request to when they receive the final result. For multi-step tasks, record the total latency, not just the latency of individual steps. Set targets for P50, P90, and P99 percentiles.

Time to First Token: In streaming scenarios, this is the time a user waits before seeing the start of a response. It significantly affects the user's perception of responsiveness.

Output Quality Score: Quantify output quality through automated evaluation (such as LLM-as-a-Judge) or user feedback (such as thumbs-up/thumbs-down ratings). You can evaluate quality across different dimensions, including accuracy, completeness, and relevance.

Set SLO targets based on your SLIs. For example: Task Completion Rate >= 95%, P90 End-to-end Latency <= 30 seconds, and Output Quality Score >= 4.0/5.0. Differentiate SLO targets based on the business scenario. A customer service assistant has stricter latency requirements, while a data analysis task has stricter accuracy requirements.

Establish an error budget mechanism. When the SLO margin is healthy, you can make aggressive changes like updating a model or optimizing a prompt. When the SLO is close to its threshold, freeze changes and focus on improving stability.

Anti-pattern: relying only on traditional infrastructure metrics, such as CPU utilization, memory usage, and process liveness, to judge the health of an AI system. This makes it impossible to detect a state where the system is "online but useless."

Alibaba Cloud practice: Application Real-Time Monitoring Service (ARMS) supports the definition of custom business metrics for SLIs and SLO dashboards. Use Log Service (SLS) to collect and calculate metrics like task completion rate and latency percentiles. Integrate with DingTalk for alerting, so that on-call personnel are notified in real-time when an SLO deteriorates. With its Prometheus compatibility, ARMS allows you to incorporate AI SLIs into a unified observability system.