Well-Architected Framework:Performance pillar

Rationale

Model parameter count directly determines the required GPU memory capacity and compute power. Undersized hardware leads to frequent memory swapping—such as out-of-memory errors or offloading to CPU memory—and a sharp performance drop. Oversized hardware wastes resources and increases cost. The right hardware choice is the foundation of performance optimization.

Implementation

Establish a mapping between parameter count and hardware. A 7B-parameter model can run inference on a single A10 GPU (16 GB memory), making it suitable for lightweight apps and dev/test environments. A 14B-parameter model needs an A30 (32 GB memory) or equivalent, making it suitable for medium-complexity production workloads. A 72B-parameter model needs at least two A100-80GB GPUs deployed with tensor parallelism, making it suitable for mission-critical scenarios that require the highest quality. Choosing a batch size involves a trade-off: a larger batch size improves GPU utilization (from 30% to over 80%) but increases per-request queueing latency (from near-immediate processing to hundreds of milliseconds of waiting). We recommend setting a batch size upper bound based on your SLA. For real-time chat scenarios, limit the batch size to no more than 8–16. For offline batch jobs, you can relax it to 64–128.

Anti-pattern

Using one hardware configuration for all scenarios, which either overprovisions and wastes money, or underprovisions and hurts performance. Ignoring the impact of batch size by pursuing maximum throughput in real-time scenarios at the expense of latency. Provisioning for average load while ignoring peak traffic.

Alibaba Cloud product mapping

PAI-EAS provides a variety of GPU instance specifications and supports auto scaling. Lingjun Intelligent Computing Cluster provides large-scale GPU clusters to support the deployment of extra-large models and is equipped with high-speed RDMA networks to ensure tensor parallelism efficiency.

Rationale

Base models often store and compute in FP32 or FP16 precision, which leaves significant room for optimization. With quantization, compilation optimization, and inference acceleration techniques, you can achieve multi-fold performance gains with limited quality loss and significantly reduce inference cost and latency.

Implementation

Quantization: FP16 to INT8 quantization reduces GPU memory usage by half and cuts latency by about 40%, with a typical quality loss of 3–5% that fits most production scenarios. INT4 quantization further reduces memory to one-quarter and cuts latency by another 30%, but quality loss can reach 8–12%. Use it for workloads with lower precision requirements, such as text classification and information extraction. Compilation optimization: Use TensorRT or ONNX Runtime to optimize the model's computation graph. Operator fusion reduces memory copies, and mixed-precision computing fully utilizes Tensor Cores. KV-Cache management: Use PagedAttention to manage GPU memory with paging. This technique avoids KV-Cache fragmentation and waste (traditional contiguous allocation often wastes 30–50% of GPU memory). KV-Cache quantization (FP16 to FP8) further reduces the cache footprint and supports longer context windows. Speculative decoding: Use a small draft model (such as a 1B-parameter model) to quickly generate candidate token sequences, then have the large target model validate them in parallel. If validation passes, accept the tokens. If not, regenerate from the point of divergence. This technique can deliver a 2x to 3x speedup with identical output quality to the original model.

Anti-pattern

Applying the most aggressive quantization scheme without evaluation, leading to unacceptable quality degradation. Neglecting KV-Cache management, causing service interruptions in long-dialogue scenarios due to GPU memory exhaustion. Failing to perform regression testing after compilation optimization, leading to sporadic errors due to numerical precision issues.

Alibaba Cloud product mapping

PAI-Blade provides a one-click model acceleration capability that integrates quantization, compilation optimization, and inference acceleration technologies. The PAI-EAS built-in vLLM engine supports PagedAttention and continuous batching. DashScope inference services come pre-configured with various optimization techniques.

Rationale

Perceived "speed" is not the same as total processing time. Psychological studies show that when a system starts responding within 200 ms and continues to output steadily, users perceive the interaction as "fast and smooth," even if the total generation takes 10 seconds. A streaming response turns a long wait into progressive information delivery and fundamentally changes how users perceive latency.

Implementation

Use the Server-Sent Events (SSE) protocol to deliver tokens progressively. A key technical point is to keep TTFT within 200 ms, which is the threshold for perceived "instant response." Configure API gateways and load balancers to support long-lived connections and streaming transport (disable response buffering). Implement incremental rendering on the client by immediately appending and displaying each token as it arrives, and use a typewriter animation effect to reinforce the feeling of smoothness. Implement a fallback strategy for stream interruptions, such as retaining received content during a network glitch, automatically reconnecting, and resuming from the breakpoint. For multi-step Agent workflows, provide progress updates during non-generation phases (such as tool-call waits) so users do not stare at a blank screen.

Anti-pattern

Returning the entire response at once after generation is complete, leaving users staring at a blank screen for seconds or even tens of seconds. Using streaming transport but failing to implement progressive rendering on the client, so content still appears all at once. Overlooking API gateway buffer settings, which causes streaming responses to be aggregated into chunked transfers.

Alibaba Cloud product mapping

The DashScope streaming API natively supports token-level push via the SSE protocol. PAI-EAS streaming endpoints support streaming output for custom models. Application Load Balancer (ALB) supports WebSocket and SSE long-lived connection forwarding.

Rationale

GPU compute characteristics make batching a key lever for efficiency. The GPU time to process one request versus eight concurrent requests may differ by only 30%, yet throughput increases severalfold. With the right batching strategy, you can greatly improve overall system capacity and GPU utilization without a large increase in per-request latency.

Implementation

Dynamic batching: Accumulate incoming requests in a short window (5–20 ms), then send them to the GPU as a batch. This window size requires a trade-off: too short, and batches are too small to fully utilize the GPU; too long, and you add avoidable waiting latency. Continuous batching: A core innovation of vLLM, it breaks the traditional constraint of waiting for all requests in a batch to finish before releasing resources. When a request in a batch finishes generation, its resources are immediately freed and a new request is admitted into the batch. This can raise GPU utilization from 50–60% with classic batching to over 85%. Priority batching: Maintain separate queues for requests with different priorities. Put VIP users and real-time interactive requests into smaller batches for low latency. Put background analytic requests into larger batches for stable, high throughput.

Anti-pattern

Using a fixed batch size and ignoring load fluctuations, which causes unnecessary waiting under low load and latency spikes under high load. Queuing all requests together with no priorities, so real-time traffic is blocked by numerous offline workloads. Using traditional batching frameworks where short requests are held up by long-running requests in the same batch ("head-of-line blocking").

Alibaba Cloud product mapping

PAI-EAS includes dynamic batching capabilities and supports continuous batching (based on vLLM). PAI-EAS supports multi-queue configuration for priority scheduling. Lingjun Intelligent Computing Cluster provides high-density GPU deployments to support large-scale concurrency.

Rationale

Embedding computation is a preliminary step in a RAG (Retrieval-Augmented Generation) system. Its performance directly affects the speed of knowledge base construction and real-time query latency. For a knowledge base with millions of documents, an inefficient embedding pipeline can cause initial index construction to take several days. Incremental updates may not be completed within an acceptable business time window.

Implementation

Batch processing: After slicing documents, send them to the embedding model in batches of 100 to 500. This fully utilizes the parallel computing power of GPUs. Compared to one-by-one processing, batch processing can increase throughput by 5 to 10 times. GPU-accelerated index construction: Use GPU acceleration for initial knowledge base construction. This reduces the HNSW index construction time for millions of documents from several hours on a CPU to tens of minutes on a GPU. Incremental processing strategy: Detect changes using document fingerprints (content hashes). Re-embed and update the index only for new and modified documents to avoid the high cost of a full rebuild. For frequently updated scenarios, use an "asynchronous double buffering" strategy. Atomically switch to the new index after it is built in the background. This prevents updates from affecting online query performance.

Anti-patterns

Processing documents one by one, resulting in GPU utilization below 10%. Triggering a full rebuild for every document change, which wastes significant computing resources. Sharing resources between index updates and online queries, causing a significant drop in query performance during updates.

Alibaba Cloud product mapping

PAI-EAS provides high-performance model inference services for embeddings and supports batch requests. The Model Studio batch embedding API supports efficient vectorization of large-scale documents. AnalyticDB for Vector provides GPU-accelerated index construction capabilities.

Rationale

The retrieval stage of a RAG system is like the first page of results from a traditional search engine. If relevant information is not recalled at this stage, no matter how powerful the subsequent generation stage is, it cannot compensate. The proper configuration of retrieval parameters directly determines whether the correct information reaches the model.

Implementation

The text slicing strategy is fundamental to retrieval quality. If slices are too small (such as under 128 tokens), context is lost. This results in retrieved fragments that lack independent meaning and are difficult for the model to understand. If slices are too large (such as over 1024 tokens), relevance is diluted. This mixes a large amount of irrelevant content with a small amount of key information, reducing retrieval precision. The optimal range is typically 256 to 512 tokens, with an overlap of 10% to 20% to ensure semantic boundary integrity. Configure retrieval quantity: Set the initial retrieval top-K to 10 to 20 for broad recall to ensure no information is lost. After reranking with a Cross-Encoder, send the top 3 to 5 items to the model. The reranking step adds about 80 to 120 ms of latency but can increase precision by 20% to 35%, making it highly cost-effective. Use different slicing methods for different document types: Slice structured documents, such as FAQs, by entry. Slice narrative documents by paragraph. Slice code documents by function or class.

Anti-patterns

Using a single, fixed slice size for all document types. Setting top-K too low, which causes key information to be missed, or too high, which causes irrelevant content to interfere with generation. Skipping the reranking step to save latency, but the resulting loss in quality far outweighs the time saved.

Alibaba Cloud product mapping

The Model Studio knowledge base provides configurable slicing strategies and retrieval parameters. AnalyticDB for Vector supports custom index parameter tuning and retrieval configurations. Model Studio has a built-in Cross-Encoder reranking capability.

Rationale

A single retrieval method has inherent blind spots. Dense vector retrieval excels at semantic similarity matching (understanding that "car" and "automobile" are synonyms) but may miss exact keyword matches. For example, when a user searches for a specific model like "ModelY", vector retrieval might return semantically similar but different models. Sparse retrieval methods such as BM25 excel at exact keyword matching but cannot handle synonymous rewrites and semantic inference. Hybrid search combines the advantages of both to significantly improve recall rates.

Implementation

Perform dense vector retrieval and BM25 sparse retrieval simultaneously. Merge the results using a score fusion strategy. Fusion methods: Reciprocal rank fusion (RRF) is the most robust choice. It calculates a weighted sum based on the reciprocal of the rank and does not depend on the absolute value of the original scores. This provides a natural normalization effect for the different score distributions of various retrieval methods. Weighted Score Fusion lets you adjust weights based on the scenario. For exact query scenarios, such as retrieving regulation numbers, increase the BM25 weight to 0.6 to 0.7. For semantic query scenarios, such as "how to reduce costs", increase the vector weight to 0.6 to 0.7. Measured data: Compared to pure vector retrieval, hybrid search improves Recall@10 by 10% to 15%. Compared to pure BM25, it improves semantic recall by 20% to 30%. The additional latency is only about 15 to 25 ms because the two retrieval paths are executed in parallel and the fusion computation overhead is extremely low.

Anti-patterns

Using only a single retrieval method, which creates systematic blind spots for certain query types. Using fixed fusion weights that are not optimized for the actual query distribution. Executing the two retrieval paths serially instead of in parallel, which unnecessarily increases latency.

Alibaba Cloud product mapping

AnalyticDB for Vector natively supports hybrid search (vector + full-text) and has built-in RRF and weighted fusion strategies. Elasticsearch, when used with a vector plug-in, can implement hybrid search capabilities. The Model Studio knowledge base supports the configuration of a hybrid search mode.

Rationale

The end-to-end latency of an AI system consists of multiple serial and parallel stages. Optimization must start by identifying the real bottlenecks. A common mistake is to focus all efforts on optimizing model inference. However, the actual bottleneck might be an unoptimized index in vector retrieval, an excessive number of tokens in a prompt template, or unnecessary hops in network transmission. Full-link profiling is essential to ensure you optimize the right areas.

Implementation

Embed timestamps at each critical stage of request processing: client request initiation → AI Gateway reception → authentication → intent recognition → knowledge retrieval (including embedding calculation, index query, and reranking) → prompt assembly → model inference (including queuing and computation) → post-processing (formatting and security checks) → response return. Use sampled data to build latency distribution charts and identify bottleneck distributions at P50, P95, and P99 percentiles. Common findings include: high first-query latency due to a cold vector index (a cold start problem). bloated prompt templates that cause unnecessarily long context inference. and unparallelized serial tool calls that make the total latency the sum of all stages. Establish a latency budget mechanism. Set a latency budget for each stage. If the budget is exceeded, automatically trigger alerts and degradation strategies.

Anti-patterns

Monitoring only the total latency without breaking it down by stage. Deciding on optimization direction based on intuition instead of data. Optimizing a stage that accounts for only 5% of the latency while ignoring the real bottleneck that accounts for 60%. Lacking continuous profiling and failing to track performance after a one-time optimization.

Alibaba Cloud product mapping

ARMS distributed tracing, which is OpenTelemetry compatible, provides full-link trace visualization. SLS pipeline visualization shows latency distribution and trends for each stage. Performance Testing Service (PTS) supports load testing and bottleneck identification.

Rationale

The high computational cost of AI inference makes caching much more valuable than in traditional applications. The cost of a single large model inference can be over 1,000 times the cost of a single Redis query. A well-designed cache architecture can reduce the number of actual inference requests by 30% to 60% without affecting the user experience. This directly translates to cost savings and reduced latency.

Implementation

Build a three-layer cache system. Each layer addresses a different reuse scenario.

Layer 1: Prompt Cache and KV-Cache reuse. When multiple requests share the same system prompt prefix, the corresponding KV-Cache for that prefix can be reused across requests to avoid redundant computation. In practice, for long system prompts (2,000 to 4,000 tokens), KV-Cache reuse can reduce Time to First Token (TTFT) by 40% to 60%. Key implementation points: Ensure the consistency of the system prompt. For example, avoid dynamically inserting changing content such as timestamps into each request. Set an appropriate cache eviction policy.

Layer 2: Semantic cache. Match user queries based on semantic similarity. When the semantic similarity between a new query and a historical query exceeds a threshold (typically 0.92 to 0.95), return the historical answer directly. This completely eliminates the inference cost. A typical scenario is in customer service systems where many users ask the same question using different wording. The semantic cache hit rate is usually between 15% and 35%. Each hit saves 2 to 10 seconds of inference time. Note the cache invalidation requirements: when the knowledge base is updated, the relevant semantic cache entries must be cleared synchronously.

Layer 3: Response cache. For identical queries (character-level match), return the cached result directly. This reduces the response time from seconds to milliseconds. This is suitable for repeated API calls in system integration scenarios.

Each cache layer uses a different invalidation strategy. The KV-Cache uses a Least Recently Used (LRU) policy and is limited by video memory capacity. The semantic cache is invalidated based on the knowledge base version and time windows. The response cache is invalidated based on Time to Live (TTL) and active refresh.

Anti-patterns

Indiscriminately caching all responses, which causes users to receive outdated information. Lacking a cache invalidation strategy, which results in old answers being returned after the knowledge base is updated. Setting the semantic similarity threshold too low, which leads to incorrect matches for unrelated queries.

Alibaba Cloud product mapping

Tair, an enterprise-ready Redis, provides high-performance semantic and response caching and supports vector similarity search. PAI-EAS has built-in KV-Cache optimization and Prefix Cache capabilities. Content Delivery Network (CDN) accelerates edge caching for static resources.

Rationale

AI inference resources such as GPUs scale much more slowly than traditional computing resources. A new GPU instance can take several minutes to be ready from a cold start, including model loading time. However, traffic spikes can occur in seconds. If a system becomes overloaded without selective load management, all requests will time out due to long queues. This leads to a catastrophic "total failure" scenario. Priority queues and load shedding are core mechanisms to "save the important and discard the less important" during overload.

Implementation

Define clear request priorities. P0 is for real-time user conversations, which directly impact user experience and require a Service Level Agreement (SLA) with a response time in seconds. P1 is for real-time background analysis, such as real-time recommendations or stream summarization, which can tolerate a delay of several seconds. P2 is for offline batch processing, such as full evaluations or data backfills, which can tolerate a delay of minutes. Maintain separate processing queues for each priority and configure different Service Level Objectives (SLOs). For example, set the P0 queue depth limit to 20 with a 30-second timeout. Set the P1 queue depth limit to 100 with a 60-second timeout. Set the P2 queue depth limit to unlimited with no hard timeout. Implement a load shedding strategy for extreme load scenarios. When GPU utilization consistently exceeds 90% and the P0 queue depth is growing, first stop accepting new P2 requests and return a 429 Retry-After status. If that is not enough, pause P1 requests to ensure the P0 SLA is not breached. Implement SLO-based scheduling. When resources are abundant, schedule queues fairly. When resources are scarce, use strict preemption based on priority. As a supporting mechanism, P2 requests that are shed should automatically enter a delayed retry queue. These requests are processed automatically after the load returns to normal to ensure eventual completion.

Anti-patterns

Placing all requests in a single queue without prioritization, forcing VIP users and background tasks to compete for the same resources. Simply dropping timed-out requests without a retry mechanism, which causes batch processing tasks to fail permanently. Lacking load pre-warning systems, so that overload is only discovered when the system crashes.

Alibaba Cloud product mapping

AI Gateway and Application Load Balancer (ALB) support priority-based routing and rate limiting based on request headers. RocketMQ provides priority message queues and delayed delivery capabilities. PAI-EAS supports multi-queue configuration and request priority scheduling. Cloud Monitor provides GPU utilization alerts and triggers for auto scaling.