Alibaba Cloud Service Mesh (ASM) manages all north-south and east-west traffic in clusters. With timeout, retry, throttling, circuit breaking, queuing, prefetch, and fallback policies, ASM strengthens high availability across distributed systems.
Capability introduction
In distributed systems, traffic protection and scheduling are essential for stability and high availability. Any traffic fluctuation or anomaly can cause service failures, avalanche effects, or resource exhaustion.
Traditional frameworks such as Resilience4j provide high availability features like throttling and circuit breaking. Service Mesh delivers these capabilities at the network infrastructure level with non-intrusive integration, offering low coupling and flexible configuration.
ASM provides the following high availability features:
Throttling: protecting the system from being overloaded
Throttling controls traffic to protect backend services from overload, resource exhaustion, or malicious attacks while reducing costs and improving user experience. In multitenancy scenarios, you can apply throttling per tenant for fine-grained traffic management and fair resource allocation.
ASM provides the following throttling policies:
Local throttling and global throttling
The mesh proxy Envoy supports two types of throttling methods: local throttling and global throttling.
Scenarios
Local and global throttling cover most common scenarios: throttling a specific service in the cluster or on specific gateway routes, with separate rate-limiting quotas based on request matching criteria.
-
Global throttling: Uses a centralized throttling service and a Redis database to enforce rate limits across multiple services.
-
Local throttling: Simpler to configure and does not rely on external components, but each replica maintains its own independent rate limiter.
Rate limiting based on ASM traffic scheduling suite
The ASM traffic scheduling suite is a unified traffic scheduling architecture for Service Mesh that provides policies for load scheduling and request management in cloud-native distributed applications. The suite supports RateLimitingPolicy for throttling.
Scenarios
RateLimitingPolicy supports grouping requests by tags and applying throttling within each group, making it suitable for fine-grained control in multitenancy environments.
Traffic concurrency control: protecting system critical resources
Concurrency control limits the number of concurrent requests to prevent resource exhaustion. Unlike throttling, it is ideal for services that depend on critical resources such as thread pools or databases, improving resource utilization and safety.
ASM provides two concurrency control mechanisms:
Concurrency control based on ASM traffic scheduling suite
The ASM traffic scheduling suite supports ConcurrencyLimitingPolicy to control concurrent requests.
Scenarios
If the concurrency limit for a system is relatively fixed, you can set a fixed limit for a specific service. Requests exceeding the limit receive a 429 response.
Using ASMAdaptiveConcurrency for adaptive concurrency control
Envoy supports adaptive concurrency control, which you can enable through ASMAdaptiveConcurrency. ASMAdaptiveConcurrency dynamically adjusts the concurrency limit to match the destination service capacity, rejecting excess requests with a 503 response and the message "reached concurrency limit".
Scenarios
If the concurrency limit for a system fluctuates significantly and is hard to estimate, use AdaptiveConcurrency to limit concurrency dynamically. We recommend also enabling retries through DestinationRule so that rejected requests can be retried successfully.
Traffic circuit breaking: isolating faulty nodes to prevent avalanche effect
Circuit breaking disconnects faulty services to isolate failures and prevent them from spreading across the system.
ASM provides circuit breaking at various levels:
Connection pool-level circuit breaking
Connection pool circuit breaking, configured through destination rules, limits the maximum number of HTTP/1 or TCP connections to the target service host.
Scenarios
This strategy triggers circuit breaking by limiting TCP connections and applies to services where HTTP status codes do not indicate issues.
Host-level circuit breaking
Host-level circuit breaking, also configured through destination rules, monitors faults within a time window and evicts hosts when the error rate exceeds a threshold.
Scenarios
Host-level circuit breaking operates independently per upstream host, temporarily evicting hosts with continuous 5xx errors from the load balancing pool. It detects persistent errors caused by a single workload issue but does not target specific API interfaces.
Route-level circuit breaking
ASM supports configuring circuit breaker rules for east-west traffic between services and on specific routes. For more information, see the referenced document.
Scenarios
Route-level circuit breaking operates at the service level, detecting persistent errors in specific APIs caused by service dependencies or logic errors.
Traffic fallback: handling call failure scenarios
When a microservice fails or becomes unavailable, a fallback mechanism routes requests to an alternative service, maintaining overall system availability.
Scenarios
Combining host-level circuit breaking with traffic fallback lets you switch to a backup service during an outage, maintaining availability during circuit breaking.
Traffic prefetch: smooth transition for new version deployment
In traditional blue-green deployments or rolling updates, a new service version may receive all traffic at once, causing excessive load. Traffic prefetch introduces traffic gradually — for example, starting at 10% and ramping up — which suits services with high cold-start costs.
ASM supports two levels of traffic prefetch:
Warm-up
The warm-up feature is configured through destination rules, allowing a service instance to gradually increase its request volume within a specified time window.
Scenarios
When a target service's upstream host is within the slow-start window, the load balancer reduces the traffic allocated to that host. This is ideal for service scale-out or new version releases, as it warms up newly started upstream hosts.
Slow-start warm-up is not applicable when the newly launched service or available upstream host endpoints are limited.
Progressive service online based on ASM traffic scheduling suite
Configuring LoadRampingPolicy during a new service release gradually increases the traffic it receives, ensuring a smooth transition.
Scenarios
The ASM traffic scheduling suite uses a request sampler to reject a proportion of requests, ensuring that total traffic increases slowly. This method suits new service releases but not service scale-out or new version releases.
Timeout and retry: ensure service reliability
Timeout and retry are common fault tolerance measures in distributed systems that maintain availability when services experience occasional errors. Timeout prevents requests from hanging indefinitely, while retry handles transient issues such as network jitter or temporary failures.
Request queuing and priority-based scheduling: survive peak hours
Request queuing builds on throttling and concurrency limits. When the traffic rate or concurrent request count exceeds the system's processing capacity, excess requests are queued instead of being immediately rejected, and are processed as earlier requests complete.
ASM provides three priority-based scheduling policies through its traffic scheduling suite, based on concurrency, traffic rate, and latency. These policies support priority-based scheduling of queued requests, ensuring that high-priority requests are processed first to maintain critical functionality or improve the experience for key tenants.
ASM supports the following request queuing and priority-based scheduling policies:
Priority-based request scheduling under controllable concurrency
Configure ConcurrencySchedulingPolicy to determine whether the system is overloaded based on a concurrency limit. When request concurrency exceeds this limit, additional requests are queued and scheduled by priority.
Scenarios
Ideal for applications with concurrency limits and services that experience significant traffic fluctuations.
Priority-based request scheduling under controllable rate
Configure QuotaSchedulingPolicy to determine whether the system is overloaded based on a traffic rate limit. When request rate exceeds this limit, additional requests are queued and scheduled by priority.
Scenarios
Ideal for applications with traffic rate limits and services that experience significant traffic fluctuations.
Priority-based request scheduling based on the average latency
Configure AverageLatencySchedulingPolicy to compare real-time request latency against the historical average. A significant deviation indicates system overload, and excess requests are queued and scheduled by priority.
Scenarios
This adaptive method suits scenarios where it is difficult to determine the maximum acceptable rate or concurrency for a service.