Alibaba Cloud Service Mesh:Build fault-tolerant distributed systems with ASM

Timeouts

How it works

A timeout sets a maximum wait time for a response. If an upstream service does not respond within this period, the sidecar proxy returns an error to the caller instead of waiting indefinitely.

A few important nuances:

• A timeout error does not mean the upstream operation failed. The upstream may still complete processing after the caller gives up.

• Timeouts free up caller resources that would otherwise be blocked waiting.

• After a timeout, the application can take a fallback action such as returning cached data or a default response.

Configure a timeout

Set a timeout on a route in a VirtualService. This example sets a 5-second timeout for all requests routed to the httpbin service:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - 'httpbin'
  http:
  - route:
    - destination:
        host: httpbin
    timeout: 5s

The timeout applies to every request that matches this route.

Retries

How it works

When a request fails due to a transient issue -- a connection timeout, a reset, or a brief service outage -- an automatic retry can recover without manual intervention.

Configure a retry policy

Define a retry policy in a VirtualService. This example retries requests to httpbin up to 3 times on connection failure or reset, with a 5-second timeout per attempt:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - 'httpbin'
  http:
  - route:
    - destination:
        host: httpbin
    retries:
      attempts: 3
      perTryTimeout: 5s
        retryOn: connect-failure,reset

HTTP retry conditions

gRPC retry conditions

gRPC uses HTTP/2 as its transport protocol. Set gRPC retry conditions in the retryOn field alongside HTTP conditions.

Default HTTP retry policy

ASM applies a default retry policy to all HTTP requests, even when no VirtualService retry policy is defined:

The default retry policy requires ASM instances version 1.15.3.120 or later. To update your ASM instance, see Update an ASM instance.

Bulkhead pattern

How it works

The bulkhead pattern caps the number of concurrent connections and requests a client can make to a service. When the limit is reached, new requests are rejected immediately with a 503 error instead of queuing indefinitely. This isolates failures: a slow or unresponsive service cannot exhaust the connection pool and starve other services.

Connection pool settings work at two levels:

• TCP level (applies to TCP, HTTP, and gRPC): maximum concurrent connections and connection timeout.

• HTTP level (applies to HTTP/1.1, HTTP/2, and gRPC): maximum pending requests and maximum requests per connection.

Configure the bulkhead pattern

Define connection pool limits in a DestinationRule. This example caps connections to httpbin at 1 concurrent connection, 1 pending request, and 1 request per connection, with a 10-second connection timeout:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
      tcp:
        connectTimeout: 10s
        maxConnections: 1

Circuit breaking

How it works

Circuit breaking detects unhealthy upstream hosts and temporarily removes (ejects) them from the load balancing pool. The sidecar proxy tracks each upstream host's error rate independently. When a host accumulates consecutive errors beyond a threshold, all traffic is redirected to the remaining healthy hosts. After the ejection period expires, the host is added back to the pool.

This host-level circuit breaking mechanism (outlier detection) differs from both the connection-level limits configured in the bulkhead pattern and the circuit breaking mechanism defined by using ASMCircuitBreaker fields. Together, outlier detection and the bulkhead pattern form a comprehensive resilience strategy:

• Bulkhead pattern: Limits total connections and requests to a service (connection pool).

• Circuit breaking: Removes individual unhealthy hosts from the pool (outlier detection).

Configure circuit breaking

Define outlier detection rules in a DestinationRule. This example ejects a host from the httpbin load balancing pool for 5 minutes if it fails 3 consecutive requests within a 5-second detection window:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    outlierDetection:
      consecutiveErrors: 3
      interval: 5s
      baseEjectionTime: 5m
      maxEjectionPercent: 100

Monitor circuit breaking metrics

When a host is ejected, the sidecar proxy generates metrics that track ejection activity. Use these metrics to detect and diagnose circuit breaking events.

To enable these metrics on the sidecar proxy:

1. Configure proxyStatsMatcher in the sidecar proxy settings. Select Regular Expression Match and set the value to .*outlier_detection.*. For details, see the proxyStatsMatcher section in Configure sidecar proxies.

2. Redeploy the workloads that use the sidecar proxy. For details, see the "Redeploy workloads" section in Configure sidecar proxies.

Set up alerts for circuit breaking

Once metrics are flowing, configure alerts in Managed Service for Prometheus to get notified when circuit breaking triggers.

1. In Managed Service for Prometheus, connect the ACK cluster on the data plane to the Alibaba Cloud ASM component, or upgrade it to the latest version. This ensures circuit breaking metrics are collected. For details, see Manage components.

   Skip this step if you already collect ASM metrics with a self-managed Prometheus instance. See Monitor ASM instances by using a self-managed Prometheus instance.

2. Create an alert rule using a custom PromQL statement. For details, see Create alert rules for Prometheus instances. Example parameter values for a circuit breaking alert:

   Parameter	Example value	Description
   Custom PromQL statement	(sum (envoy_cluster_outlier_detection_ejections_active) by (cluster_name, namespace)) > 0	Checks whether any hosts are currently ejected, grouped by namespace and service name.
   Alert message	Host-level circuit breaking is triggered. Some workloads encounter errors repeatedly and the hosts are ejected from the load balancing pool. Namespace: {{$labels.namespace}}, Service: {{$labels.cluster_name}}. Ejected hosts: {{ $value }}	Includes the namespace, service name, and number of ejected hosts.

What's next

• Configure sidecar proxies -- Customize sidecar proxy behavior, including proxy stats matching for circuit breaking metrics.

• Update an ASM instance -- Upgrade your ASM instance to access the latest fault tolerance features, including the default HTTP retry policy (requires version 1.15.3.120 or later).