Alibaba Cloud Service Mesh:Use a service mesh for service-level disaster recovery

Failover mechanism: locality-aware failover in a service mesh

ASM provides a locality-aware failover mechanism. The sidecar proxy of each workload continuously monitors whether the destination workload returns consecutive response errors within a configurable time window. When the sidecar proxy detects that a workload is faulty, it removes the workload from the load balancing pool and redirects traffic to other healthy workloads.

The following example uses two clusters in two different regions, with two peer sets of an application deployed across them. In each cluster, the workloads of each service are evenly distributed across different zones by using topology spread constraints.

• Normal traffic topology

  When all services are healthy, the service mesh keeps traffic within the same zone to minimize the impact of the destination location on request latency.

• Automatic failover based on locality planning when a workload becomes faulty

  When the sidecar proxy detects consecutive response errors from a workload, it removes the workload from the load balancing pool and redirects requests to other healthy workloads. Traffic failover follows this priority order:

• Other workloads in the same zone

• Workloads in a different zone within the same region

  To notify business contacts when a failover occurs, configure a Prometheus instance to collect service mesh metrics and set up alerts.

• Cross-region traffic failover

  When no healthy workloads are available in any zone of the region, traffic is redirected to the same service deployed as a peer in a cluster in another region. The service mesh handles this process automatically. Cross-region failover requires network connectivity between the clusters. For the available methods and their trade-offs, see Application deployment topologies.

Application deployment topologies: supported cluster topologies and a comparison

The locality-aware failover mechanism of ASM applies to multiple application deployment topologies. Each topology covers a different range of failure dimensions and differs in resource costs and O&M complexity.

Single-cluster, multi-zone deployment

This is the simplest deployment mode for disaster recovery. Configure a multi-zone node pool, select vSwitches from multiple zones for the node pool, and select the balanced distribution policy when you configure the scaling policy. ECS instances are then evenly distributed across the zones specified for the scaling group.

Pros	Cons
Requires only one ACK cluster that contains a multi-zone node pool, with applications evenly deployed across zones by using topology spread constraints.	Cannot handle failures caused by Kubernetes misconfigurations or application dependencies.
Cloud resources such as load balancing, databases, Kubernetes clusters, and the service mesh need to be prepared only once within a single region.	Cannot handle region-level failures.

Single-region, multi-zone, multi-cluster deployment

This mode extends the single cluster of a single-cluster, multi-zone deployment to multiple clusters, each with its own configuration. When a failover occurs, ASM first looks for other workloads in the same zone, and then selects one of the healthy workloads in a different zone within the same region as the failover destination. Select separate zones for the two clusters. If the zones of the two clusters overlap, workloads in the same zone may take precedence over workloads in the same cluster, which can be unexpected for your business.

Pros	Cons
Compared with a single-cluster deployment, this topology covers more failure causes and further improves the overall availability of the system.	Services must be deployed as peers in two Kubernetes cluster environments, which makes deployment and O&M more complex. Cloud infrastructure resources such as load balancing and Kubernetes clusters must be deployed multiple times, which increases costs.
The two clusters only need to reside in the same VPC. This topology has a low inter-cluster connectivity cost when traffic must fail over to a service in the other cluster.	Cannot handle region-level failures. Requires network planning to avoid conflicts between cluster configurations. For more information, seePlan CIDR blocks for multiple clusters.

Multi-region, multi-cluster deployment

This mode turns the single-region deployment of a multi-cluster architecture into a multi-region deployment. It keeps the deployment of each service dispersed across every dimension: zones, regions, clusters, and dependencies. This way, healthy workloads keep serving traffic when a failure occurs for any possible reason.

Pros	Cons
Distributes the workloads of the same service across different regions, zones, and clusters to improve service availability.	Services must be deployed as peers in two Kubernetes cluster environments. For better configuration push latency, two service mesh instances are also required.
The multi-master control plane architecture effectively controls latency.	Cross-region network connectivity is complex.

For high availability requirements, multi-region, multi-cluster deployment provides the highest level of service availability among the topology options that ASM manages. If you need to connect networks across clusters, use either of the following methods:

• Use Cloud Enterprise Network (CEN) to directly connect the underlying networks of clusters in different regions. First, plan the network configurations of the clusters to avoid network conflicts. For more information, seePlan CIDR blocks for multiple clusters. Then, use CEN to connect the networks across regions. For more information, seeCreate a cross-region connection. This method suits customers with high-quality cross-region traffic requirements.

• Use the ASM cross-cluster network proxy to connect networks across clusters and implement failover. This method optimizes resource costs and resolves complex network connectivity issues. With this method, the networks of the two clusters do not need to be directly connected, and no cluster network planning is required. The proxy establishes an mTLS-secured channel between clusters for necessary cross-cluster communication and secure cross-region traffic failover. Because this method relies on public network communication with limited quality, it suits scenarios that do not have strict inter-cluster communication requirements or where the physical networks of the clusters cannot be connected. For more information about the ASM cross-cluster network proxy, seeDisaster recovery for multiple ACK clusters in different VPCs (using ASM cross-cluster mesh proxy).

Multi-region, multi-cluster deployment across a multicloud environment

This is the most advanced deployment topology. ASM fully supports the management of Kubernetes clusters from any cloud provider and from on-premises data centers. Run two Kubernetes cluster environments across different cloud providers, or run one in the cloud and one on-premises. If one infrastructure fails, the other continues to provide your services.

Similar to a multi-region, multi-cluster deployment, a multicloud, multi-region, multi-cluster deployment also requires network connectivity between clusters. However, ASM cannot provide a managed control plane in non-Alibaba Cloud environments. Use an ASM remote control plane to resolve configuration push latency for the service mesh in cross-cloud scenarios. For more information, seeUse a remote control plane to reduce push latency.

Pros	Cons
Distributes the workloads of the same service across different regions, zones, clusters, and cloud providers to improve service availability.	Services must be deployed as peers in two Kubernetes cluster environments. Because infrastructure and cloud service capabilities differ between cloud providers, a multicloud deployment may run into many compatibility issues and other unknown issues.
The multi-master control plane architecture effectively controls latency.	Cross-region network connectivity is complex.

Step 1: Prepare the environment

Complete the following sub-steps in order to prepare the environment for the disaster recovery demonstration.

1. Create a Kubernetes cluster in each of two regions and name them cluster-1 and cluster-2. Configure multiple zones for both clusters and enable Expose API server with EIP. For more information, seeCreate an ACK managed cluster.

2. In the same two regions as the ACK clusters, create the ASM instances mesh-1 and mesh-2. When you create the instances, make sure that you select vSwitches in the same zones as the ACK clusters. For more information, see Step 1, Step 2, and Step 3 inImplement multi-cluster disaster recovery with an ASM multi-master control plane architecture.

3. In mesh-1 and mesh-2, create an NLB-type ingress gateway named ingressgateway. For the NLB zones, select the same two zones as the ASM instances and the ACK clusters. For more information, seeUse an NLB instance for an ASM ingress gateway.

4. In mesh-1 and mesh-2, enable automatic sidecar injection for the default namespace of the two clusters. This is configured as part of global namespace management in ASM. For more information, seeManage global namespaces.

5. Deploy the sample application as peers in the two ACK clusters. This example uses two zones: cn-hangzhou-h and cn-hangzhou-k. Adjust the zones in the YAML based on your actual environment.

   Note

   The container images in the following YAML are pulled from the China (Hangzhou) registry (registry-cn-hangzhou.ack.aliyuncs.com). If you deploy across regions, make sure that the images are available in all target regions. You may need to configure cross-region image synchronization through Container Registry.

   Expand to view the command

   kubectl apply -f- <<EOF
   apiVersion: v1
   kind: Service
   metadata:
     name: mocka
     labels:
       app: mocka
       service: mocka
   spec:
     ports:
     - port: 8000
       name: http
     selector:
       app: mocka
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mocka-cn-hangzhou-h
     labels:
       app: mocka
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mocka
     template:
       metadata:
         labels:
           app: mocka
           locality: cn-hangzhou-h
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-h  
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-h 
           - name: app
             value: mocka
           - name: upstream_url
             value: "http://mockb:8000/"
           ports:
           - containerPort: 8000
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mocka-cn-hangzhou-k
     labels:
       app: mocka
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mocka
     template:
       metadata:
         labels:
           app: mocka
           locality: cn-hangzhou-k
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-k
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-k
           - name: app
             value: mocka
           - name: upstream_url
             value: "http://mockb:8000/"
           ports:
           - containerPort: 8000
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: mockb
     labels:
       app: mockb
       service: mockb
   spec:
     ports:
     - port: 8000
       name: http
     selector:
       app: mockb
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mockb-cn-hangzhou-h
     labels:
       app: mockb
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mockb
     template:
       metadata:
         labels:
           app: mockb
           locality: cn-hangzhou-h
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-h
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-h
           - name: app
             value: mockb
           - name: upstream_url
             value: "http://mockc:8000/"
           ports:
           - containerPort: 8000
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mockb-cn-hangzhou-k
     labels:
       app: mockb
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mockb
     template:
       metadata:
         labels:
           app: mockb
           locality: cn-hangzhou-k
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-k
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-k
           - name: app
             value: mockb
           - name: upstream_url
             value: "http://mockc:8000/"
           ports:
           - containerPort: 8000
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: mockc
     labels:
       app: mockc
       service: mockc
   spec:
     ports:
     - port: 8000
       name: http
     selector:
       app: mockc
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mockc-cn-hangzhou-h
     labels:
       app: mockc
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mockc
     template:
       metadata:
         labels:
           app: mockc
           locality: cn-hangzhou-h
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-h
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-h
           - name: app
             value: mockc
           ports:
           - containerPort: 8000
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: mockc-cn-hangzhou-k
     labels:
       app: mockc
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: mockc
     template:
       metadata:
         labels:
           app: mockc
           locality: cn-hangzhou-k
       spec:
         nodeSelector:      
           topology.kubernetes.io/zone: cn-hangzhou-k
         containers:
         - name: default
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
           imagePullPolicy: IfNotPresent
           env:
           - name: version
             value: cn-hangzhou-k
           - name: app
             value: mockc
           ports:
           - containerPort: 8000
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: Gateway
   metadata:
     name: mocka
     namespace: default
   spec:
     selector:
       istio: ingressgateway
     servers:
       - hosts:
           - '*'
         port:
           name: test
           number: 80
           protocol: HTTP
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
     name: demoapp-vs
     namespace: default
   spec:
     gateways:
       - mocka
     hosts:
       - '*'
     http:
       - name: test
         route:
           - destination:
               host: mocka
               port:
                 number: 8000
   EOF

   After you run the preceding commands in both clusters, the services mocka, mockb, and mockc are deployed as peers in both clusters. Each service contains two stateless deployments that have one replica each. Different nodeSelector settings distribute the deployments to nodes in different zones, and environment variables make each deployment return the zone that it resides in.

   For a straightforward demonstration, this example uses the nodeSelector field to manually select the zone of each pod. When you build a real high-availability environment, configure topology spread constraints to distribute pods across different zones as much as possible. For more information, seeConfigure high availability for workloads.

6. Configure Global Traffic Manager (GTM) with the NLB instances to implement active-active disaster recovery for the ingress gateway. For more information, seeHow GTM implements active-active disaster recovery.

Step 2: Enable locality-aware failover

ASM enables locality-aware failover by default, but the feature takes effect only together with host-level circuit breaking rules that are configured in destination rules.

1. Deploy host-level circuit breaking rules in the cluster-1 and cluster-2 clusters.

   kubectl apply -f- <<EOF
   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: mocka
   spec:
     host: mocka
     trafficPolicy:
       outlierDetection:
         splitExternalLocalOriginErrors: true
         consecutiveLocalOriginFailures: 1
         baseEjectionTime: 5m
         consecutive5xxErrors: 1
         interval: 30s
         maxEjectionPercent: 100
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: mockb
   spec:
     host: mockb
     trafficPolicy:
       outlierDetection:
         splitExternalLocalOriginErrors: true
         consecutiveLocalOriginFailures: 1
         baseEjectionTime: 5m
         consecutive5xxErrors: 1
         interval: 30s
         maxEjectionPercent: 100
   ---
   apiVersion: networking.istio.io/v1beta1
   kind: DestinationRule
   metadata:
     name: mockc
   spec:
     host: mockc
     trafficPolicy:
       outlierDetection:
         splitExternalLocalOriginErrors: true
         consecutiveLocalOriginFailures: 1
         baseEjectionTime: 5m
         consecutive5xxErrors: 1
         interval: 30s
         maxEjectionPercent: 100
   EOF

   outlierDetection is the mechanism that the service mesh uses for service fault detection and faulty endpoint ejection. The following table describes the configuration items:

   Configuration item	Description
   interval	The interval at which fault detection is performed.
   baseEjectionTime	The amount of time to wait before an endpoint is ejected from the load balancing pool after the endpoint is determined to be faulty.
   maxEjectionPercent	The maximum percentage of endpoints that can be ejected.
   consecutive5xxErrors	The number of consecutive 5xx errors that an endpoint returns before the endpoint is determined to be faulty.
   splitExternalLocalOriginErrors	Specifies whether to count connection failures and connection timeouts as faults.
   consecutiveLocalOriginFailures	The number of consecutive non-5xx errors, such as connection failures and connection timeouts, that an endpoint produces before the endpoint is determined to be faulty.

2. Verify the service status.

   Send a request to the service domain name.

   curl mock.asm-demo.work/mock -v

   * Host mock.asm-demo.work:80 was resolved.
   * IPv6: (none)
   * IPv4: 8.xxx.xxx.47, 8.xxx.xxx.42
   *   Trying 8.xxx.xxx.47:80...
   * Connected to mock.asm-demo.work (8.209.XXX.XX) port 80
   > GET /mock HTTP/1.1
   > Host: mock.asm-demo.work
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/1.1 200 OK
   < date: Wed, 11 Dec 2024 12:10:56 GMT
   < content-length: 153
   < content-type: text/plain; charset=utf-8
   < x-envoy-upstream-service-time: 3
   < server: istio-envoy
   < 
   * Connection #0 to host mock.asm-demo.work left intact
   -> mocka(version: cn-hangzhou-k, ip: 10.1.225.40)-> mockb(version: cn-hangzhou-k, ip: 10.1.225.31)-> mockc(version: cn-hangzhou-k, ip: 10.1.225.32)%

   The service call chain stays on workloads in the same zone.

Step 3: Perform a failure drill

The following example simulates a workload failure by manually replacing the container image of the mockb service in one zone.

1. Replace the image of mockb-cn-hangzhou-h in cluster-2 to simulate a failure.

   1. Log on to theACK console. In the left navigation pane, click Clusters.

   2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

   3. In the list, find the mockb-cn-hangzhou-h workload. In the Actions column on the right, click Edit. Change the image to registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2, and add ["sleep","3600"] to the command in the Start field so that the container can start properly. Click Update.

2. Continuously access the application domain name to view the request path after the failure occurs.

   curl mock.asm-demo.work/mock -v

   Expected result 1: The request is sent to a zone other than cn-hangzhou-h.

   * Host mock.asm-demo.work:80 was resolved.
   * IPv6: (none)
   * IPv4: 8.209.XXX.XX, 8.221.XXX.XX
   *   Trying 8.209.247.47:80...
   * Connected to mock.asm-demo.work (8.209.247.47) port 80
   > GET /mock HTTP/1.1
   > Host: mock.asm-demo.work
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/1.1 200 OK
   < date: Wed, 11 Dec 2024 12:10:56 GMT
   < content-length: 153
   < content-type: text/plain; charset=utf-8
   < x-envoy-upstream-service-time: 3
   < server: istio-envoy
   < 
   * Connection #0 to host mock.asm-demo.work left intact
   -> mocka(version: cn-hangzhou-k, ip: 10.1.225.40)-> mockb(version: cn-hangzhou-k, ip: 10.1.225.31)-> mockc(version: cn-hangzhou-k, ip: 10.1.225.32)%

   The request stays within a healthy zone.

   Expected result 2: The request is sent to the cn-hangzhou-h zone for the first time.

   * Host mock.asm-demo.work:80 was resolved.
   * IPv6: (none)
   * IPv4: 112.124.XX.XXX, 121.41.XXX.XXX
   *   Trying 112.124.65.120:80...
   * Connected to mock.asm-demo.work (112.124.65.120) port 80
   > GET /mock HTTP/1.1
   > Host: mock.asm-demo.work
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/1.1 200 OK
   < date: Wed, 11 Dec 2024 12:08:45 GMT
   < content-length: 220
   < content-type: text/plain; charset=utf-8
   < x-envoy-upstream-service-time: 48
   < server: istio-envoy
   < 
   * Connection #0 to host mock.asm-demo.work left intact
   -> mocka(version: cn-hangzhou-h, ip: 192.168.122.135)upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused%

   The connection is refused when the request reaches the mockb service.

   Expected result 3: The request is sent to the cn-hangzhou-h zone again after the sidecar proxy detects the failure.

   * Host mock.asm-demo.work:80 was resolved.
   * IPv6: (none)
   * IPv4: 8.209.XXX.XX, 8.221.XXX.XX
   *   Trying 8.209.247.47:80...
   * Connected to mock.asm-demo.work (8.209.247.47) port 80
   > GET /mock HTTP/1.1
   > Host: mock.asm-demo.work
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/1.1 200 OK
   < date: Wed, 11 Dec 2024 12:10:59 GMT
   < content-length: 154
   < content-type: text/plain; charset=utf-8
   < x-envoy-upstream-service-time: 4
   < server: istio-envoy
   < 
   * Connection #0 to host mock.asm-demo.work left intact
   -> mocka(version: cn-hangzhou-h, ip: 10.0.239.141)-> mockb(version: cn-hangzhou-k, ip: 10.1.225.31)-> mockc(version: cn-hangzhou-k, ip: 10.1.225.32)%

   mockb traffic is redirected to cn-hangzhou-k, a different zone in the same region.

(Optional) Step 4: Configure alerts for service-level failures

1. Configure the sidecar proxy to report circuit breaking metrics through proxyStatsMatcher. When you configure proxyStatsMatcher, select Regular Expression Match and set the value to .*outlier_detection.*. For more information, seeproxyStatsMatcher.

   The following table describes some of the circuit breaking metrics:

   Metric	Metric type	Description
   envoy_cluster_outlier_detection_ejections_active	Gauge	The number of hosts that are currently ejected.
   envoy_cluster_outlier_detection_ejections_enforced_total	Counter	The number of host ejection events.
   envoy_cluster_outlier_detection_ejections_overflow	Counter	The number of times that a host ejection was skipped because the maximum ejection percentage was exceeded.
   ejections_detected_consecutive_5xx	Counter	The number of times that a host was detected to return consecutive 5xx errors.

2. Redeploy the stateless workloads (mocka, mockb, and mockc) to apply the updated proxyStatsMatcher configuration. For more information, seeRedeploy the workload.

3. Create an alert rule for host-level circuit breaking.

   1. Integrate the Alibaba Cloud ASM component with the data plane cluster or upgrade the component to the latest version, so that Managed Service for Prometheus can collect the exposed circuit breaking metrics. For more information about how to update the integration component, seeManage integrations. (If you already collect service mesh metrics with a self-managed Prometheus instance as described inIntegrate a self-managed Prometheus instance for mesh monitoring, skip this step.)

   2. Create an alert rule for host-level circuit breaking. For more information, seeCreate an alert rule for a Prometheus instance.

      The following table provides examples for the key parameters of the alert rule: Configure the other parameters based on your requirements by referring to the preceding documents.

   Parameter	Example	Description
   Custom PromQL statement	(sum (envoy_cluster_outlier_detection_ejections_active) by (cluster_name, namespace)) > 0	The example queries the envoy_cluster_outlier_detection_ejections_active metric to determine whether any hosts are currently ejected in the cluster, and groups the query results by the namespace and the service name.
   Alert content	Host-level circuit breaking is triggered. A workload returned consecutive errors and was ejected from the service load balancing pool! Namespace: {{$labels.namespace}}. Service where the ejection occurred: {{$labels.cluster_name}}. Number of ejections: {{ $value }}	The example alert message shows the namespace and the name of the service that triggered circuit breaking, and the number of hosts that are currently ejected for that service.