Alibaba Cloud Service Mesh:Disaster recovery scenario for multiple ACK clusters in different VPCs (based on CEN for VPC network connectivity)

VPC configuration

vSwitch configuration

Pod and Service CIDR blocks

Create a CEN instance and transit routers

1. Log on to the CEN console and create a CEN instance. See Create a CEN instance.

2. On the CEN Instances page, click the CEN instance name. On the Basic Information tab, click Create Transit Router. Create two transit routers:

   • Region: China (Shanghai), Name: shanghai-router

   • Region: China (Hangzhou), Name: hangzhou-router

Attach VPCs to transit routers

Repeat the following steps for each transit router:

1. Click the transit router ID.

2. On the Intra-region Connections tab, click Create Connection.

3. Set Instance Type to Virtual Private Cloud (VPC) and select the VPC that corresponds to the transit router's region under Network Instance.

4. Keep other settings at their defaults and click OK.

Configure inter-region bandwidth

1. Click the name of a transit router, then click Create Connection.

2. In the Connection With Peer Network Instance dialog box, set Region to the local transit router's region and Peer Region to the remote region. For example, set Region to China (Hangzhou) and Peer Region to China (Shanghai). For parameter details, see Inter-region connections.

3. After creation, verify the connection appears on the Inter-region Connections tab.

Add security group rules

Allow cross-cluster pod traffic by adding the peer cluster's Pod CIDR block to each cluster's security group.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. Get the Pod CIDR block for each cluster:

   1. On the Clusters page, select the China (Shanghai) region. Click the ack-shanghai cluster name. On the Cluster Information page, click the Basic Information tab to find the Pod CIDR block.

   2. Repeat for the ack-hangzhou cluster in the China (Hangzhou) region.

3. Add the peer cluster's Pod CIDR to each cluster's security group:

   1. On the Cluster Information page of each cluster, click the Basic Information tab. Click the security group ID next to Control Plane Security Group.

   2. On the Inbound tab, click Add Rule.

   3. Set Protocol Type to All and Source to the Pod CIDR block of the peer cluster. Keep other defaults and click Save.

4. Verify connectivity by logging on to a node in each cluster and running ping against a node in the other cluster. See Log on to nodes.

Deploy the application

Deploy Bookinfo in both the ack-hangzhou and ack-shanghai clusters:

kubectl apply -f bookinfo.yaml

Create routing rules

Switch kubectl to the ASM instance context and apply the following routing rules.

1. Save the following YAML as asm.yaml:

   View the complete YAML

      apiVersion: networking.istio.io/v1alpha3
      kind: Gateway
      metadata:
        name: bookinfo-gateway
      spec:
        selector:
          istio: ingressgateway
        servers:
        - port:
            number: 80
            name: http
            protocol: HTTP
          hosts:
          - "*"
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: bookinfo
      spec:
        hosts:
        - "*"
        gateways:
        - bookinfo-gateway
        http:
        - match:
          - uri:
              exact: /productpage
          - uri:
              prefix: /static
          - uri:
              exact: /login
          - uri:
              exact: /logout
          - uri:
              prefix: /api/v1/products
          route:
          - destination:
              host: productpage
              port:
                number: 9080
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: productpage
      spec:
        host: productpage
        subsets:
        - name: v1
          labels:
            version: v1
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: reviews
      spec:
        host: reviews
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
        - name: v3
          labels:
            version: v3
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: ratings
      spec:
        host: ratings
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
        - name: v2-mysql
          labels:
            version: v2-mysql
        - name: v2-mysql-vm
          labels:
            version: v2-mysql-vm
      ---
      apiVersion: networking.istio.io/v1alpha3
      kind: DestinationRule
      metadata:
        name: details
      spec:
        host: details
        subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2

2. Apply the routing rules:

      kubectl apply -f asm.yaml

Verify the deployment

1. Get the ingress gateway address.

2. Open http://<ingress-gateway-ip>/productpage in a browser and refresh the page 10 times. The Bookinfo application distributes requests across v1, v2, and v3 of the reviews service in roughly equal proportions (1:1:1).

Simulate a regional failure

Scale the reviews Deployment in the ack-hangzhou cluster to zero replicas:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the ack-hangzhou cluster name. In the left-side navigation pane, choose Workloads > Deployments.

3. On the Deployments page, set Namespace to default. Click Scale in the Actions column for the reviews deployment.

4. Set Desired Number Of Pods to 0 and click OK.

Configure outlier detection

Add outlier detection to the reviews DestinationRule so that unhealthy endpoints are ejected and failover is triggered.

1. On the ASM instance details page, choose Traffic Management Center > DestinationRule in the left-side navigation pane.

2. Click Edit YAML in the Actions column of the reviews DestinationRule.

3. Add the following trafficPolicy block under spec and click OK: The following table explains each parameter and its role in failover behavior:

   Parameter	Value	Description
   maxRequestsPerConnection	1	Limits each connection to a single request. This disables keep-alive, forcing new connections that can be routed to healthy endpoints.
   baseEjectionTime	1m	Keeps an unhealthy endpoint ejected for 1 minute before re-evaluating.
   consecutive5xxErrors	1	Ejects an endpoint after a single 5xx error.
   interval	1s	Runs the ejection scan every 1 second.

      spec:
        # ... existing content ...
        trafficPolicy:
          connectionPool:
            http:
              maxRequestsPerConnection: 1
          outlierDetection:
            baseEjectionTime: 1m
            consecutive5xxErrors: 1
            interval: 1s

Enable geolocation-based failover

Verify failover

Run the following command to send 10 requests to the Bookinfo application and count responses from the v2 reviews service:

for ((i=1;i<=10;i++)); do
  curl http://<ingress-gateway-ip>/productpage 2>&1 | grep full.stars
done | wc -l

Replace <ingress-gateway-ip> with port 80 of the ingress gateway IP in the ack-hangzhou cluster.

Expected output:

20

Each request routed to the v2 reviews service returns two lines containing full stars. An output of 20 confirms that all 10 requests reached the v2 reviews service in the ack-shanghai cluster, which means failover is working.

Configure traffic distribution

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the ASM instance name. In the left-side navigation pane, choose ASM Instance > Base Information.

3. Click Configure a Geolocation-based Load Balancing next to Geolocation-based Load Balancing.

4. Click Congifure a traffic distribution rule. Set Source to cn-hangzhou, Destination to cn-shanghai, and Traffic Percentage to 10%.

5. Click Save Configuration.

Verify traffic distribution

Run the following command to send 10 requests and observe the distribution:

for ((i=1;i<=10;i++)); do
  curl http://<ingress-gateway-ip>/productpage 2>&1 | grep full.stars
done

Replace <ingress-gateway-ip> with port 80 of the ingress gateway IP in the ack-hangzhou cluster.

Expected output:

<!-- full stars: -->
<!-- full stars: -->

Out of 10 requests, approximately 9 reach the v1 reviews service in ack-hangzhou (no full stars output), and 1 reaches the v2 reviews service in ack-shanghai (2 lines of full stars). This confirms a 90%/10% traffic split matching the configured weights.

Why does adding a cluster to ASM fail after connecting VPC networks through CEN?

The most likely cause is a missing or misconfigured inter-region data transfer plan in CEN. Without proper inter-region bandwidth, the ASM control plane in one region cannot reach the data plane cluster in another region, even though intra-region VPC connectivity works.

To fix this, verify and reconfigure the inter-region connection settings in CEN as described in Step 2: Connect VPC networks through CEN. Make sure inter-region bandwidth is allocated between the two transit routers.