Inter-region disaster recovery scenario for multiple ACK clusters in different VPCs (based on CEN for VPC network connectivity) - Alibaba Cloud Service Mesh

Service Mesh ASM provides inter-region traffic distribution and inter-region failover capabilities for application services. The inter-region traffic distribution feature routes traffic to multiple clusters according to specified weights, achieving multi-region load balancing. The inter-region failover feature transfers traffic from one region to another when services in a region fail, implementing inter-region disaster recovery. This topic uses the Bookinfo application to demonstrate how to implement inter-region disaster recovery and traffic load balancing using inter-region failover and traffic distribution capabilities.

Network configuration

Configure the CIDR blocks, names, and other information for vSwitches, VPCs, and clusters. See Plan CIDR blocks for multiple clusters on the data plane for more information. Below is an example of network configuration.

Network configuration of vSwitches and VPCs
- VPC
  Object
  VPC name
  Region
  IPv4 CIDR block
  Cluster
  vpc-hangzhou
  cn-hangzhou
  20.0.0.0/8
  vpc-shanghai
  cn-shanghai
  21.0.0.0/8
  Service Mesh
  vpc-hangzhou2
  cn-hangzhou
  192.168.0.0/16
- vSwitch
  Important
  To avoid route conflicts when connecting VPC networks using CEN, two vSwitches cannot use the same CIDR block.
  Object
  vSwitch name
  VPC
  IPv4 CIDR block
  Cluster
  vpc-hangzhou-switch-1
  vpc-hangzhou
  20.0.0.0/16
  vpc-shanghai-switch-1
  vpc-shanghai
  21.0.0.0/16
  Service Mesh
  vpc-hangzhou-switch-2
  vpc-hangzhou2
  192.168.0.0/24
Pod and Service network configuration for clusters
Cluster name
Region
VPC
Pod CIDR
Service CIDR
ack-hangzhou
cn-hangzhou
vpc-hangzhou
10.0.0.0/16
172.16.0.0/16
ack-shanghai
cn-shanghai
vpc-shanghai
10.1.0.0/16
172.17.0.0/16

Step 1: Create clusters in different regions

Create VPCs and vSwitches in Hangzhou and Shanghai regions according to the configurations above. For more information, see Create a vSwitch and Create a VPC and a vSwitch.
Use the VPCs that you created to create clusters in the Hangzhou and Shanghai regions. For more information, see Create an ACK managed cluster.
Create an ASM instance in the Hangzhou region according to the configurations above. For more information, see Create an ASM instance.

Step 2: Use CEN to implement inter-region VPC network connectivity

Use CEN to connect the VPC networks between clusters, along with the VPC networks between clusters and Service Mesh.

Log on to the Cloud Enterprise Network (CEN) console, and create a CEN instance. For more information, see Create a CEN instance.
Create transit routers.
1. On the CEN Instances page, click the name of the CEN instance created in Step 1, and then under the Basic Information tab, click Create Transit Router.
2. Select a Region, enter a Name, and then click OK.
In the example, two transit routers are configured:
- Region: China (Shanghai); Name: shanghai-router.
- Region: China (Hangzhou); Name: hangzhou-router.
Add VPCs to both transit routers by repeating the following steps
1. Click the ID of the transit router.
2. Select the Intra-region Connections tab, and then click Create Connection.
3. For Instance Type, select Virtual Private Cloud (VPC), and for Network Instance, select the VPC instance corresponding to the region.
4. Keep other options as default, and click OK.
Configure the bandwidth for inter-region communication.
1. Click the name of the transit router that you created in Step 2, and then click Create Connection on the right side of the page.
2. In the Connection With Peer Network Instance dialog box, configure the related parameters, and then click OK.
  The following figure shows an example connection from Hangzhou to Shanghai. Region represents the region of the transit router, and Peer Region represents the destination region. For information about configuration items, see inter-region connections.
  After the creation is complete, you can view the instance on the Inter-region Connections tab.
Add security group rules.
The following example uses the Flannel network plugin to add the Pod network CIDR of the peer cluster to the security group of each cluster.
Note
For the Terway network plug-in, use the CIDR of the cluster vSwitch. You can log on to the VPC console, and obtain the CIDR of the vSwitch from the IPv4 CIDR Block column on the vSwitch page.
1. Log on to the ACK console. In the navigation pane on the left, click Clusters.
2. Obtain the Pod network CIDR.
  1. On the Clusters page, select the Shanghai region, click the ack-shanghai cluster name, and then on the Cluster Information page, click the Basic Information tab to view the Pod CIDR block of the ack-shanghai cluster.
  2. Return to the Clusters page, select the Hangzhou region, click the ack-hangzhou cluster name, and then on the Cluster Information page, click the Basic Information tab to obtain the Pod CIDR block of the ack-hangzhou cluster.
3. Configure security groups.
  1. On the Cluster Information page of each cluster, click the Basic Information tab, and then click the security group ID on the right of Control Plane Security Group.
  2. Click Add Rule under the Inbound tab.
  3. Set Protocol Type to All, Source to the Pod CIDR block of the peer cluster, retain other default values, and then click Save in the Actions column.
4. Log on to the nodes of both clusters and use the ping command to test whether the network is connected to the other cluster. For more information, see Log on to nodes.

Step 3: Add clusters to the ASM instance and create a hosted ingress gateway

Add the clusters from Hangzhou and Shanghai regions to the ASM instance. For more information, see Add a cluster to an ASM instance.

Create an ingress gateway.

apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  annotations:
    asm.alibabacloud.com/managed-by-asm: 'true'
  name: ingressgateway
  namespace: istio-system
spec:
  gatewayType: ingress
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  hostNetwork: false
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  replicaCount: 1
  resources:
    limits:
      cpu: '2'
      memory: 2G
    requests:
      cpu: 200m
      memory: 256Mi
  rollingMaxSurge: 100%
  rollingMaxUnavailable: 25%
  runAsRoot: true
  serviceType: LoadBalancer

Step 4: Deploy the Bookinfo demo application

Important

The following steps involve switching kubeconfig multiple times. We recommend that you configure the kubeconfig of both clusters in the same config file and use kubectl config use-context to switch between them. You can also use kubecm or kubectx to manage kubeconfig for multiple clusters.

Deploy the Bookinfo application in both the ack-hangzhou and ack-shanghai clusters.
```
kubectl apply -f bookinfo.yaml
```

Use kubectl to connect to the ASM instance and create routing rules.

Create asm.yaml with the following content.

Expand to view YAML content

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: productpage
spec:
  host: productpage
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v3
    labels:
      version: v3
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ratings
spec:
  host: ratings
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  - name: v2-mysql
    labels:
      version: v2-mysql
  - name: v2-mysql-vm
    labels:
      version: v2-mysql-vm
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: details
spec:
  host: details
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---

Create routing rules.
```
kubectl apply -f asm.yaml
```

Verify that the Bookinfo application is deployed successfully.
1. Get the ingress gateway address.
2. In the address bar of your browser, enter http://{IP address of the ingress gateway}/productpage and refresh the page 10 times to access the Bookinfo application.
  The Bookinfo application accesses the v1, v2, and v3 versions of the reviews service. You can see that the ratio of requests to the three versions of the reviews service is close to 1:1:1.

Step 5: Use inter-region failover and inter-region traffic distribution

Configure inter-region failover

Disable the review service in the ack-hangzhou cluster.
1. Log on to the ACK console. In the navigation pane on the left, click Clusters.
2. On the Clusters page, find the cluster you want to manage and click its name. In the left-side pane, choose Workloads > Deployments.
3. On the Deployments page, set Namespace to default, and click Scale in the Actions column.
4. Set Desired Number Of Pods to 0, and then click OK.
Configure a destination rule.
Configure a destination rule so that when the review service cannot be accessed within 1 second, it will be ejected for 1 minute.
1. On the details page of the ASM instance, choose Traffic Management Center > DestinationRule in the left-side navigation pane.
2. Click Edit YAML in the Actions column of reviews.
3. Add the following content, and then click OK.
```
spec:
  ......
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      interval: 1s
```
  - maxRequestsPerConnection: maximum number of connections.
  - baseEjectionTime: minimum ejection duration.
  - consecutive5xxErrors: number of consecutive errors.
  - interval: time interval for ejection detection.
Enable inter-region failover.
1. On the details page of the ASM instance, choose ASM Instance > Base Information in the left-side navigation pane.
2. For ASM instance version below 1.22.6.66
  1. On the Base Information page, click Setting to the right of Geolocation-based Load Balancing.
  2. In the Geolocation-based Failover dialog box, set the Failover to to cn-hangzhou when the Policy is from cn-shanghai, and set the Failover to to cn-shanghai when the Policy is from cn-hangzhou, then click Confirm.
  For ASM instance version 1.22.6.66 and above
  1. On the Base Information page, click Configure a Geolocation-based Load Balancing to the right of Geolocation-based Load Balancing.
  2. Click Specify priority rules for regions. set Region in which the failure occurs to cn-shanghai, and The region to which the traffic is preferentially routed to cn-hangzhou.
  3. Click Add, then set Region in which the failure occurs to cn-hangzhou, The region to which the traffic is preferentially routed to cn-shanghai, and then click Save Configuration.
Run the following command to request the Bookinfo application 10 times and count the number of results routed to the v2 version of the reviews service.
```
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done|wc -l
```
Expected output:
```
20
```
The output indicates that 10 access requests are made and 20 rows of results are returned (each time a request is routed to the v2 version of the reviews service, it returns two rows containing full stars). This indicates that all 10 requests are routed to the v2 version of the reviews service in the ack-shanghai cluster, and the inter-region failover succeeds.

Configure inter-region traffic distribution

Important

inter-region traffic distribution capability is only supported for instances with version ≥1.22.6.66.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
On the Base Information page, click Configure a Geolocation-based Load Balancing to the right of Geolocation-based Load Balancing.
Click Congifure a traffic distribution rule, set Source to cn-hangzhou, Destination to cn-shanghai, Traffic Percentage to 10%, and click Save Configuration.
Note
The load balancing based on geographic location is enabled in failover mode by default. You need to click Disable in the upper-right corner of the page before you can reselect and enable the traffic distribution mode.
Run the following command to request the Bookinfo application 10 times to verify whether the inter-region traffic distribution is successful.
```
for ((i=1;i<=10;i++));do curl http://<Port 80 of the IP address of the ingress gateway in the ack-hangzhou cluster>/productpage 2>&1|grep full.stars;done
```
Expected output:
```


```
The output shows that after 10 accesses, 2 lines of full stars are displayed, indicating that out of the 10 requests, 9 were routed to the v1 version of the reviews service in the ack-hangzhou cluster, and 1 was routed to the v2 version of the reviews service in the ack-shanghai cluster. This confirms that traffic is successfully routed to multiple clusters according to the configured weights.

FAQ

Why does adding a Kubernetes cluster to ASM fail even though I connected the cluster VPC networks through CEN?

When your clusters are in different regions, if you have not purchased a inter-region data transfer plan or have not correctly configured inter-region traffic when connecting cluster VPC networks, the ASM control plane will not be able to connect to the data plane clusters, resulting in failure to add clusters to ASM.

To resolve this issue, you need to reconfigure inter-region traffic settings in Cloud Enterprise Network (CEN) to connect the cluster VPC networks. For more information, see Step 2: Use CEN to implement inter-region VPC communication.