Disaster recovery scenario for multiple ACK clusters in the same VPC - Alibaba Cloud Service Mesh

When you use Service Mesh (ASM) to manage multiple clusters, workloads for a service can be deployed across them. By default, traffic to the service is load-balanced across the clusters. To keep traffic within a local cluster, you can use the cluster-local traffic retention feature. This topic uses the Bookinfo application as an example to demonstrate how to enable the cluster-local traffic retention feature of ASM in a multi-cluster scenario.

Prerequisites

Create two ACK clusters in the same Virtual Private Cloud (VPC). This topic uses m1c1 and m1c2 as examples. For more information, see Create an ACK dedicated cluster (new creations are discontinued).
Create an ASM instance of version 1.15.3.101 or later. This topic uses mesh1 as an example. For more information, see Create an ASM instance.

Preparations

Step 1: Configure connectivity between clusters

(Optional) Modify the security group names of the clusters

You can change the names of the security groups for the two clusters to make them easier to identify. This simplifies the subsequent configuration. In this example, the Security Group is set to m1c1-sg and m1c2-sg.

Expand to view how to modify the security group name of a cluster

Log on to the ECS console. In the navigation pane on the left, choose Network & Security > Security Groups.
In the top menu bar, select the region where the target ECS instance is located.
On the Security Groups page, hover over the name of the target security group, click the icon, enter a new name, and then click Confirm.

Add a security group rule

Add security group rules to allow the two clusters to access each other. On the configuration page for the m1c1-sg security group, add a rule to authorize access from m1c2-sg. Similarly, on the configuration page for the m1c2-sg security group, add a rule to authorize access from m1c1-sg. For more information, see Add a security group rule.

Step 2: Add the clusters to the ASM instance and deploy an ingress gateway

After you add both clusters to the ASM instance, they can access each other. Therefore, you need to deploy an ingress gateway for only one of the clusters.

Add the two clusters to the ASM instance. For more information, see Add a cluster to an ASM instance.
Deploy an ingress gateway for the m1c1 cluster. For more information, see Create an ingress gateway.

Step 3: Deploy the Bookinfo application in each of the two clusters

This topic uses the deployment of the Bookinfo application in two separate clusters as an example to demonstrate the ASM cluster-local traffic retention feature. The only difference between the services in the two clusters is the version of the Reviews component. The m1c1 cluster uses v1, and the m1c2 cluster uses v2. All other components are the same.

Use the following YAML file to deploy the Bookinfo application with the v1 Reviews deployment in m1c1. For more information, see Deploy an application in a cluster associated with an ASM instance.

The v1 Reviews deployment does not show star ratings in the book review section of the web page.

Expand to view the YAML file for the Bookinfo application in m1c1

apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
        cluster: m1c1
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-details-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
        cluster: m1c1
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-ratings-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v1
  labels:
    app: reviews
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v1
  template:
    metadata:
      labels:
        app: reviews
        version: v1
        cluster: m1c1
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-reviews-v1:1.20.1
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
        cluster: m1c1
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-productpage-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir: {}
---

Use the following YAML file to deploy the Bookinfo application with the v2 Reviews deployment in m1c2.

The v2 Reviews deployment shows black and white star ratings in the book review section of the web page.

Expand to view the YAML file for the Bookinfo application in m1c2

apiVersion: v1
kind: Service
metadata:
  name: details
  labels:
    app: details
    service: details
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: details
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-details
  labels:
    account: details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
        cluster: m1c2
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-details-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Ratings service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: ratings
  labels:
    app: ratings
    service: ratings
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: ratings
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-ratings
  labels:
    account: ratings
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratings-v1
  labels:
    app: ratings
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratings
      version: v1
  template:
    metadata:
      labels:
        app: ratings
        version: v1
        cluster: m1c2
    spec:
      serviceAccountName: bookinfo-ratings
      containers:
      - name: ratings
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-ratings-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
---
##################################################################################################
# Reviews service
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: reviews
  labels:
    app: reviews
    service: reviews
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: reviews
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-reviews
  labels:
    account: reviews
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: reviews-v2
  labels:
    app: reviews
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: reviews
      version: v2
  template:
    metadata:
      labels:
        app: reviews
        version: v2
        cluster: m1c2
    spec:
      serviceAccountName: bookinfo-reviews
      containers:
      - name: reviews
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-reviews-v2:1.20.1
        imagePullPolicy: IfNotPresent
        env:
        - name: LOG_DIR
          value: "/tmp/logs"
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: wlp-output
          mountPath: /opt/ibm/wlp/output
      volumes:
      - name: wlp-output
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
##################################################################################################
# Productpage services
##################################################################################################
apiVersion: v1
kind: Service
metadata:
  name: productpage
  labels:
    app: productpage
    service: productpage
spec:
  ports:
  - port: 9080
    name: http
  selector:
    app: productpage
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookinfo-productpage
  labels:
    account: productpage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
        cluster: m1c2
    spec:
      serviceAccountName: bookinfo-productpage
      containers:
      - name: productpage
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-bookinfo-productpage-v1:1.20.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        volumeMounts:
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: tmp
        emptyDir: {}
---

Step 4: Create a gateway rule, a virtual service, and a destination rule in ASM

Use the following YAML file to create a gateway rule in the default namespace of the ASM instance. For more information, see Manage gateway rules.

Expand to view the gateway rule YAML

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

Use the following YAML file to create a virtual service in the default namespace of the ASM instance. For more information, see Manage virtual services.

Expand to view the virtual service YAML

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo-cluster-local
spec:
  hosts:
  - "*"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        exact: /productpage1
    rewrite:
      uri: /productpage
    route:
    - destination:
        host: productpage
        port:
          number: 9080
        subset: m1c1
  - match:
    - uri:
        exact: /productpage2
    rewrite:
      uri: /productpage
    route:
    - destination:
        host: productpage
        port:
          number: 9080
        subset: m1c2
  - match:
    - uri:
        prefix: /static
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080

Use the following YAML file to create a destination rule in the default namespace of the ASM instance. For more information, see Manage destination rules.

Expand to view the destination rule YAML

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: productpage-cluster-local
spec:
  host: productpage
  subsets:
  - name: m1c1
    labels:
      cluster: m1c1
  - name: m1c2
    labels:
      cluster: m1c2

In a browser, access http://{INGRESS_GATEWAY_IP}/productpage1 or http://{INGRESS_GATEWAY_IP}/productpage2 and refresh the page.
To obtain the ingress gateway IP address, see Create an ingress gateway.
If star ratings appear alternately in the book review section, it indicates that the Bookinfo application is deployed successfully.

Procedure

The following procedure shows how to enable cluster-local traffic retention at the service level to keep traffic for the Reviews service within its local cluster. For information about how to enable this feature at the global or namespace level, see Related operations.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
On the Basic Information page, in the Config Info section, click Edit to the right of Keep traffic in cluster.
In the Configure services that keep traffic in cluster dialog box, turn on the Enable Keep traffic in cluster switch, select Select Service, and then click Select services.
In the Add services dialog box, select Service. Set Namespace to default. In the Select services section, select reviews, click the icon, and then click OK.
After the configuration is complete, click OK in the Configure services that keep traffic in cluster dialog box.
In the Config Info section, Configured appears to the right of Keep traffic in cluster. This indicates that the Keep traffic in cluster feature is enabled.
Verify that the cluster-local traffic retention feature is effective.
1. In a browser, access http://{INGRESS_GATEWAY_IP}/productpage1 and refresh the page.
  The book review section never shows ratings.
2. In a browser, access http://{INGRESS_GATEWAY_IP}/productpage2 and refresh the page.
  The book review section always shows the black and white star ratings.
These results indicate that the cluster-local traffic retention feature is effective.

Result analysis

The following diagram shows the traffic flow in the service mesh before and after the cluster-local traffic retention feature is enabled. 网格中流量的调用链路分析

Type

Description

Cluster-local traffic retention disabled

In the Bookinfo application, the Productpage component calls the Reviews service for book review information. Accessing http://{INGRESS_GATEWAY_IP}/productpage1 in a browser directs you to the Productpage in the m1c1 cluster. Accessing http://{INGRESS_GATEWAY_IP}/productpage2 directs you to the Productpage in the m1c2 cluster.

Workloads for the Reviews service exist in both the m1c1 and m1c2 clusters. By default, even when accessing the Productpage in a specific cluster like m1c1, requests to the Reviews service are load-balanced between both clusters. You will see star ratings appear alternately in the book review section of the Productpage.

Cluster-local traffic retention enabled

When you access a specific cluster, such as the Productpage in the m1c1 cluster, requests to the Reviews service are not load-balanced between the two clusters. The book review section of the Productpage page never shows ratings. The alternating star ratings no longer appear.

Important

After the traffic retention feature is enabled, if Reviews-v1 in the m1c1 cluster goes offline due to a failure, Productpage cannot provide the service by accessing Reviews-v2 in the m1c2 cluster.

Note

If you enable cluster-local traffic retention for a service, workloads from other services can only access the endpoints of that service within the same cluster. Even if the endpoints in the local cluster are unavailable, traffic is not routed to the service's endpoints in other clusters.

If you want to implement both cluster-local traffic retention and traffic degradation for a service, see Configure traffic lanes and traffic degradation based on traffic rules. You can use traffic rules to create traffic lanes and implement traffic degradation for services in different clusters.

Related operations

Enable cluster-local traffic retention globally

On the Basic Information page, in the Config Info section, click Edit to the right of Keep traffic in cluster.
In the Configure services that keep traffic in cluster dialog box, turn on the Enable Keep traffic in cluster switch, select All Services, and then click OK.
After you enable this feature, traffic for all services is retained within their local clusters.

Enable cluster-local traffic retention by namespace

On the Basic Information page, in the Config Info section, click Edit to the right of Keep traffic in cluster.
In the Configure services that keep traffic in cluster dialog box, turn on the Enable Keep traffic in cluster switch, select Select Service, and then click Select services.
In the Add services dialog box, select Namespace. In the Select services section, select the target namespace, click the icon, and then click OK.
In the Configure services that keep traffic in cluster dialog box, click OK.
After you enable this feature, traffic to services in the target namespace is retained within the local cluster.