Set up multi-cluster disaster recovery with an ASM multi-master architecture - Alibaba Cloud Service Mesh

Background

A multi-master control plane architecture is a model for managing multiple Kubernetes clusters using a service mesh. In this architecture, each Service Mesh (ASM) instance manages the data plane components of its corresponding Kubernetes cluster and distributes configurations to the mesh proxies within that cluster. By sharing a common root certificate, these instances enable cross-cluster service discovery and communication.

Compared to adding multiple clusters to a single ASM instance, a multi-master control plane architecture offers the following advantages:

Lower configuration delivery latency: Multiple clusters often span different regions, zones, or VPCs. In this scenario, using multiple ASM instances that are geographically closer to their respective Kubernetes clusters provides faster configuration delivery to mesh proxies.
Improved configuration and environment isolation: Each cluster is managed by a separate ASM instance. This allows you to deploy different control plane resources, enabling canary releases or isolation for configurations and versions. When you upgrade ASM instances, you can update the control planes in batches, which improves the availability of your production environment.
Greater stability: In extreme scenarios, such as a zone or region outage, a single control plane connected to all clusters can become a single point of failure, preventing configuration synchronization. In a multi-master control plane architecture, mesh proxies in healthy regions or zones can still connect to their local control planes, ensuring that configuration delivery and mesh proxy startup are unaffected.

To build a multi-master control plane architecture, you must create multiple ASM instances that share the same ASM root certificate. The control plane uses this root certificate to sign identity certificates for the mesh proxies. By using a shared root certificate, mesh proxies connected to different ASM instances can establish mutual trust and communicate with each other using mTLS.

Prerequisites

Two ACK managed clusters, named cluster-1 and cluster-2, are required. Both clusters must have the Expose API server with EIP option enabled. For more information, see Create an ACK managed cluster.

Step 1: Create two ASM instances with a shared root certificate

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

On the Mesh Management page, click Create ASM Instance. The following table describes the key parameters.

Parameter	Value
Service mesh name	mesh-1.
Region	Select the same region as the cluster-1 cluster.
Istio version	Select v1.22.6.71-g7d67a80b-aliyun or a later version.
Kubernetes clusters	Select cluster-1. The VPC and vSwitch parameters populate automatically.

For more information about other parameters, see Create an ASM instance. After the instance is created, wait 2 to 3 minutes until the status of the mesh-1 instance changes to Running.

On the Mesh Management page, click Create ASM Instance again. The following table describes the key parameters.

Parameter	Value
Service mesh name	mesh-2.
Region	Select the same region as the cluster-2 cluster.
Istio version	Select v1.22.6.71-g7d67a80b-aliyun or a later version.
Kubernetes clusters	Select cluster-2. The VPC and vSwitch parameters populate automatically.
ASM root certificate	Click Show Advanced Settings, select Reuse an Existing Root Certificate of ASM Instance, and then select mesh-1 from the drop-down list.

For other parameters not specified in the table, use the same values as for mesh-1. After the instance is created, wait 2 to 3 minutes until the status of the mesh-2 instance changes to Running.

Step 2: Add clusters in service-discovery-only mode

After you complete Step 1, mesh-1 manages cluster-1, and mesh-2 manages cluster-2. To enable service discovery between the clusters, you must add the other cluster to each ASM instance in service-discovery-only mode. This enables each instance to discover the services and endpoints in the other cluster.

Add cluster-2 to the mesh-1 instance in service-discovery-only mode.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the mesh-1 instance. In the left-side navigation pane, choose Cluster & Workload Management (Data Plane) > Kubernetes Clusters and then click Add.
3. On the Add Kubernetes Cluster page, find cluster-2, and then click Add (For Service Discovery Only) in the Actions column for the cluster. In the dialog box that appears, click OK. After the cluster is added, on the Instance Information > Basic Information page, the status of the ASM instance changes to Updating. After a few seconds (the time required depends on the number of clusters that are added), click the icon in the upper-right corner of the page, and the status of the ASM instance changes to Running. On the Kubernetes Clusters page, you can view information about the added cluster.
Add cluster-1 to the mesh-2 instance in service-discovery-only mode.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the mesh-2 instance. In the left-side navigation pane, choose Cluster & Workload Management (Data Plane) > Kubernetes Clusters and then click Add.
3. On the Add Kubernetes Cluster page, find cluster-1 and click Add (For Service Discovery Only) in the Actions column. In the confirmation dialog box that appears, click OK. After you add the cluster, navigate to the Instance Information > Basic Information page. The status of the ASM instance changes to Updating. Wait a few seconds, and then click the refresh icon in the upper-right corner. The instance status changes to Running. You can view information about the added cluster on the Kubernetes Clusters page.
  
  On the Kubernetes Clusters page, the status for cluster-1 is For Service Discovery Only, Synced, and the status of cluster-2 is Running, Synced. This confirms that both clusters were successfully added to the ASM instance.

Important

When you add a Kubernetes cluster in service-discovery-only mode, the ASM instance discovers services and endpoints in the cluster but does not deploy any data plane components to the cluster. Any configuration changes made in the ASM instance are not applied to clusters added in this mode.

The service-discovery-only mode is intended only for building a multi-master control plane architecture. If you want an ASM instance to fully manage your Kubernetes cluster, add the cluster directly to the ASM instance. For more information, see Add a cluster to an ASM instance.

Step 3 (Optional): Configure multi-cluster networking

If the cluster-1 and cluster-2 Kubernetes clusters are in different networks, such as across different VPCs or regions, and the networks are not connected using Cloud Enterprise Network (CEN), you must configure multi-cluster networking in both ASM instances. You also need to deploy a cross-cluster mesh proxy for both cluster-1 and cluster-2 to ensure connectivity between the clusters. For more information about the cross-cluster mesh proxy, see Disaster recovery for multiple ACK clusters in different VPCs (by using an ASM cross-cluster mesh proxy).

In the mesh-1 instance, configure the network settings for cluster-1 and cluster-2.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the mesh-1 instance. In the left-side navigation pane, choose Cluster & Workload Management (Data Plane) > Kubernetes Clusters.
3. Click Multi-cluster Network Configurations and configure the network settings as follows:
  1. For cluster-1, set the logical network to network1 and enable access through the cross-cluster mesh proxy.
  2. For cluster-2, set the logical network to network2.
Services within the same logical network can communicate directly. Services in different logical networks must communicate through the cross-cluster mesh proxy. When you enable access through the cross-cluster mesh proxy, a LoadBalancer Service is automatically created, which incurs CLB fees.

In the mesh-2 instance, configure the network settings for cluster-1 and cluster-2.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the mesh-2 instance. In the left-side navigation pane, choose Cluster & Workload Management (Data Plane) > Kubernetes Clusters.
3. Click Multi-cluster Network Configurations and configure the network settings as follows:
  1. For cluster-1, set the logical network to network1.
  2. For cluster-2, set the logical network to network2 and enable access through the cross-cluster mesh proxy.
Services within the same logical network can communicate directly. Services in different logical networks must communicate through the cross-cluster mesh proxy. When you enable access through the cross-cluster mesh proxy, a LoadBalancer Service is automatically created, which incurs CLB fees.

Step 4: Deploy the sample application

This section describes how to deploy the sleep service and v1 of the helloworld service in cluster-1, and v2 of the helloworld service in cluster-2, as shown in the following figure. Services in the two clusters can access each other through the cross-cluster mesh proxy.

In both the mesh-1 and mesh-2 instances, enable sidecar automatic injection for the default namespace. For more information, see Manage global namespaces.

Use the following YAML manifest to create the sleep application and v1 of the helloworld application.

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---
apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helloworld
  labels:
    account: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  labels: 
    apps: helloworld
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      serviceAccount: helloworld
      serviceAccountName: helloworld
      containers:
      - name: helloworld
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-helloworld-v1:1.0
        imagePullPolicy: IfNotPresent 
        ports:
        - containerPort: 5000

Deploy this manifest to cluster-1. For more information, see Create a stateless Deployment.

Use the following YAML manifest to create v2 of the helloworld application.

YAML

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helloworld
  labels:
    account: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v2
  labels: 
    apps: helloworld
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v2
  template:
    metadata:
      labels:
        app: helloworld
        version: v2
    spec:
      serviceAccount: helloworld
      serviceAccountName: helloworld
      containers:
      - name: helloworld
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/examples-helloworld-v2:1.0
        imagePullPolicy: IfNotPresent 
        ports:
        - containerPort: 5000

Deploy this manifest to cluster-2. For more information, see Create a stateless Deployment.

Step 5: Verify cross-cluster communication

Use the kubeconfig file of cluster-1 to run the following command:

 kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 10); do curl helloworld:5000/hello; done;'

Expected output:

Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx

The output shows that requests are load-balanced between v1 and v2 of the helloworld service.

Run the following command to scale the number of replicas for the sleep application in cluster-1 to 0:
```
kubectl scale deploy sleep --replicas=0
```

Use the following YAML manifest to create the sleep application.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent

Deploy the preceding manifest to cluster-2. For more information, see Create a stateless Deployment.

On cluster-2, run the following command again:

 kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 10); do curl helloworld:5000/hello; done;'

Expected output:

Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v1, instance: helloworld-v1-7b888xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx
Hello version: v2, instance: helloworld-v2-7b949xxxxx-xxxxx

The output shows that requests are still load-balanced between v1 and v2 of the helloworld service.