Implement an end-to-end canary release based on Message Queue for Apache RocketMQ - Microservices Engine

This topic describes how to implement an end-to-end canary release based on Message Queue for Apache RocketMQ by using Microservices Engine (MSE) without modifications to business code. Message Queue for Apache RocketMQ is used to provide asynchronous throttling.

Prerequisites

MSE Microservices Governance Professional Edition is activated. For more information, see Activate Microservices Governance.
A Container Service for Kubernetes (ACK) cluster is created. For more information, see Create an ACK managed cluster.

Step 1: Enable MSE Microservices Governance for applications

Install the component of Microservices Governance in the ACK cluster.
1. Log on to the ACK console.
2. In the left-side navigation pane, choose Marketplace > Marketplace. On the Marketplace page, click the App Catalog tab.
3. On the App Catalog tab, search for ack-onepilot and click it.
4. On the details page of the component, click Deploy. In the panel that appears, select the created cluster from the Cluster drop-down list and click Next. Then, click OK.
  Note Namespace is set to ack-onepilot and cannot be modified.
If the ack-onepilot application appears in the ack-onepilot namespace, the component is installed.
Enable MSE Microservices Governance for a specified namespace.
1. Log on to the MSE console.
2. In the left-side navigation pane, choose Microservices Governance > Kubernetes Clusters.
3. On the Kubernetes Clusters page, enter the name of the cluster that you want to manage in the search box and click to find the cluster. Then, click Manage in the Actions column.
4. In the namespace list of the cluster details page, find the namespace and click Activate Microservices Governance in the Actions column.
5. In the Activate Microservices Governance message, click OK.

Step 2: Deploy applications to simulate a real scenario

To simulate a real scenario, deploy four business applications: spring-cloud-zuul, spring-cloud-a, spring-cloud-b, and spring-cloud-c. You also need to deploy the nacos-server application as a registry and the rocketmq-name-server application as a message service. You can visit alibabacloud-microservice-demo to obtain the source code of this demo.

The following figure shows the structure of this demo. Application calls involve Spring Cloud service calls and Dubbo service calls. Spring Cloud and Dubbo are the two most commonly used microservices frameworks in the market. spring-cloud-c produces RocketMQ messages for spring-cloud-a. When the messages are consumed by spring-cloud-a, new calls are initiated. The application calls comply with the standard use of Spring Cloud, Dubbo, and RocketMQ.

Before you deploy the applications, we recommend that you have a basic understanding of the call process in this demo. In the call process, spring-cloud-zuul receives a request from /A/dubbo and forwards the request to spring-cloud-a. Then, spring-cloud-a accesses spring-cloud-b by using the Dubbo protocol, and spring-cloud-b accesses spring-cloud-c in the same manner. After the request is received by spring-cloud-c, a message is produced and its environment tag and IP address are returned. spring-cloud-a consumes produced messages and calls spring-cloud-b by using the Spring Cloud protocol at the same time. Then, spring-cloud-b calls spring-cloud-c in the same manner and displays results in logs.

# When you access /A/dubbo,
# the return value is A[10.25.xx.xx] -> B[10.25.xx.xx] -> C[10.25.xx.xx].
# At the same time, spring-cloud-a receives the message and returns the following logs:

2021-12-28 10:58:50.301  INFO 1 --- [essageThread_15] c.a.mse.demo.service.MqConsumer
          : topic:TEST_MQ,producer:C[10.25.xx.xx],invoke result:A[10.25.xx.xx] -> B[10.25.xx.xx] -> C[10.25.xx.xx]

In this topic, the ACK console is used to deploy applications. You can also use kubectl to deploy applications.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
In the left-side navigation pane of the details page, choose Workloads > Deployments.

On the Deployments page, select a namespace and then click Create from YAML.

The YAML file contains the following content:

# Deploy the nacos-server application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos-server
spec:
  selector:
    matchLabels:
      app: nacos-server
  template:
    metadata:
      annotations:
      labels:
        app: nacos-server
    spec:
      containers:
        - env:
            - name: MODE
              value: "standalone"
          image: registry.cn-shanghai.aliyuncs.com/yizhan/nacos-server:latest
          imagePullPolicy: IfNotPresent
          name: nacos-server
          ports:
            - containerPort: 8848

---
apiVersion: v1
kind: Service
metadata:
  name: nacos-server
spec:
  type: ClusterIP
  selector:
    app: nacos-server
  ports:
    - name: http
      port: 8848
      targetPort: 8848

# Deploy the business applications.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-zuul
spec:
  selector:
    matchLabels:
      app: spring-cloud-zuul
  template:
    metadata:
      labels:
        app: spring-cloud-zuul
        msePilotCreateAppName: spring-cloud-zuul
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
            - name: enable.mq.invoke
              value: 'true'
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-zuul:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-zuul
          ports:
            - containerPort: 20000

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: slb.s1.small
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
  name: zuul-slb
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 20000
  selector:
    app: spring-cloud-zuul
  type: LoadBalancer
status:
  loadBalancer: {}

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
spec:
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      labels:
        app: spring-cloud-a
        msePilotCreateAppName: spring-cloud-b
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-a
          ports:
            - containerPort: 20001
          livenessProbe:
            tcpSocket:
              port: 20001
            initialDelaySeconds: 10
            periodSeconds: 30


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b
spec:
  selector:
    matchLabels:
      app: spring-cloud-b
  template:
    metadata:
      labels:
        app: spring-cloud-b
        msePilotCreateAppName: spring-cloud-b
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-b
          ports:
            - containerPort: 20002
          livenessProbe:
            tcpSocket:
              port: 20002
            initialDelaySeconds: 10
            periodSeconds: 30

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  selector:
    matchLabels:
      app: spring-cloud-c
  template:
    metadata:
      labels:
        app: spring-cloud-c
        msePilotCreateAppName: spring-cloud-c
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-c
          ports:
            - containerPort: 20003
          livenessProbe:
            tcpSocket:
              port: 20003
            initialDelaySeconds: 10
            periodSeconds: 30
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rockectmq-broker
spec:
  selector:
    matchLabels:
      app: rockectmq-broker
  template:
    metadata:
      labels:
        app: rockectmq-broker
    spec:
      containers:
        - command:
            - sh
            - mqbroker
            - '-n'
            - 'mqnamesrv:9876'
            - '-c /home/rocketmq/rocketmq-4.5.0/conf/broker.conf'
          env:
            - name: ROCKETMQ_HOME
              value: /home/rocketmq/rocketmq-4.5.0
          image: registry.cn-shanghai.aliyuncs.com/yizhan/rocketmq:4.5.0
          imagePullPolicy: Always
          name: rockectmq-broker
          ports:
            - containerPort: 9876
              protocol: TCP
            - containerPort: 10911
              protocol: TCP
            - containerPort: 10912
              protocol: TCP
            - containerPort: 10909

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-name-server
spec:
  selector:
    matchLabels:
      app: rocketmq-name-server
  template:
    metadata:
      labels:
        app: rocketmq-name-server
    spec:
      containers:
        - command:
            - sh
            - mqnamesrv
          env:
            - name: ROCKETMQ_HOME
              value: /home/rocketmq/rocketmq-4.5.0
          image: registry.cn-shanghai.aliyuncs.com/yizhan/rocketmq:4.5.0
          imagePullPolicy: Always
          name: rocketmq-name-server
          ports:
            - containerPort: 9876
              protocol: TCP
            - containerPort: 10911
              protocol: TCP
            - containerPort: 10912
              protocol: TCP
            - containerPort: 10909
              protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: mqnamesrv
spec:
  type: ClusterIP
  selector:
    app: rocketmq-name-server
  ports:
    - name: mqnamesrv-9876-9876
      port: 9876
      targetPort: 9876

You can also run the following command to deploy the applications:

kubectl get svc,deploy

Expected output:

➜  ~ kubectl get svc,deploy
NAME                   TYPE           CLUSTER-IP        EXTERNAL-IP    PORT(S)        AGE
service/kubernetes     ClusterIP      192.168.xx.xx       <none>         4xx/TCP        7d
service/mqnamesrv      ClusterIP      192.168.xx.xx    <none>         98xx/TCP       47h
service/nacos-server   ClusterIP      192.168.xx.xx    <none>         88xx/TCP       47h
service/zuul-slb       LoadBalancer   192.168.xx.xx   123.56.xx.xx   80:302xxx/TCP   47h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nacos-server           1/1     1            1           4m
deployment.apps/rockectmq-broker       1/1     1            1           4m
deployment.apps/rocketmq-name-server   1/1     1            1           5m
deployment.apps/spring-cloud-a         1/1     1            1           5m
deployment.apps/spring-cloud-b         1/1     1            1           5m
deployment.apps/spring-cloud-c         1/1     1            1           5m
deployment.apps/spring-cloud-zuul      1/1     1            1           5m

Step 3: Enable the RocketMQ-based canary release feature for applications

In this example, spring-cloud-c produces messages and spring-cloud-a consumes the messages. Enable the RocketMQ-based canary release feature for the two applications.

Note

After you enable or disable the RocketMQ-based canary release feature for an application, you must re-deploy the application in the ACK console to make the change take effect.
Canary release takes effect on messages only when you enable the RocketMQ-based canary release feature for both the producers and consumers of the messages.
Canary release is available for only RocketMQ messages, including Apache RocketMQ messages and Message Queue for Apache RocketMQ messages.
- If you want to use Apache RocketMQ messages, RocketMQ Server 4.5.0 or later and RocketMQ Client 4.5.0 or later must be used.
- If you want to use Message Queue for Apache RocketMQ messages, Platinum Edition and ons-client 1.8.0 Final or later must be used.
After you enable the RocketMQ-based canary release feature, MSE modifies the consumer groups of messages. For example, if a consumer group named group1 is tagged with gray. After you enable the RocketMQ-based canary release feature, the consumer group is renamed as group1_gray. If you use Message Queue for Apache RocketMQ, you must create a consumer group in advance.
By default, MSE uses SQL-92 to filter messages. If you use Apache RocketMQ, you must enable SQL-92 filtering on the server side by setting enablePropertyFilter to true in the broker.conf file.

Log on to the MSE console.
In the left-side navigation pane, choose Microservices Governance > Applications.
On the Applications page, click the name of the application for which you want to enable the RocketMQ-based canary release feature to go to the application details page.
On the application details page, click the Message Canary Release tab. On the Message Canary Release tab, turn on Enable Message Canary Release, and then click OK.
Re-deploy the application in the ACK console to make the change take effect.

Step 4: Deploy canary versions of the applications

Deploy spring-cloud-a-gray, spring-cloud-b-gray, and spring-cloud-c-gray in the ACK cluster. spring-cloud-a-gray, spring-cloud-b-gray, and spring-cloud-c-gray are the canary versions of spring-cloud-a, spring-cloud-b, and spring-cloud-c.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
In the left-side navigation pane of the details page, choose Workloads > Deployments.

On the Deployments page, select a namespace and then click Create from YAML.

The YAML file contains the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-a-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
      labels:
        app: spring-cloud-a-gray
        msePilotCreateAppName: spring-cloud-a
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-a-gray
          ports:
            - containerPort: 20001
          livenessProbe:
            tcpSocket:
              port: 20001
            initialDelaySeconds: 10
            periodSeconds: 30
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-b-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray 
      labels:
        app: spring-cloud-b-gray
        msePilotCreateAppName: spring-cloud-b
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-b-gray
          ports:
            - containerPort: 20002
          livenessProbe:
            tcpSocket:
              port: 20002
            initialDelaySeconds: 10
            periodSeconds: 30

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c-gray
spec:
  selector:
    matchLabels:
      app: spring-cloud-c-gray
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
      labels:
        app: spring-cloud-c-gray
        msePilotCreateAppName: spring-cloud-c
    spec:
      containers:
        - env:
            - name: JAVA_HOME
              value: /usr/lib/jvm/java-1.8-openjdk/jre
          image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:1.0.0
          imagePullPolicy: Always
          name: spring-cloud-c-gray
          ports:
            - containerPort: 20003
          livenessProbe:
            tcpSocket:
              port: 20003
            initialDelaySeconds: 10
            periodSeconds: 30

Step 5: Ingest traffic and perform verification

In the left-side navigation pane of the MSE console, choose Microservices Governance > Applications.
On the Applications page, click the spring-cloud-a application. On the application details page, the displayed information shows that all traffic is routed to the untagged version of spring-cloud-a. The untagged version is considered a stable version.
In the lower part of the application details page, click the Tag-based Routing tab. Find the version that is tagged with gray and click Add in the Traffic Rules column.
In the Create Tag-based Routing panel, configure the parameters and click OK.
In this example, name=xiaoming is configured as the traffic rule.
For more information about traffic rules, see Configure tag-based routing.
If the traffic rule takes effect, you can view the traffic distribution on the application details page.
On the details page of spring-cloud-a, click the Tag-based Routing tab. Find the version that is tagged with gray and click the name of the traffic rule that corresponds to the version in the Traffic Rules column. In the Rule Details panel, click Edit. In the Modify Tag-based Routing panel, turn on Link Delivery and click OK.
Note After link delivery is enabled, canary traffic that meets the traffic rule name=xiaoming can be routed within all canary environments involved in the call process. You do not need to configure other traffic rules.
In the left-side navigation pane of the cluster details page in the ACK console, choose Network > Services.
Click the public endpoint of the zuul-slb service.
On the service call page, enter /A/a?name=xiaoming and then click Start Calling.
The end-to-end canary release takes effect.

Step 6: Adjust the tag-based message filtering rule and perform verification

In the left-side navigation pane of the MSE console, choose Microservices Governance > Applications.
On the Applications page, click spring-cloud-a to go to the application details page.
In the lower part of the application details page, click the Message Canary Release tab, set Tags Ignored by Untagged Environment to gray, and then click OK.
If you set Tags Ignored by Untagged Environment to gray, messages that are tagged with gray can be consumed only by spring-cloud-a-gray, but not spring-cloud-a. After you set the Tags Ignored by Untagged Environment parameter, the configuration immediately takes effect. You do not need to restart the application.
Note By default, untagged applications consume messages from all environments. If you do not want one untagged application to consume messages that are produced by tagged environments, you can set the Tags Ignored by Untagged Environment parameter.
Log on to the ACK console to view the logs of spring-cloud-a and verify configurations.
The following figure shows that the base environment can consume messages that are produced from both the canary and base environments. The figure also shows that the Spring Cloud calls from the canary or base environment are routed to the downstream canary or base environment.