Microservices Engine:End-to-end canary release with RocketMQ

Demo architecture

This topic provides an example of deploying applications in the ACK console. You can also use kubectl to deploy applications and simulate a real-world call chain. The following figure shows the architecture of the demo application. The calls between applications use both Spring Cloud and Dubbo, the two most common microservice frameworks. In this architecture, Application C produces RocketMQ messages that are consumed by Application A. When Application A consumes a message, it initiates a new call. These applications are based on standard implementations of Spring Cloud, Dubbo, and RocketMQ.

Step 1: Onboard applications to MSE Microservices Governance

• Enable Microservices Governance for microservice applications in the Container Service for Kubernetes (ACK) cluster. You can select an appropriate method based on your business requirements. For more information, see Enable Microservices Governance for Java microservice applications in an ACK or ACS cluster.

  Enable Microservices Governance for applications in an ACK namespace

  1. Log on to the MSE console, and select a region in the top navigation bar.

  2. In the left-side navigation pane, choose Microservices Governance > Application Governance.

  3. On the Application list page, click ACK Application Access.

  4. In the ACK Application Access dialog box, configure the parameters and click OK.

     Set Onboarding Type to Namespace or Single Application. From the Container Cluster Namespace drop-down list, select the target namespace, such as default. In the Governance Namespace field, enter a name for the corresponding namespace.

     Parameter	Description
     Cluster type	Select ACK Cluster, ACK Serverless Cluster, or ACS Cluster. Note If you have not authorized Container Service for Kubernetes to call MSE, click Authorize Now to grant the required permissions.
     Cluster Name/ID	Select the Cluster Name/ID of the cluster that you want to connect to the MSE governance center. You can search for the cluster by keyword.
     ack-onepilot	Displays the status of the ack-onepilot component. For more information about this component and how to upgrade it, see ack-onepilot component and Install and upgrade the MSE microservices governance component. If ack-onepilot is not installed, the system automatically starts the installation after you select a cluster and displays the message "The installation process takes about 1 minute. Wait for a while." If you use a RAM user and receive a permission error, log on to the Container Service for Kubernetes console. Go to the target cluster, and in the navigation pane on the left, click Add-ons. Find ack-onepilot and click Install. If ack-onepilot is already installed, the UI displays the installed version, for example, Installed 4.2.0. Note You can view details about the ack-onepilot component by logging on to the Container Service for Kubernetes console, navigating to the target cluster, and choosing Operations > Add-ons. After ack-onepilot is installed, it automatically injects a Java agent. This may increase application startup time by up to 10 seconds. When connecting by namespace, if your target cluster is not in one of the listed regions, ensure it has public network access and can connect to acm.aliyun.com:8080: China (Qingdao), China (Hangzhou), China (Beijing), China (Shanghai), Shanghai Finance Cloud, China (Shenzhen), Hong Kong (China), Singapore, Germany (Frankfurt), Australia (Sydney), US (Silicon Valley), and US (Virginia).
     Access Type	Select Namespace Access.
     Cluster Namespace	Select the Cluster Namespace.
     Microservices Governance Namespace	Select a Microservices Governance Namespace.

  Enable Microservices Governance for an application

  1. Log on to the MSE console, and select a region in the top navigation bar.

  2. In the left-side navigation pane, choose Microservices Governance > Application Governance.

  3. On the Application list page, click ACK Application Access.

  4. In the ACK Application Access dialog box, configure the parameters and click OK.

     Set Onboarding Type to Single Application. In the Edit Labels (YAML format) section, add the following labels under spec.template.metadata.labels in the application's Deployment YAML file: msePilotAutoEnable: "on" to enable microservices governance, mseNamespace: default to specify the MSE namespace, and msePilotCreateAppName: "your-deployment-name" to set the application name. Replace your-deployment-name with the actual Deployment name.

     Parameter	Description
     Cluster type	Select ACK Cluster, ACK Serverless Cluster, or ACS Cluster. Note If you have not authorized Container Service for Kubernetes to call MSE, click Authorize Now to grant the required permissions.
     Cluster Name/ID	Select the Cluster Name/ID of the cluster that you want to connect to the MSE governance center. You can search for the cluster by keyword.
     ack-onepilot	Displays the status of the ack-onepilot component. For more information about this component and how to upgrade it, see ack-onepilot component and Install and upgrade the MSE microservices governance component. If ack-onepilot is not installed, the system automatically starts the installation after you select a cluster and displays the message "The installation process takes about 1 minute. Wait for a while." If you use a RAM user and receive a permission error, log on to the Container Service for Kubernetes console. Go to the target cluster, and in the navigation pane on the left, click Add-ons. Find ack-onepilot and click Install. If ack-onepilot is already installed, the UI displays the installed version, for example, Installed 4.2.0. Note You can view details about the ack-onepilot component by logging on to the Container Service for Kubernetes console, navigating to the target cluster, and choosing Operations > Add-ons. After ack-onepilot is installed, it automatically injects a Java agent. This may increase application startup time by up to 10 seconds. When connecting by namespace, if your target cluster is not in one of the listed regions, ensure it has public network access and can connect to acm.aliyun.com:8080: China (Qingdao), China (Hangzhou), China (Beijing), China (Shanghai), Shanghai Finance Cloud, China (Shenzhen), Hong Kong (China), Singapore, Germany (Frankfurt), Australia (Sydney), US (Silicon Valley), and US (Virginia).
     Access Type	Select Single Application Access.
     Access Procedure	Follow these steps. Step 1: In the Container Service for Kubernetes console, navigate to the Workloads > Deployments page of your cluster and switch to your application's namespace. Step 2: Find the target application and click Edit YAML. Step 3: Edit the labels field as shown in the following example and click Update. spec: template: metadata: labels: # Set to "on" to enable connection. The value must be enclosed in double quotation marks. msePilotAutoEnable: "on" # Specify the target governance namespace. If the namespace does not exist, it is automatically created. mseNamespace: default # Specify the application name to be displayed in MSE. The value must be enclosed in double quotation marks. msePilotCreateAppName: "your-deployment-name"

Step 2: Deploy the applications

Deploy the four business applications (spring-cloud-zuul, spring-cloud-a, spring-cloud-b, and spring-cloud-c), a Nacos Server for service registration, and a RocketMQ Server for messaging. You can find the source code for these components in this demo.

The spring-cloud-zuul application forwards requests from its /A/dubbo path to spring-cloud-a. Then, spring-cloud-a uses the Dubbo protocol to call spring-cloud-b, which in turn calls spring-cloud-c using the same protocol. After receiving the request, spring-cloud-c produces a message and returns its environment tag and IP address. These messages are consumed by the spring-cloud-a application. When consuming a message, spring-cloud-a calls spring-cloud-b by using Spring Cloud, which then calls spring-cloud-c. The final result is written to the logs of spring-cloud-a.

# When a request is sent to /A/dubbo, the returned value is: A[10.25.xx.xx] -> B[10.25.xx.xx] -> C[10.25.xx.xx]
# At the same time, after Application A receives the message, the following log is generated:

2021-12-28 10:58:50.301  INFO 1 --- [essageThread_15] c.a.mse.demo.service.MqConsumer
          : topic:TEST_MQ,producer:C[10.25.xx.xx],invoke result:A[10.25.xx.xx] -> B[10.25.xx.xx] -> C[10.25.xx.xx]

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Cluster List page, click the name of the destination cluster or click Details in the Actions column.

3. In the left navigation pane of the cluster management page, choose Workload > Deployments.

4. On the Stateless page, select a Namespaces and click Create from YAML. Use the following YAML file to deploy the applications:

   YAML file

   # Deploy Nacos Server
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nacos-server
   spec:
     selector:
       matchLabels:
         app: nacos-server
     template:
       metadata:
         labels:
           app: nacos-server
       spec:
         containers:
           - env:
               - name: MODE
                 value: "standalone"
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/nacos-server:v2.1.2
             imagePullPolicy: IfNotPresent
             name: nacos-server
             ports:
               - containerPort: 8848
   
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nacos-server
   spec:
     type: ClusterIP
     selector:
       app: nacos-server
     ports:
       - name: http
         port: 8848
         targetPort: 8848
   
   # Deploy business applications
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-zuul
   spec:
     selector:
       matchLabels:
         app: spring-cloud-zuul
     template:
       metadata:
         labels:
           app: spring-cloud-zuul
           msePilotCreateAppName: spring-cloud-zuul
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
               - name: enable.mq.invoke
                 value: 'true'
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-zuul:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-zuul
             ports:
               - containerPort: 20000
   
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: slb.s1.small
       service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
     name: zuul-slb
   spec:
     ports:
       - port: 80
         protocol: TCP
         targetPort: 20000
     selector:
       app: spring-cloud-zuul
     type: LoadBalancer
   status:
     loadBalancer: {}
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-a
   spec:
     selector:
       matchLabels:
         app: spring-cloud-a
     template:
       metadata:
         labels:
           app: spring-cloud-a
           msePilotCreateAppName: spring-cloud-a
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-a:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-a
             ports:
               - containerPort: 20001
             livenessProbe:
               tcpSocket:
                 port: 20001
               initialDelaySeconds: 10
               periodSeconds: 30
   
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-b
   spec:
     selector:
       matchLabels:
         app: spring-cloud-b
     template:
       metadata:
         labels:
           app: spring-cloud-b
           msePilotCreateAppName: spring-cloud-b
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-b:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-b
             ports:
               - containerPort: 20002
             livenessProbe:
               tcpSocket:
                 port: 20002
               initialDelaySeconds: 10
               periodSeconds: 30
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-c
   spec:
     selector:
       matchLabels:
         app: spring-cloud-c
     template:
       metadata:
         labels:
           app: spring-cloud-c
           msePilotCreateAppName: spring-cloud-c
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-c:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-c
             ports:
               - containerPort: 20003
             livenessProbe:
               tcpSocket:
                 port: 20003
               initialDelaySeconds: 10
               periodSeconds: 30
   ---
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: rockectmq-broker
   spec:
     selector:
       matchLabels:
         app: rockectmq-broker
     template:
       metadata:
         labels:
           app: rockectmq-broker
       spec:
         containers:
           - command:
               - sh
               - mqbroker
               - '-n'
               - 'mqnamesrv:9876'
               - '-c /home/rocketmq/rocketmq-4.9.7/conf/broker.conf'
             env:
               - name: ROCKETMQ_HOME
                 value: /home/rocketmq/rocketmq-4.9.7
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/rocketmq:4.9.7
             imagePullPolicy: Always
             name: rockectmq-broker
             ports:
               - containerPort: 9876
                 protocol: TCP
               - containerPort: 10911
                 protocol: TCP
               - containerPort: 10912
                 protocol: TCP
               - containerPort: 10909
   
   ---
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: rocketmq-name-server
   spec:
     selector:
       matchLabels:
         app: rocketmq-name-server
     template:
       metadata:
         labels:
           app: rocketmq-name-server
       spec:
         containers:
           - command:
               - sh
               - mqnamesrv
             env:
               - name: ROCKETMQ_HOME
                 value: /home/rocketmq/rocketmq-4.9.7
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/rocketmq:4.9.7
             imagePullPolicy: Always
             name: rocketmq-name-server
             ports:
               - containerPort: 9876
                 protocol: TCP
               - containerPort: 10911
                 protocol: TCP
               - containerPort: 10912
                 protocol: TCP
               - containerPort: 10909
                 protocol: TCP
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: mqnamesrv
   spec:
     type: ClusterIP
     selector:
       app: rocketmq-name-server
     ports:
       - name: mqnamesrv-9876-9876
         port: 9876
         targetPort: 9876
                           

5. Run the following command to check if the applications are deployed successfully:

   kubectl get svc,deploy

   Expected output:

   NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
   service/kubernetes     ClusterIP      192.168.xx.xx    <none>         4xx/TCP        7d
   service/mqnamesrv      ClusterIP      192.168.xx.xx    <none>         98xx/TCP       47h
   service/nacos-server   ClusterIP      192.168.xx.xx    <none>         88xx/TCP       47h
   service/zuul-slb       LoadBalancer   192.168.xx.xx   123.56.xx.xx   80:302xxx/TCP   47h
   
   NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/nacos-server           1/1     1            1           4m
   deployment.apps/rockectmq-broker       1/1     1            1           4m
   deployment.apps/rocketmq-name-server   1/1     1            1           5m
   deployment.apps/spring-cloud-a         1/1     1            1           5m
   deployment.apps/spring-cloud-b         1/1     1            1           5m
   deployment.apps/spring-cloud-c         1/1     1            1           5m
   deployment.apps/spring-cloud-zuul      1/1     1            1           5m

Step 3: Enable message canary release

Assume that spring-cloud-c produces messages and spring-cloud-a consumes them. Enable canary release for messaging for both applications.

1. Log on to the MSE console, and select a region in the top navigation bar.

2. In the left-side navigation pane, choose Microservices Governance > Application Governance.

3. On the Application list page, click the resource card of the desired application. In the left-side navigation pane, click Traffic management.

4. On the Canary Release for Messaging tab, turn on the switch and click OK.

   Configure Tags Ignored by Untagged Environment to gray, and set Message Canary Release Filter to Server-side Filtering (Recommended).

5. In the ACK console, redeploy the application in the ACK console for the configuration to take effect.

Step 4: Deploy the new application versions

Deploy the canary versions of the applications: spring-cloud-a-gray, spring-cloud-b-gray, and spring-cloud-c-gray.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Cluster List page, click the name of the destination cluster or click Details in the Actions column.

3. In the left navigation pane of the cluster management page, choose Workload > Deployments.

4. On the Stateless page, select a Namespaces and click Create from YAML. Use the following YAML file to deploy the applications:

   YAML file

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-a-gray
   spec:
     selector:
       matchLabels:
         app: spring-cloud-a-gray
     template:
       metadata:
         labels:
           alicloud.service.tag: gray
           app: spring-cloud-a-gray
           msePilotCreateAppName: spring-cloud-a
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-a:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-a-gray
             ports:
               - containerPort: 20001
             livenessProbe:
               tcpSocket:
                 port: 20001
               initialDelaySeconds: 10
               periodSeconds: 30
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-b-gray
   spec:
     selector:
       matchLabels:
         app: spring-cloud-b-gray
     template:
       metadata:
         labels:
           alicloud.service.tag: gray 
           app: spring-cloud-b-gray
           msePilotCreateAppName: spring-cloud-b
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-b:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-b-gray
             ports:
               - containerPort: 20002
             livenessProbe:
               tcpSocket:
                 port: 20002
               initialDelaySeconds: 10
               periodSeconds: 30
   
   ---
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spring-cloud-c-gray
   spec:
     selector:
       matchLabels:
         app: spring-cloud-c-gray
     template:
       metadata:
         labels:
           alicloud.service.tag: gray
           app: spring-cloud-c-gray
           msePilotCreateAppName: spring-cloud-c
       spec:
         containers:
           - env:
               - name: JAVA_HOME
                 value: /usr/lib/jvm/java-1.8-openjdk/jre
             image: registry.cn-hangzhou.aliyuncs.com/mse-governance-demo/spring-cloud-c:3.0.1
             imagePullPolicy: Always
             name: spring-cloud-c-gray
             ports:
               - containerPort: 20003
             livenessProbe:
               tcpSocket:
                 port: 20003
               initialDelaySeconds: 10
               periodSeconds: 30

Step 5: Route traffic and verify

1. Log on to the MSE console, and select a region in the top navigation bar.

2. In the left-side navigation pane, choose Microservices Governance > Application Governance.

3. Click the resource card for the spring-cloud-a application. On the Application Overview page, observe the QPS Trend Chart chart to confirm that all traffic is routed to the base environment of the spring-cloud-a application, which is the stable version.

4. In the left navigation bar, click Traffic management, and then click the Tag-based Routing tab. In the Traffic Rule column for the gray tag, click Add.

5. In the Create Tag-Based Routing panel, configure the traffic rule, and then click OK.

   In this example, the traffic rule is set to name=xiaoming. For more information about how to configure traffic rules, see Configure tag-based routing.

   On the Create Tag Route page, configure the following settings:

   • Route Name: test-a

   • Application: spring-cloud-a

   • Tag: gray

   • Link Delivery: Enabled

   • Framework Type: Spring Cloud

   • Path: /dubbo

   • Condition Mode: All conditions must be met

   After the traffic rule takes effect, you can view the traffic distribution on the Application Overview page.

6. On the Tag-based Routing tab, click the name of the rule associated with the gray tag. In the Rule details panel that appears, click Edit. In the Edit Tag-Based Routing panel, enable Trace Propagation and click OK.

   Note

   After you enable context propagation, canary traffic that matches the name=xiaoming rule propagates throughout the entire call chain. You do not need to configure the rule for each application.

7. Log on to the ACK console. In the left navigation pane, click Clusters.

8. On the Clusters page, click the name of your cluster. In the left navigation pane, click Network > Services.

9. Click the zuul-slb service. In the Basic Information section, click the address next to External IP.

10. On the service call page, enter /A/dubbo?name=xiaoming and click Start Call.

    The end-to-end canary release is now active.

    On the test page, enter the URL /A/dubbo?name=xiaoming and click Start Invocation. The invocation log shows all requests routed through canary nodes: Agray[10.5.117.199] → Bgray[10.5.117.200] → Cgray[10.5.117.157].

    The following figure shows the request data for the spring-cloud-b application.

    

Step 6: Adjust message tag filtering and verify

1. Log on to the MSE Microservices Governance console.

2. In the navigation pane on the left, choose Microservices Governance > Application Governance, and then click the resource card for the spring-cloud-a application.

3. In the left-side navigation pane, click Traffic management, and then click the Canary Release for Messaging tab.

4. On the Canary Release for Messaging tab, click Edit next to Tags Ignored in Base Environment. In the input box, select gray and click OK.

   Selecting gray for the Tags to ignore in the base environment parameter means that messages with the gray environment tag can be consumed only by spring-cloud-a-gray and not by spring-cloud-a. The configuration for the Tags to ignore in the base environment parameter takes effect dynamically, and you do not need to perform a restart.

   Note

   By default, a base environment node consumes messages from all environments. To prevent it from consuming messages produced by a specific tagged environment, simply configure the Tags to be ignored by the base environment parameter.

5. Log on to the ACK console and view the logs for the spring-cloud-a application to verify the configuration.

   The logs show that before this change, the base environment consumed messages from both the gray and base environments. After setting the Tags Ignored by Base Environment parameter to gray, the base environment consumes only messages produced in the base environment.

   In the MqConsumer log of the spring-cloud-a container, canary messages are routed through Bgray→Cgray, while baseline messages are routed through A→B→C, following separate paths.

   2021-12-28 11:51:47.880  INFO 1 --- [essageThread_11] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:Cgray[10.xxx 2],invoke result:A[10.xxx,xxx100] ->
   Bgray[xxx] -> Cgray[xxx]
   
   2021-12-28 11:51:47.942  INFO 1 --- [essageThread_10] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.xxx],invoke result:A[10.xxx 00] -> B[1
   0.25.xxx] -> C[xxx 7]
   
   2021-12-28 11:51:48.783  INFO 1 --- [MessageThread_8] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.2xxx 7],invoke result:A[10.xxx] -> B[1
   0.25xxx] -> C[10.xxx 7]
   
   2021-12-28 11:51:48.885  INFO 1 --- [essageThread_12] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:Cgray[10.2xxx 2],invoke result:A[10.2xxx] ->
   Bgray[10.25 xxx] -> Cgray[10.25xxx102]
   
   2021-12-28 11:51:50.908  INFO 1 --- [essageThread_17] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.xxx.157],invoke result:A[10.2xxx 00] -> B[1
   0.25.xxx] -> C[10.2xxx57]
   
   2021-12-28 11:51:50.909  INFO 1 --- [essageThread_16] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:Cgray[10.25.xxx 2],invoke result:A[10.2xxx100] ->
   Bgray[10.xxx25] -> Cgray[10.xxx.02]
   
   2021-12-28 11:51:51.642  INFO 1 --- [essageThread_14] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:Cgray[10.25xxx],invoke result:A[10.25.xxx] ->
   Bgray[10.xxx.25] -> Cgray[10.xxx 102]
   
   2021-12-28 11:51:51.684  INFO 1 --- [essageThread_15] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.25xxx57],invoke result:A[10.xxx0] -> B[1
   0.25.xxx] -> C[10.xxx 7]
   
   2021-12-28 11:51:52.765  INFO 1 --- [essageThread_13] c.a.mse.demo.service.MqConsumer          : topic:TEST_MQ,producer:C[10.25.xxx],invoke result:A[10.25.xxx] -> B[1