How to configure message canary release - Microservices Engine

Enable message canary release for scenarios such as end-to-end canary release and development environment isolation. MSE currently supports message canary release only for RocketMQ.

Background information

In most business scenarios, the requirements for message canary release are less strict than those for Remote Procedure Calls (RPCs). However, end-to-end message canary release is still necessary in the following scenarios.

After modifying the message consumption logic, you can use a small amount of traffic to verify the new logic. This requires message canary release.
During message consumption, new RPCs may be generated. If these calls do not follow the predefined end-to-end traffic shaping rules, the traffic generated by the messages can "escape". This violates the end-to-end canary release rules and can lead to unexpected results.

How it works

The MSE service governance feature for message canary release works by adding the current environment context to message attributes when an application sends a message. During message consumption, messages are filtered based on this environment context in the attributes.

Definition of a canary message: A canary message is a message generated in one of the following two ways.

Messages directly generated by a canary producer: Messages produced by a canary node (canary producer) are considered canary messages by default.
Fallback messages from a baseline node: When no canary producer nodes exist in the system, canary traffic falls back to the baseline nodes. Messages produced by baseline nodes in this case are also considered canary messages.

By default, after you enable message canary release, baseline messages from the baseline environment are consumed only by baseline consumers. Canary messages are consumed by both baseline consumers and their corresponding canary consumers. As a result, both the baseline and canary consumers independently consume the same canary message.

If you want messages produced by applications in a canary environment to be consumed only by the corresponding canary consumers, see Configure canary message consumers.

Limits

Framework support

Message type: Only RocketMQ is supported, including the open source and Alibaba Cloud commercial versions.
For RocketMQ Server:
- If you use the open source version, you must use RocketMQ Server 4.5.0 or later.
- If you use the Alibaba Cloud RocketMQ 4.x series, you must use the Platinum Edition to enable server-side filtering. For other editions, you can use client-side filtering.
- If you use the Alibaba Cloud RocketMQ 5.x series, both server-side and client-side filtering are supported.
For RocketMQ Client:
- If you use the Ons Client provided by Alibaba Cloud RocketMQ, you must use a 1.x series version that is later than 1.8.0.Final.
- If you use the org.apache.rocketmq:rocketmq-client client from open-source RocketMQ, the supported version of org.apache.rocketmq:rocketmq-client is any 4.x version later than 4.2.0. This client supports both server-side and client-side filtering.
- If you use the org.apache.rocketmq:rocketmq-client-java client from open source RocketMQ, use a version in the 5.x series for org.apache.rocketmq:rocketmq-client and version 4.2.2 or later for the MSE Agent. This client only supports server-side filtering.
- For information about the differences between the SDKs for open-source RocketMQ, see the RocketMQ official documentation.
The MSE Agent does not currently support transactional messages.

Feature limits

Message canary release takes effect only when the feature is enabled for both message producers and consumers in MSE service governance.
After you enable message canary release, the MSE Agent modifies the Consumer Group of message consumers in the canary environment.
Note
If you use Alibaba Cloud RocketMQ, you must create the Consumer Group for the corresponding canary environment before you enable message canary release.
1. Modification rule: For a canary environment, `_tag` is appended to the original Consumer Group, where `tag` is the corresponding environment tag.
2. For example, if the original Consumer Group is `group1` and the environment tag is `gray`, the Consumer Group is changed from `group1` to `group1_gray` after you enable message canary release.
MSE uses SQL-92 filtering for server-side filtering. If you use open source RocketMQ 4.x, you must enable SQL-92 filtering support on the RocketMQ Server. To do this, set enablePropertyFilter=true in the `broker.conf` file.
If your scenario does not meet the conditions for SQL-92 filtering, you can use client-side filtering. To do this, select the client-side filtering method when you enable message canary release in MSE service governance. Note that client-side filtering requires each environment to process all messages, which places a high load on both producers and consumers. We do not recommend using client-side filtering in a production environment.

Enable message canary release

Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose Microservices Governance > Application Governance.
On the Application list page, click the resource card of the desired application. In the left-side navigation pane, click Traffic management.
On the Traffic management page, click the Message grayscale tab.
To the right of Open message grayscale, click Edit. Configure the message canary release, filtering settings, and consumption rules. Then, click OK.
Note
- After you change the message canary release settings, you must restart all nodes of the corresponding application for the changes to take effect. This applies to both client-side and server-side filtering.
- Message canary release is a sub-feature of end-to-end canary release. For best results, we recommend that you also configure the related end-to-end canary release capabilities when you enable this feature.

Configure canary message consumers

If you want messages from a canary environment to be consumed only by canary consumers, you must prevent baseline consumers from consuming them. To do this, you can configure the consumption rules for the canary tag. After this configuration, baseline consumers will only process baseline messages and any canary messages for which the baseline consumer is explicitly selected.

Procedure:

Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose Microservices Governance > Application Governance.
On the Application list page, click the resource card of the desired application. In the left-side navigation pane, click Traffic management.
On the Traffic management page, click the Message grayscale tab.
In the Message Canary Release Configuration section, click Edit. In the consumption rule configuration, clear the Baseline Consumer checkbox for the corresponding tag. Then, click OK.

Note

Changes to the consumption rule configuration take effect dynamically. You do not need to restart the application.
Because the baseline environment no longer consumes messages with the ignored tag, if all consumers in the corresponding canary environment go offline, canary messages with that tag will not be consumed.

Canary release workflows

Scenario 1: Permanent deployment of canary consumers

In scenarios with long-term deployment of canary message consumers, you can connect producers and consumers to microservice governance and enable message canary release. This ensures that canary messages are consumed by canary consumers and baseline messages are consumed by baseline consumers. This entire process requires only a single configuration, making it efficient and easy to maintain.

Note: If your messages are not idempotent, you must ensure that canary messages are consumed by canary consumers and baseline messages are consumed by baseline consumers. To do this, follow these steps.

Prerequisites: Both the message producer and consumer applications must be connected to microservice governance. You must also restart the applications after enabling message canary release.

Configure consumption rules for the consumer. For the tag corresponding to canary messages, clear the Baseline Consumer checkbox.
After performing the operation in the MSE console, wait 1 to 3 minutes to ensure that the subscription relationship on the RocketMQ side is updated.
Bring the canary nodes of the message consumer online.
Bring the canary nodes of the message producer online and send canary messages.

Scenario 2: Dynamic release of canary consumers

In scenarios where canary nodes are released with application iterations, the canary nodes are temporary and may be enabled or disabled depending on the release phase. To prevent message accumulation or loss, you must adjust the filtering behavior of baseline consumer nodes based on the status of the canary nodes. Follow this procedure during the release:

Modify the consumption rule configuration for the consumer application. For the tag corresponding to canary messages, clear the Baseline Consumer checkbox so that canary messages are consumed only by canary consumers. After performing the operation in the MSE console, wait 1 to 3 minutes to ensure that the subscription relationship on the RocketMQ side is updated.
Bring the canary nodes of the message consumer application online.
Send canary messages from the canary nodes of the message producer application.
Verify the canary messages.
Take the canary message producer nodes offline.
Wait until the offset of the baseline consumer_group is less than or equal to the offset of the canary consumer_group. Use logs and monitoring to confirm that all previous canary messages have been successfully consumed.
Note
If the offset of the baseline consumer_group is less than or equal to the offset of the canary consumer_group, the remaining canary messages can be consumed by baseline consumers and will not be lost. You can then proceed to the next step. In this scenario, some canary messages will be consumed repeatedly. You must ensure that your business logic handles message idempotency.
If the offset of the baseline consumer_group is greater than the offset of the canary consumer_group, canary messages will be lost if you proceed to the next step. Therefore, you must wait.
Modify the consumption rule configuration for the consumer application. For the tag corresponding to canary messages, select the Baseline Consumer checkbox so that the remaining canary messages can be consumed by baseline consumers. After performing the operation in the MSE console, wait 1 to 3 minutes to ensure that the subscription relationship on the RocketMQ side is updated.
Take the canary message consumers offline and delete the canary consumer_group.
Note
If you do not delete the canary consumer_group, messages will accumulate in that group. The next time canary consumers are brought online during an iteration, they will start consuming from the last recorded canary consumer offset.