Custom tag-based canary release in MSE Nacos - Microservices Engine

To address the limitations of IP address-based canary release, Microservices Engine (MSE) Nacos optimizes the canary release feature. The optimization allows you to implement a canary release based on custom tags. MSE supports both IP address-based canary release and custom tag-based canary release to provide you with more flexible and easy-to-use canary release capabilities and to ensure business stability.

Background information

Widespread use of configuration centers

Configuration center is one of the core features of Nacos instances. You can use configuration centers to modify application configurations online without restarting applications. This decreases the software development and deployment costs, improves operations efficiency, and accelerates business development.

Nacos instances are widely used in multiple fields, such as microservices, high availability, frontend, and databases. Configuration centers play a key role in scenarios, such as traffic management, route settings, emergency response, business switches, and database configurations. This ensures the accuracy of these configurations, which is critical to the stable operation of online businesses.

Risks of configuration changes

Dynamic configuration changes can significantly improve the software iteration efficiency, but also pose stability risks. Misoperations on online configurations may lead to errors in online program execution. As a result, your business is adversely affected. If the change process is not perfect, dynamic configuration changes extend the impact scope of online stability issues from a specific perspective. Failures caused by configuration changes occasionally occur in the industry. Therefore, it is necessary to control configuration changes.

For stability reasons, the system must have the following basic capabilities for changes:

Canary release: You can perform a canary release to control the effective scope of any changes. During the change process, you can gradually extend the effective scope.
Monitoring: Business metrics are displayed in a visualized manner to accurately reflect the running status of businesses.
Rollback: If a business exception is detected during the change, the business can be quickly rolled back to the status before the change.

Canary release is an essential and basic feature of a configuration center to control the risks of configuration changes.

The following figure shows a standard configuration change process.

Select specific low-risk application servers to perform a canary release of the first version, deliver new configurations, and then observe the system and business metrics of the servers. If you confirm that your business works as expected, you can gradually extend the canary release range until the official version is released to complete the entire configuration change process. If business exceptions occur during the change process, terminate the canary release and roll back the configurations of the servers to the configurations before the change for fast business recovery.

Limitations of IP address-based canary release

In earlier versions, Nacos instances provide the IP address-based canary release. Before you release the configurations of the official version, you can select specific IP addresses to deliver new configurations for verification and check whether your business is normal. If no business exceptions occur during the check period, you can perform a full release after the check period elapses.

For small-sized applications, IP address-based canary release is well suited. However, for medium- and large-sized applications, a configuration may be subscribed to by a large number of application nodes. These nodes may be from different businesses and have different impacts on online businesses. If a large number of node IP addresses are available, operations are inconvenient. IP addresses are determined by the underlying network infrastructure and are difficult to directly associate with specific businesses. When you extend the canary release range, you must manually filter IP addresses to obtain the IP addresses that meet specific business requirements. In addition, in a Kubernetes container environment, the IP address changes after a node restart may cause the canary release policy to become invalid.

Custom tag-based canary release

Custom tag-based canary release allows you to configure different tags for different nodes of an application based on business scenarios. Tags allow you to categorize application nodes by object such as application, data center, or environment. For example, if nodes of an application in Alibaba Group are deployed in different development branches, you can configure different tags for the application nodes that are deployed in different development branches. Then, you can use the custom tag-based canary release capability to deliver different configurations in different project environments. The following content describes how to enable the custom tag-based canary release capability for your application.

Supported versions

To implement the custom tag-based canary release feature, make sure that the version of the open source Nacos client is 2.3.2 or later. If the Nacos client version is earlier than 2.3.2, upgrade the Nacos client version.
To use environment variables to configure tags, you must upgrade the version of the Nacos client to 2.4.2 or later.
For MSE Nacos instances, their versions must be upgraded to 2.2.3.3 or later to support the custom tag-based canary release capability.

Single-tag canary release

For some small- and medium-sized applications, you can use a single-tag canary release to meet the requirements of a custom tag-based canary release.

Configure tags for application nodes on the client

After you upgrade the client and server versions to the required versions, you can configure custom tags for your application to implement a canary release.

Configure application tags

Application tags are in the form of key-value pairs. You can use one of the following methods to configure application tags.

Use Properties: nacos.config.gray.label is the default built-in property that is used to configure canary tags in Nacos instances. You can use nacos.config.gray.label={value} to configure application tags.

Properties properties = new Properties();
properties.put(PropertyKeyConst.SERVER_ADDR, "{server addr}");
// Add other properties.
properties.put("nacos.config.gray.label", "yourgrayname");
ConfigFactory.createConfigService(properties);

Use JVM parameters: Add the configuration -Dnacos.config.gray.label={value} to the application startup command.
```
java -Dnacos.config.gray.label=yourgrayname -jar your-app.jar
```
Use environment variables: Set nacos_config_gray_label to {value} in environment variables.
```
export nacos_config_gray_label=yourgrayname
```

Effective scope and priorities of the methods

Properties: This method takes effect only on the current ConfigService object. The effective scope is small.
JVM parameters: This method takes effect on all ConfigService objects in the current process.
Environment variables: This method takes effect on all ConfigService objects on the current node.
Priorities: If duplicate tag keys exist after you use the three methods, the tag keys that are obtained by using the methods are read in the following order of precedence: Properties > JVM parameters > Environment variables.

Implement a custom tag-based canary release on the server

View listener tags

After you configure application tags on the client and restart your application, you can view the configured listener list in the MSE Nacos console and view the tags carried by each listener.

Configure application tags for canary release

On the Configurations page of the MSE Nacos console, find the desired configuration, and click Edit in the Actions column. In the Edit Configuration panel, select Tag-based Canary Release for Release Type, specify Canary Version Name, and then select the tag key-value pair of the application node. You can view the expected quantity and list of matched application nodes. In the right-side section, edit the configuration content in the Configuration Content code editor, and click Release.

After the tag-based canary version is released, go to the configuration details page, and click the Configuration Content tab. The Official and Beta subtabs are displayed.

On the Listening Query tab, you can view the currently matched version of the application node. The displayed version is in the format of Beta({Canary version name}). MD5 is MD5 of the version, and Push Status indicates that the push operation is successful.

On the Push track tab, you can also view the tracking information of the tag-based canary release and the queries of canary versions on the client. You can also view the configuration version that is pushed in the configuration push tracking details of the node.

After a canary release is performed on the first batch of nodes and the canary version is proven stable, you can extend the value range of the tag to gradually extend the canary release range until a full release is performed. After you click Full Release, the relevant canary version is terminated. If a business exception occurs during the canary release, you can click Stop Canary Release to trigger automatic rollback.

The following content describes the methods that you can use to gradually extend the canary release range.

Method 1: Implement a canary release based on application tags

Configure different tags for different applications: Add the nacos.config.gray.label=A tag for all servers of Application A, and add the nacos.config.gray.label=B tag for all servers of Application B.
Implement an initial canary release: Implement a canary release on Application A that is tagged with nacos.config.gray.label=A.
Extend the canary release range: After you confirm that the new version is properly running on Application A, you can update the tag value to extend the canary release range. For example, you can change the tag from nacos.config.gray.label=A to nacos.config.gray.label=A,B. This way, the canary release is also implemented on Application B.

Method 2: Implement a canary release based on environment groups

Divide machines in a production environment into multiple groups, and specify different tags for the groups, such as nacos.config.gray.label=gray1, nacos.config.gray.label=gray2, and nacos.config.gray.label=gray3. Keep nacos.config.gray.label=online as the tag used for the online environment that is not involved in the canary release.
Change the tag value in the configuration file to control the environment or group to which the latest canary version is pushed.
At the end of each stage, check whether the canary version is stable. If the canary version is stable, add servers with the same tag on which the canary release is implemented until the canary release is implemented on all objects in the environment or group.

Subsequent steps

After you implement a canary release on a small number of nodes, monitor relevant business metrics.
If the business metrics are normal, you can further extend the effective scope of the canary release until the canary release is implemented on all expected nodes.
If all canary release results show that the new version is stable and reliable, you can implement a full release and change the current canary version to the official version. After the official version is released, the current canary version is terminated.

This method allows developers to introduce new features or optimizations in a more strict and controlled manner. This reduces potential risks and improves the user experience.

Multi-tag canary release

When you handle complex medium- and large-sized applications, a single tag may be difficult to meet the diversity of business requirements. MSE provides a flexible tag management mechanism. This mechanism does not impose strict limits on tag keys. You can configure tag keys based on your business characteristics and configure multiple tags for the same application to implement finer-grained service management and canary release policies. The following content describes how to add and manage multiple tag key-value pairs for your business nodes. This helps you cope with complex business scenarios.

Configure application tags on the client

Similar to configuring a single tag, you can also configure multiple tags by using Properties, JVM parameters, or environment variables.

Use Properties: Configure the nacos.app.conn.labels property. Format: nacos.app.conn.labels=k1=v1,k2=v2,k3=v3

//1. Use Properties.
Properties properties = new Properties();
properties.put(PropertyKeyConst.SERVER_ADDR, "your endpoint");
properties.put("project.name", "your app name");
properties.put("nacos.app.conn.labels","app=demo,site=hangzhou-c,otherkey=othervaue");

Use JVM parameters: Configure the nacos.app.conn.labels parameter. Format: nacos.app.conn.labels=k1=v1,k2=v2,k3=v3

//2. Configure JVM parameters.
Configure the startup parameter -Dnacos.app.conn.labels=app=demo,site=hangzhou-c,otherkey=othervaue

Use environment variables: Configure the nacos_app_conn_labels environment variable. Format: nacos_app_conn_labels=k1=v1,k2=v2,k3=v3.

//3. Configure environment variables.
Configure the nacos_app_conn_labels environment variable. Example: nacos_app_conn_labels=app=demo,site=hangzhou-c,otherkey=othervaue.

NacosConfigService configService = new NacosConfigService(properties);

String dataId = "gray_test_dataid";
String group = "test-group";
        
configService.addListener(dataId, group, new Listener() {
        @Override
        public Executor getExecutor() {
            return null;
        }
            
        @Override
        public void receiveConfigInfo(String configInfo) {
            System.out.println("receiveConfig:" + configInfo);
        }
});

In addition to the preceding parameter settings, you can implement a service provider interface (SPI) to configure tags if the logic for reading tags is complex. The Nacos client provides the SPI mechanism that is used to define a custom application tag. To implement this mechanism, you must implement the com.alibaba.nacos.common.labels.LabelsCollector interface, and use the implementation class to publish the interface as an SPI in the META-INF/services/ directory. This can control and extend the tag collection and processing logic in a flexible manner and meet business requirements in different scenarios.

Sample code for configuring custom tags

package your.demo.test;

import com.alibaba.nacos.common.labels.LabelsCollector;

import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

/**
 * TestLabelsCollector.
 *
 * @author yourname
 */
public class TestLabelsCollector implements LabelsCollector {
    
    @Override
    public String getName() {
        return "testlables";
    }
    
    @Override
    public Map<String, String> collectLabels(Properties properties) {
        Map<String, String> labels = new HashMap<>();
        labels.put("test", "implements your lables logic");
        return labels;
    }

  
    @Override
    public int getOrder() {
        return 1;
    }
}

Usage notes:

To ensure that applications are properly running when they use tags for canary release. MSE Nacos instances impose the following limits on the keys and values of tags:

Allowed characters: The key and value parts of a tag can contain only letters, digits, underscores (_), hyphens (-), and periods (.). The tags that contain characters other than the preceding allowed characters are ignored by the system.
Format requirements: If you specify multiple key-value pairs for the nacos.app.conn.labels parameter, the key-value pairs must be listed in the k1=v1,k2=v2,k3=v3 format. For example, if the input is k1=v1,k2, only the kay-value pair k1=v1 is identified and k2 is ignored due to no value available.
Upper limit on the number of tag key-value pairs: You can configure up to 10 tag key-value pairs for an application node. If the number of tag key-value pairs exceeds 10, the system removes random extra tags. You must take this limit into considerations when you plan the tags required for an application.

You must comply with the following matching rules for canary release:

The current version allows you to filter nodes based on different tag values. However, to reduce the complexity of the feature, you cannot specify multiple tag keys in the same canary release rule for compound matching. For example, you cannot use a logical expression, such as k1=v1&&k2=v2 or k1=v1||k2=v2 to accurately control or extend the test range of the canary release.

Support for multiple canary versions

Scenarios

MSE Nacos instances allow you to release multiple canary versions of one configuration and provide custom configurations for different application nodes based on different conditions. For example, you can deliver the graycontent1 configuration content to the nodes that meet the k1=v1 condition and deliver the graycontent2 configuration content to the nodes that meet the k1=v2 condition.

Procedure

In the Edit Configuration panel, click + to create a canary version.

Important

MSE specifies the maximum number of canary versions that can be released for a configuration. By default, a maximum of five canary versions are supported. If more than five canary versions are available, MSE blocks the extra canary versions.

After you create multiple canary versions, you can view the information about the canary versions that are displayed as tabs in the Edit Configuration panel.

Note

Canary versions are displayed in the order of precedence. You can explicitly view the priority of each version. To change the priority of a canary version, change the priority value in the Edit Configuration panel, and click Release to make the new settings take effect.

If multiple canary versions exist, select one of the versions to implement a full release. After the release, the canary version is changed to the official version. After the official version is successfully released, the selected canary version is automatically terminated. Note that other canary versions that are in the running state are not adversely affected. The nodes that correspond to the running canary versions can still receive the configuration content of their respective canary versions.

Priorities of multiple canary versions

If your configuration has multiple versions, including the official version, IP address-based canary version, and tag-based canary versions, the system may match non-overlapping nodes or the same application node by using tag key-value pairs of canary versions. One or more tag-based canary versions are supported. Nacos servers determine the versions that are applied to specific nodes based on the following order of precedence: IP address-based canary version > Tag-based canary versions > Official version. The following priority rules are applied:

IP address-based canary version: If a node meets both IP address-based canary release conditions and other types of canary release conditions, the node preferentially uses the IP address-based canary version.
Tag-based canary versions: If IP address-based canary release conditions are not met but tag-based canary release conditions are met, Nacos servers check the priority field value that is defined in each tag-based canary version. The larger the priority value, the higher the priority. If the priority value is the same, the match is based on the character sequence of the canary version name. We recommend that you configure different rules for different priorities.
Official version: The official version is used only if no canary versions are applied to the current node.

Note

To ensure more precise control over the canary release process, we recommend that you configure different priority values for different purposes and plan your canary release policies based on your business requirements. This can help you flexibly manage the impact of configuration changes on the system.

References

For more information about how to configure a canary release in MSE, see Configure canary release.