Microservice Full Link Grayscale New Capabilities

Background

In the micro-service architecture, the dependency relationship between services is complex. Sometimes the release of a function depends on the simultaneous upgrade of multiple services. We hope that the new versions of these services can be verified at the same time by small traffic grayscale. This is the unique full-link grayscale scenario in the microservice architecture. We can verify the grayscale of multiple different versions of services by building an environment isolation from the gateway to the entire back-end service. During the release process, we only need to deploy the grayscale version of the service. When the traffic flows on the call link, the grayscale traffic is identified by the gateway, each middleware and each microservice that flows through, and is dynamically forwarded to the grayscale version of the corresponding service. As shown below:

The above figure can well show the effect of this scheme. We use different colors to represent different versions of grayscale traffic. It can be seen that both the micro-service gateway and the micro-service itself need to identify traffic and make dynamic decisions according to governance rules. When the service version changes, the forwarding of this call link will also change in real time. Compared with the gray-scale environment built by machines, this scheme can not only save a lot of machine costs and operation and maintenance manpower, but also help developers to carry out real-time and rapid fine full-link control of online traffic.

Full-link grayscale capability improves the advantages of fast iteration and stability verification brought by microservice architecture, and brings real benefits to the system of enterprise production environment. This article will focus on the application scenarios and pain points of MSE service governance based on the full link gray level capability, which has extended new capabilities.

Full-link operation white-screen capability

In the process of using full link grayscale in our production environment, we often encounter some problems.

• Whether we configure the flow direction of full-link grayscale traffic to meet the expectations, and whether our traffic is matched according to the grayscale rules we configure.

• There are a lot of slow calls and exceptions in our grayscale traffic. How can I determine whether it is the business problem of our new version of code or the system problem caused by our incomplete consideration in the grayscale process of traffic, and how to quickly locate the problem so as to achieve efficient iteration.

• In the process of designing the grayscale system, we need to consider how to mark our grayscale traffic. Sometimes it may be difficult to find the appropriate traffic characteristics (parameters, headers, and other logos with business semantics) in the entrance application and microservice interface. In such a scenario, how can we quickly mark our traffic.

Based on the problems listed above, we also encounter problems in the process of supporting cloud customers to land in full link grayscale. The white-screen capability at runtime is also a capability that we abstractly designed in this process.

The purpose of white-screen operation is to help us gain insight into the flow matching and operation behavior of the whole link.

Based on the traffic routing rules, we abstract the white-screen rules at runtime as follows:

WhiteScreenRule = Taget + Action

Target：

• ResourceTarget: target interface, supporting Web, Rpc and custom methods

• WorkloadTarget: target instance, you can select all machines or specify machine IP

• TrafficCondition: only for exception, slow call and full-link grayscale labels

Action：

• Collection of relevant contextual diagnostic information

• Follow up links for traffic coloring

• Whether the subsequent links are log printed

Let's take a detailed look at how to use the white-screen capability at runtime to solve the problems we encounter in the grayscale process of the whole link.

Matching of grayscale flow and whether the flow direction meets the expectation

For the above scenario, we only need to configure the white-screen matching rules of Zuul application portal:

We can quickly observe the parameters, return values, headers and other characteristic attributes of grayscale traffic in the whole link. We can also quickly find out whether the whole link meets the expectation and locate the reason why it does not meet the expectation.

Full-link configuration grayscale

In addition to the grayscale of micro-service instances and traffic, the configuration items in micro-service applications should also have corresponding grayscale capabilities to meet the requirements of grayscale applications for special configuration values.

Microservice applications usually introduce the configuration center for configuration management, which provides dynamic configuration push capability, so that applications can dynamically change the running logic without restarting. However, the management dimension of the configuration center is only the configuration item itself, and it cannot sense the environment information of the service instance coming to get the configuration, that is, it cannot distinguish whether the instance requesting configuration is a formal environment instance or a grayscale environment instance. In this context, if a configuration needs to use different values in the formal environment and grayscale environment, they must be used as different configuration items in the configuration center. We may need to write the following code:

This scenario is very common in A/B testing. With the increase of configuration items and grayscale environment, this code will be repeated many times. In addition, there are often multiple services in a grayscale environment, and each service needs to maintain a set of similar codes independently. The final solution is shown in the figure. The configuration values used by the same configuration item in different environments need to be actively differentiated in user applications.

The reason is that the configuration center is unable to perceive the environment information of the service instance, which makes us have to perform this task in the code instead of the configuration center, resulting in the intrusion of the environment information into the business code.

To solve this problem, the configuration tag push function of MSE sinks the perception of environment information in the configuration management scenario to the platform side, and the agent is responsible for it. Users can easily use the configuration push capability in the full-link grayscale scenario by simply accessing MSE, and the cumbersome environment information detection logic in the business code is eliminated. As shown in the figure:

For specific operation steps, please refer to the microservice governance practitioners: https://help.aliyun.com/practice_detail/447313

Step 1: Restore the online scene

We will deploy four business applications and registration centers, namely, spring-cloud-zuul, spring-cloud-a, spring-cloud-b, and spring-cloud - c. The call link is as follows:

These applications are the simplest Spring Cloud and Dubbo applications. You can get the project source code at the following link:

https://github.com/aliyun/alibabacloud-microservice-demo/tree/master/mse-simple-demo

Log in to the MSE governance center console, click application governance in the left navigation bar, enter cfg-spring-cloud-a, and click the search icon. Select the cfg-spring-cloud-a application card to enter the application details page.

Then select Application Configuration>Configuration List in the left navigation bar, click+before the switch configValue, and you can see that the configuration values of the baseline version and the grayscale version in the A application are the same, both are the initial values in the application.

Step 2: Configure and apply the grayscale rules of spring-cloud-a

On the application details page of cfg-spring-cloud-a, select the traffic governance in the left navigation bar, and click the label route. As shown in the figure:

Next, we configure grayscale rules for grayscale instances. Click the label gray>traffic rules>add, configure the following gray rules, and click OK.

Step 3: Verify the configuration grayscale

Next, let's verify the configuration grayscale.

1. Execute label push for grayscale instance

Go back to the application configuration>configuration list on the application details page of cfg-spring-cloud-a, and select the push by tag after the switch configValue. Select the label gray in the pop-up window and set the configuration value in the grayscale environment.

Then click Next: value comparison, and then click Tag Push to complete the push. At this time, you can see on the console that the configuration value of the grayscale instance has changed to the value we just set.

2. Verify that the configuration grayscale is effective

Log in to the container service console, click the cluster on the left navigation bar, and enter the cluster where the application is deployed. In the cluster details page, select Network>Service, find zuul-slb, click its external endpoint, and access the service invocation page.

You can access the baseline version of App A. You can see that its configuration value is still the initial value.

You can access the grayscale version of App A through the grayscale rule you just configured. You can see that its configuration value has changed to the value we just pushed.

3. Verify the persistence of grayscale configuration values

The configuration value of label push is persistent. This means that even if the application in the grayscale environment is restarted, it can automatically obtain the previously pushed configuration values from the MSE Agent.

Log in to the container service console, click the cluster on the left navigation bar, and enter the cluster where the application is deployed. In the cluster details page, select Workload>Stateless. Check the spring-cloud-a-gray load and click Batch Redeploy. You can also use the kubectl tool to redeploy the load and simulate the restart process of grayscale applications.

After the application is restarted, the process of accessing the grayscale application in the previous step is re-executed. It can be found that the configuration value is still the configuration value pushed previously.

Summary

This paper introduces the ability of white-screen and gray-scale configuration at runtime based on the full-link gray-scale extension, which improves the scene of full-link gray-scale and further improves the usability of full-link gray-scale. Full-link grayscale is an important scenario in micro-service governance. The full-link grayscale capability of MSE is still expanding and iterating with the deepening of customer scenarios. We need to continue to invest in making such an important scenario more in-depth and easier to use. We can foresee that there are still many ways to continue to explore the polishing of the full-link grayscale capability. At present, the full-link grayscale has been used by nearly 100 enterprises, We have always believed that only products that have been continuously polished by customers will become more and more lasting. If you are also interested, welcome to use and experience