How do we do full link grayscale on the database?

What is full link grayscale?

In the microservice architecture, the dependencies between services are complex, and sometimes the release of a function relies on multiple services being upgraded and launched simultaneously. We hope that small traffic grayscale verification can be performed on new versions of these services simultaneously, which is a unique full-link grayscale scenario in the microservices architecture. By building an environmental isolation from the gateway to the entire back-end service, grayscale verification can be performed on multiple different versions of services.

Click the link below to view the live tutorial:

https://yqh.aliyun.com/live/detail/29004

During the publishing process, we only need to deploy the grayscale version of the service. When traffic flows over the calling link, the gateways, middleware, and microservices that flow through identify the grayscale traffic, and dynamically forward it to the grayscale version of the corresponding service. As shown below:

The above figure can well demonstrate the effect of this scheme. We use different colors to represent different versions of grayscale traffic. It can be seen that both the microservice gateway and the microservice itself need to identify traffic and make dynamic decisions based on governance rules. When the service version changes, the forwarding of this call link will also change in real time. Compared to grayscale environments built using machines, this scheme not only can save a lot of machine costs and operation and maintenance manpower, but also can help developers quickly and accurately control online traffic over the full link.

OpenSergo [1] traffic routing standard

Q: What is OpenSergo?

A: OpenSergo is a set of open, universal, distributed service architecture oriented service governance standards that cover the entire link heterogeneous ecosystem. It forms a common standard for service governance based on industry service governance scenarios and practices. The biggest feature of OpenSergo is that it defines service governance rules with a unified set of configurations/DSLs/protocols, and faces a multilingual heterogeneous architecture to achieve full link ecological coverage. Regardless of whether the language of microservices is Java, Go, Node.js, or other languages, whether it is standard microservices or mesh access, from gateways to microservices, from databases to caches, and from service registration and discovery to configuration, developers can use the same set of OpenSergo CRD standard configurations to uniformly manage and control each layer, without paying attention to the differences between various frameworks and languages, reducing the complexity of heterogeneous and full link service governance and control

Q: Why introduce OpenSergo to me before understanding full link grayscale?

A: OpenSergo defines a unified set of YAML configuration methods to implement full link service governance specifications for distributed architectures. While introducing the specifications and standards, we can understand the implementation of the technical details. At the same time, we can also implement new components with OpenSergo standards.

Traffic routing, as the name implies, is to route traffic with certain attribute characteristics to a specified destination. Traffic routing is an important part of traffic governance. Developers can implement various scenarios based on traffic routing standards, such as grayscale publishing, canary publishing, disaster tolerance routing, label routing, and so on.

Full link grayscale example:

The traffic routing rules (v1alpha1) are mainly divided into three parts:

• Workload LabelRule: Label a group of workloads accordingly. This block can be understood as labeling the application or corresponding storage layer, which means labeling the database workload (database, table) accordingly

• Traffic Label Rule: Label traffic with certain attribute characteristics accordingly

• Perform matching routing based on Workload tags and traffic tags, routing traffic with specified tags to the matching Workload

Marking the flow:

Traffic with certain attribute characteristics needs to be labeled accordingly.

Assume that it is now necessary to grayscale users from the Shenzhen region to the new version of the home page, and the test user location=cn shenzhen is located in the location header

Through the above configuration, the HTTP traffic with the location header as cn-shenzhen is marked with a gray flag, indicating that this traffic is grayscale traffic.

Label Workload:

In business systems that use Nacos as a service discovery tool, it is generally necessary for businesses to determine the marking method based on the microservice framework they use. If Java applications use the Spring Cloud microservice development framework, we can add corresponding environment variables to the business container to complete the tag addition operation. For example, if we want to add a version grayscale to a node, add traffic.opensergo.io/label: gray to the business container, so that the framework will add a gray label to the node when registering it with Nacos.

For some complex workload marking scenarios (such as database instances and cache instance tags), we can use WorkloadLabelRule CRD for marking.

Common schemes for full link grayscale in databases

Scheme 1: Shadow Database

Each database maintains a separate set of independent databases. Assuming that the name of the database in the baseline environment is mse-demo, the traffic in the gray environment can be mapped to the database in mse-demo-gray. We establish a shadow database for the corresponding environment traffic on the same instance. We maintain a connection pool for each database connection in the business, and select the corresponding shadow database connection to access based on different traffic indicators, so as to achieve the effect of isolating data from the baseline environment database, Thereby avoiding the pollution of the baseline environment database caused by the data generated by the grayscale environment flow.

Scheme 2: Shadow Table

Similar to the shadow database scheme, for the shadow table scheme, the corresponding shadow table is created on the same database on the same instance. During the execution of SQL, we parse and modify the SQL for grayscale traffic to enable SQL for different environments to access the corresponding tables. Assuming that the name of the table in the baseline environment is mse_ demo_ Table, then the traffic in the gray environment can be mapped to the mse_ demo_ table_ Gray's table. Thereby achieving the effect of isolating grayscale data from baseline environmental data tables.

MSE [2] database full link grayscale capability

MSE provides a data isolation scheme that allows you to achieve full link grayscale at the database level without modifying any business code. The following describes MSE's ability to achieve full link grayscale through a shadow table scheme based on MySQL data storage.

prerequisite

• Application access to MSE

• Deploy Demo applications

Deploy three applications A, B, and C in Alibaba Cloud container service, with each application deploying a base version and a gray version respectively; And deploy a Nacos Server application for service discovery. For details, please refer to the tutorial to complete application deployment: Deploying a Demo application [3].

Step 1: Configure full link grayscale rules

You need to configure and complete the full link publishing of MSE. For specific operation details, please refer to the tutorial: Configuring Full Link Gray Scale [4].

Create the following swimlane rules:

Step 2: Configure the database full link grayscale

• We need to configure the following environment variables to additionally enable/configure the full link grayscale capability of the database

Step 3: Result verification

We initiated a grayscale request and found that all traffic requests access the grayscale environment:

Data found in grayscale environments are inserted into the shadow table.

Not just full link grayscale

So far, MSE service governance full link grayscale capabilities have supported cloud native gateways, ALB, APISIX, Apache Dubbo, Spring Cloud, RocketMQ, and databases. At the database level, we have achieved traffic isolation at the data level through shadow tables. In the next step, we will further commercialize this capability, and the full link grayscale will also support cache level capabilities.

Service governance is the only way to go after microservice transformation has reached a certain stage, and in this process, we continue to encounter new problems.

• What other capabilities does service governance have besides full link grayscale?

• Is there a standard definition of service governance capability, and what does it include?

• Are there any best practices or standards for full links in multilingual scenarios?

• How can heterogeneous microservices be uniformly governed?

When exploring service governance and connecting with other microservices, we found that the difficulties caused by different governance systems are enormous, and the cost of connecting two or even multiple governance systems is also enormous. For this reason, we proposed the OpenSergo project. OpenSergo aims to solve the problem of fragmentation and interoperability of concepts in microservice governance in different frameworks and languages.