Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Traffic Labeling and Routing of ASM (4): Implement an End-to-end Canary Release Based on ASM

Traffic Labeling and Routing of ASM (4): Implement an End-to-end Canary Release Based on ASM

Part 4 of this 4-part series describes how to use traffic labels to implement an end-to-end canary release for microservices.

By Xining Wang

If you need to implement an end-to-end canary release for multiple services, you can configure traffic labels to identify traffic characteristics and divide the ingress traffic of a gateway into regular traffic and canary traffic. The canary traffic characteristics are passed among the services that are used to process user requests. This way, an end-to-end canary release is implemented. This article describes how to use traffic labels to implement an end-to-end canary release for microservices.

Prerequisites

• An Alibaba Cloud Service Mesh (ASM) instance of Enterprise Edition or Ultimate Edition is created. The version of the ASM instance is 1.17.2.22 or later. For more information, see Create an ASM instance.

• An ACK cluster is created and added to the ASM instance. For more information, see Add a cluster to an ASM instance.

• An ingress gateway is created. For more information, see Create an ingress gateway service.

Overview

Canary releases can be implemented in various ways. For example, you can use ASM to deploy blue-green release and phased release mode. The sample mode is applicable to the release of a single service. This mode uses VirtualService of open source Istio to specify the traffic weight of each version of the service. This mode cannot meet requirements in canary release scenarios of multiple services.

ASM provides the end-to-end canary release feature based on the traffic labeling and label-based routing features. This allows you to implement the canary release of multiple services at the same time. During canary testing of business, the system divides ingress traffic into regular traffic and canary traffic and identifies the characteristics of traffic. If the traffic of a request is identified as canary traffic, the request is routed to the canary release service. This way, the system no longer simply distributes traffic to backend services of different versions at specific traffic ratios. Instead, the canary traffic characteristics are passed among all the services that are used to process user requests. For more information, see Traffic Labeling and Label-based Routing.

Configuration Description

You can click here to download the sample configuration files. The following figure shows the call chains of the services in the example:

1

In this example, when Service A calls Service B, Service A contains the code implementation logic to pass the HTTP header my-request-id. For more information, see the main.go configuration file in the src/mock-abc/go path.

Note: If you use other applications, make sure that the applications contain the logic to pass the HTTP header. Subsequent functions rely on the HTTP header.

Application deployment file for the base version: application-base.yaml

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-base
  labels:
    app: mocka
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: base
  template:
    metadata:
      labels:
        app: mocka
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-base
  labels:
    app: mockb
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: base
  template:
    metadata:
      labels:
        app: mockb
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

Application deployment file for the canary release version: application-canary.yaml

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-canary
  labels:
    app: mocka
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: canary
  template:
    metadata:
      labels:
        app: mocka
        version: canary
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-canary
  labels:
    app: mockb
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: canary
  template:
    metadata:
      labels:
        app: mockb
        version: canary
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

Step 1: Deploy Sample Application in the ACK cluster

Enable automatic sidecar proxy injection for the desired namespace (the default namespace in this example). For more information, see https://www.alibabacloud.com/help/en/alibaba-cloud-service-mesh/latest/manage-namespaces#section-30o-vil-3n7

Use the kubeconfig file of the ACK cluster to run the following command to deploy the sample application:

kubectl apply -n default -f application-base.yaml
kubectl apply -n default -f application-canary.yaml

Step 2: Configure Initial Routing Rules

Create a file named routing.yaml and copy the following content to the file.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: simple-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mocka
spec:
  host: mocka
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mockb
spec:
  host: mockb
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb

Use the kubeconfig file of the ASM instance to run the following command to configure routing rules:

kubectl apply -n default -f routing.yaml

Check whether a service can be accessed.

Obtain the public IP address of the ingress gateway in the ASM Console:

export ASM_GATEWAY_IP=xxx.xxx.xxx.xxx

Run the following command to check whether a service can be accessed:

for i in {
     1..200};  do curl   -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'{
     mathJaxContainer[0]}{
     ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)

The output shows that the routing policy of random load balancing is adopted for traffic flows from the ingress gateway to Service A and from Service A to Service B. The my-asm-prefer-tag header or other request headers specified by using the curl command do not change the random routing policy.

Step 3: Configure Traffic Labels for the Sample Services

ASM allows you to configure traffic labels by using a TrafficLabel CustomResourceDefinition (CRD) and route traffic to different workloads based on the traffic labels. For more information, see Label traffic.

1) Create a file named trafficlabel.yaml and copy the following content to the file:

Note:

  • In this example, the my-asm-prefer-tag request header is used to distinguish versions. Therefore, set the first parameter in getExternalInboundRequestHeader to my-asm-prefer-tag. Modify the configuration based on your business requirements.
  • In this example, my-request-id is the HTTP header that needs to be passed. Therefore, set the second parameter in getExternalInboundRequestHeader to my-request-id. Modify the configuration based on your business requirements.
apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: trafficlabel-ns
  namespace: default
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getExternalInboundRequestHeader(my-asm-prefer-tag, my-request-id)
---
apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getInboundRequestHeader(my-asm-prefer-tag)
  workloadSelector:
    labels:
      istio: ingressgateway

2) Use kubectl to connect to the ASM instance based on the information in the kubeconfig file and run the following command to configure the traffic labels:

kubectl apply -f trafficlabel.yaml

The trafficlabel-ns traffic label takes effect on all services in the default namespace, such as the mocka and mockb services deployed in this example. The ingressgateway traffic label takes effect only on the ingressgateway gateway.

Step 4: Configure Label-based Routing Rules

1. Check the Canary Release on the Service Provider Side

Check whether the traffic routing from Service A to Service B meets the expectation. To be specific, check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B, and whether the base traffic that flows into Service A is forwarded to the base version of Service B.

The following figure shows the traffic flows after the traffic label-based routing rule is configured for Service B:

2

1) Create a file named vs-tf-mockb.yaml and copy the following content to the file.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb
            subset: $asm-labels-sample

2) Run the following command to make the routing rule take effect for Service A:

kubectl apply -n default -f vs-tf-mockb.yaml

3) Run the following command to check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B:

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'{
   mathJaxContainer[1]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)

4) Run the following command to check whether the base traffic that flows into Service A is forwarded to the base version of Service B:

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'{
   mathJaxContainer[2]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)

The results show that the ingress canary traffic and ingress base traffic that flow into Service A are not routed to the expected versions of Service B. You need to configure a traffic label-based routing rule for Service A.

2. Check the End-to-end Canary Release Feature

Configure a traffic label-based routing rule file named a-vs-tf.yaml for Service A and make the file take effect for the ingress gateway. The expected result is that the canary traffic of ingress requests is first forwarded to the canary version of Service A, and the base traffic is forwarded to the base version of Service A and then to that of Service B.

The following figure shows the traffic flows after the traffic label-based routing rule is configured for Service A:

3

1) Create a file named vs-tf-mocka.yaml and copy the following content to the file.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: $asm-labels-sample

2) Run the following command to make the routing rule take effect for the ASM gateway:

kubectl apply -n default -f vs-tf-mocka.yaml

3) Run the following command to check whether the canary traffic of ingress requests is forwarded to the canary version of Service A:

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'{
   mathJaxContainer[3]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

4) Run the following command to check whether the base traffic is forwarded to the base version of Service A and then to that of Service B.

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'{
   mathJaxContainer[4]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

3. Check the Weight-based Traffic Distribution

The following figure shows the traffic flows after the weight-based and traffic label-based routing rules are configured for Service A:

4

1) Create a file named vs-tf-mocka-90-10.yaml and copy the following content to the file.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: version-base
        weight: 90
      - destination:
          host: mocka
          subset: $asm-labels-sample
        weight: 10

2) Run the following command to make the routing rules take effect for the ASM gateway:

kubectl apply -n default -f vs-tf-mocka-90-10.yaml

3) Run the following command to check whether the canary traffic of ingress requests is forwarded to the canary release version of Service A:

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'{
   mathJaxContainer[5]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

The results show that about 90% of the canary traffic of ingress requests is forwarded to the base version of Service A and about 10% is forwarded to the canary release version of Service A. According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-canary. Therefore, 10% of the request traffic is forwarded to the canary release version of Service A.
Expected output:

-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

4) Run the following command to check whether all base traffic of ingress requests is forwarded to the base version of Service A.

According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-base. Therefore, 10% of the request traffic is forwarded to the base version of Service A.

for i in {
   1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'{
   mathJaxContainer[6]}{
   ASM_GATEWAY_IP}/; echo; sleep 1; done;

Expected output:

-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
middleware Microservices Developers cloud native Service Mesh ASM Canary Release Labelling Traffic Label
0 1 0
Share on

Read previous post:

Traffic Labeling and Routing of ASM (3): Use Lanes to Manage Traffic

Read next post:

Enable cgroup V2 on Alibaba Cloud Container Service for Kubernetes

Xi Ning Wang(王夕宁)

56 posts | 8 followers

You may also like

Comments

Xi Ning Wang(王夕宁)

56 posts | 8 followers

Related Products

  • Microservices Engine (MSE)

    MSE provides a fully managed registration and configuration center, and gateway and microservices governance capabilities.

    Learn More

  • Cloud-Native Applications Management Solution

    Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.

    Learn More

  • Function Compute

    Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.

    Learn More

  • Lindorm

    Lindorm is an elastic cloud-native database service that supports multiple data models. It is capable of processing various types of data and is compatible with multiple database engine, such as Apache HBase®, Apache Cassandra®, and OpenTSDB.

    Learn More

More Posts by Xi Ning Wang(王夕宁)

See All