If you need to implement an end-to-end canary release for multiple services, you can configure traffic labels to identify traffic characteristics and divide the ingress traffic of a gateway into regular traffic and canary traffic. The canary traffic characteristics are passed among the services that are used to process user requests. This way, an end-to-end canary release is implemented. This topic describes how to use traffic labels to implement an end-to-end canary release for microservices.

Prerequisites

Overview

Canary releases can be implemented in various ways. For example, you can use ASM to deploy an application in blue-green release mode and phased release mode. The sample mode is applicable to the release of a single service. This mode uses VirtualService of open source Istio to specify the traffic weight of each version of the service. This mode cannot meet requirements in canary release scenarios of multiple services.

ASM provides the end-to-end canary release feature based on the traffic labeling and label-based routing features. This allows you to implement the canary release of multiple services at the same time. During canary testing of business, the system divides ingress traffic into regular traffic and canary traffic and identifies the characteristics of traffic. If the traffic of a request is identified as canary traffic, the request is routed to the canary release service. This way, the system no longer simply distributes traffic to backend services of different versions at specific traffic ratios. Instead, the canary traffic characteristics are passed among all the services that are used to process user requests. For more information, see Label traffic.

Configuration description

You can click here to download the sample configuration files. The following figure shows the call chains of the services in the example:

Call chains of the services

In this example, when Service A calls Service B, Service A contains the code implementation logic to pass the HTTP header my-request-id. For more information, see the main.go configuration file in the src/mock-abc/go path.

Note If you use other applications, make sure that the applications contain the logic to pass the HTTP header. Subsequent functions rely on the HTTP header.
Application deployment file for the base version: application-base.yaml
apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-base
  labels:
    app: mocka
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: base
  template:
    metadata:
      labels:
        app: mocka
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-base
  labels:
    app: mockb
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: base
  template:
    metadata:
      labels:
        app: mockb
        version: base
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mockb
        ports:
        - containerPort: 8000
Application deployment file for the canary release version: application-canary.yaml
apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-canary
  labels:
    app: mocka
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: canary
  template:
    metadata:
      labels:
        app: mocka
        version: canary
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-canary
  labels:
    app: mockb
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: canary
  template:
    metadata:
      labels:
        app: mockb
        version: canary
    spec:
      containers:
      - name: default
        image: registry.cn-beijing.aliyuncs.com/aliacs-app-catalog/go-http-sample:1.0
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

Step 1: Deploy sample services in the ACK cluster

  1. Enable automatic sidecar proxy injection for the desired namespace (the default namespace in this example). For more information, see the "Enable automatic sidecar injection" section of the Manage global namespaces topic.
  2. Use kubectl to connect to the ACK cluster based on the information in the kubeconfig file and run the following commands to deploy sample services.
    For more information about the YAML files, see Configuration description. 
    kubectl apply -n default -f application-base.yaml
kubectl apply -n default -f application-canary.yaml

Step 2: Configure initial routing rules

  1. Create a file named routing.yaml and copy the following content to the file.

    Content of routing.yaml

    apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: simple-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mocka
spec:
  host: mocka
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mockb
spec:
  host: mockb
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb
  2. Use kubectl to connect to the ASM instance based on the information in the kubeconfig file and run the following command to configure routing rules: 
    kubectl apply -n default -f routing.yaml
  3. Check whether the services of different versions can be accessed.
    1. Obtain the public IP address of the ingress gateway in the ASM console. For more information, see the "Step 2: Query the IP address of the ASM instance's ingress gateway" section of the Integrate the cloud-native inference service KServe with ASM topic.
    2. Run the following command to configure an environment variable.
      xxxx is the IP address obtained in Substep a. 
      export ASM_GATEWAY_IP=xxxx
    3. Run the following command to check whether the services of different versions can be accessed: 
      for i in {1..200};  do curl   -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
      Expected output:
      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
      The output shows that the routing policy of random load balancing is adopted for traffic flows from the ingress gateway to Service A and from Service A to Service B. The my-asm-prefer-tag header or other request headers specified by using the curl command do not change the random routing policy.

Step 3: Configure traffic labels for the sample services

ASM allows you to configure traffic labels by using a TrafficLabel CustomResourceDefinition (CRD) and route traffic to different workloads based on the traffic labels. For more information, see Label traffic.

  1. Create a file named trafficlabel.yaml and copy the following content to the file: 
    apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: trafficlabel-ns
  namespace: default
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getExternalInboundRequestHeader(my-asm-prefer-tag, my-request-id)
---
apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getInboundRequestHeader(my-asm-prefer-tag)
  workloadSelector:
    labels:
      istio: ingressgateway
    Configuration description:
    • The trafficlabel-ns traffic label takes effect on all services in the default namespace, such as the mocka and mockb services deployed in this example. The ingressgateway traffic label takes effect only on the ingressgateway gateway.
    • In this example, the my-asm-prefer-tag request header is used to distinguish versions. Therefore, set the first parameter in getExternalInboundRequestHeader to my-asm-prefer-tag. Modify the configuration based on your business requirements.
    • In this example, my-request-id is the HTTP header that needs to be passed. Therefore, set the second parameter in getExternalInboundRequestHeader to my-request-id. Modify the configuration based on your business requirements.
  2. Use kubectl to connect to the ASM instance based on the information in the kubeconfig file and run the following command to configure the traffic labels: 
    kubectl apply -f trafficlabel.yaml

Step 4: Configure label-based routing rules

You can configure label-based routing rules to control traffic flows.

1. Check the canary release on the service provider side

Check whether the traffic routing from Service A to Service B meets the expectation. To be specific, check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B, and whether the base traffic that flows into Service A is forwarded to the base version of Service B.

The following figure shows the traffic flows after the traffic label-based routing rule is configured for Service B: Check the canary release on the service provider side
  1. Create a file named vs-tf-mockb.yaml and copy the following content to the file: 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb
            subset: $asm-labels-sample
  2. Run the following command to make the routing rule take effect for Service A: 
    kubectl apply -n default -f vs-tf-mockb.yaml
  3. Run the following command to check whether the canary traffic that flows into Service A is forwarded to the canary release version of Service B: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
  4. Run the following command to check whether the base traffic that flows into Service A is forwarded to the base version of Service B: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
    The results show that the ingress canary traffic and ingress base traffic that flow into Service A are not routed to the expected versions of Service B. You need to configure a traffic label-based routing rule for Service A. For more information, proceed to the next step.

2. Check the end-to-end canary release feature

Configure a traffic label-based routing rule file named a-vs-tf.yaml for Service A and make the file take effect for the ingress gateway. The expected result is that the canary traffic of ingress requests is first forwarded to the canary release version of Service A and then to that of Service B, and the base traffic is first forwarded to the base version of Service A and then to that of Service B.

The following figure shows the traffic flows after the traffic label-based routing rule is configured for Service A: Check the end-to-end canary release feature
  1. Create a file named vs-tf-mocka.yaml and copy the following content to the file: 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: $asm-labels-sample
  2. Run the following command to make the routing rule take effect for the ingress gateway: 
    kubectl apply -n default -f vs-tf-mocka.yaml
  3. Run the following command to check whether the canary traffic of ingress requests is first forwarded to the canary release version of Service A and then to that of Service B: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
  4. Run the following command to check whether the base traffic of ingress requests is first forwarded to the base version of Service A and then to that of Service B: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

3. Check whether the weight-based traffic distribution that corresponds to the traffic label-based routing meets the requirements

The following figure shows the traffic flows after the weight-based and traffic label-based routing rules are configured for Service A: Check the route weight
  1. Create a file named vs-tf-mocka-90-10.yaml and copy the following content to the file: 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: version-base
        weight: 90
      - destination:
          host: mocka
          subset: $asm-labels-sample
        weight: 10
  2. Run the following command to make the routing rules take effect for the ingress gateway: 
    kubectl apply -n default -f vs-tf-mocka-90-10.yaml
  3. Run the following command to check whether the canary traffic of ingress requests is forwarded to the canary release and base versions of Service A: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-canary'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
    The results show that about 90% of the canary traffic of ingress requests is forwarded to the base version of Service A and about 10% is forwarded to the canary release version of Service A. According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-canary. Therefore, 10% of the request traffic is forwarded to the canary release version of Service A.
  4. Run the following command to check whether all base traffic of ingress requests is forwarded to the base version of Service A: 
    for i in {1..200};  do curl -H'my-asm-prefer-tag: version-base'  -H'my-request-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;
    Expected output:
    -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
    According to the above routing rules, about 90% of the request traffic is forwarded to the base version of Service A, and the remaining 10% is forwarded based on the value of the my-asm-prefer-tag header. The request command sets the value of my-asm-prefer-tag to version-base. Therefore, 10% of the request traffic is forwarded to the base version of Service A.