All Products
Search

This Product

    Alibaba Cloud Service Mesh:Use traffic labels to implement an end-to-end canary release

    Document Center

    Alibaba Cloud Service Mesh:Use traffic labels to implement an end-to-end canary release

    Last Updated:Sep 29, 2026

    Standard canary releases route traffic to different versions of a single service based on weight ratios. This works for isolated services but breaks down when a request spans multiple services in a call chain. For example, if canary traffic reaches Service A but then gets randomly routed to the base version of Service B, test results become unreliable.

    End-to-end canary releases solve this by tagging ingress traffic at the gateway and propagating that tag across the entire call chain. Every service reads the tag and routes to the matching version, so canary traffic flows exclusively through canary versions and base traffic stays on base versions.

    Service Mesh (ASM) implements this through two capabilities:

    • Traffic labeling: A TrafficLabel CustomResourceDefinition (CRD) that extracts labels from request headers and attaches them to traffic flows

    • Label-based routing: VirtualService rules that route traffic to subsets matching the label value

    How it works

    The following diagram shows the call chain used in this tutorial. Service A (mocka) receives ingress traffic and calls Service B (mockb):

    Application service call chain

    Both services have two deployed versions: base and canary. Without traffic labels, requests are distributed across versions by round-robin load balancing.

    After you configure traffic labels, the system:

    1. Reads the my-asm-prefer-tag header from incoming requests at the gateway to classify traffic as version-base or version-canary.

    2. Routes the request to the matching version of Service A.

    3. Propagates the tag to Service B through the my-trace-id header, so Service B routes to the matching version as well.

    Note

    Your applications must propagate the HTTP header across service calls. The sample services in this tutorial already include this logic. For your own applications, implement header propagation for the header used in traffic labeling. See src/mock-abc/go/main.go in the sample code for a reference implementation.

    Prerequisites

    Before you begin, ensure that you have:

    Sample configuration files

    Download the sample configuration files used in this tutorial. The archive contains the following files:

    FilePurpose
    application-base.yamlDeploys the base versions of Service A (mocka) and Service B (mockb)
    application-canary.yamlDeploys the canary versions of both services
    routing.yamlDefines the Gateway, DestinationRules, and VirtualServices for initial routing
    trafficlabel.yamlConfigures the TrafficLabel CRDs for traffic labeling
    vs-tf-mockb.yamlApplies label-based routing to Service B
    vs-tf-mocka.yamlApplies label-based routing to Service A at the gateway
    vs-tf-mocka-90-10.yamlAdds weight-based distribution (90% base, 10% label-based) to Service A

    application-base.yaml -- Base version deployments

    apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-base
  labels:
    app: mocka
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: base
  template:
    metadata:
      labels:
        app: mocka
        version: base
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-base
  labels:
    app: mockb
    version: base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: base
  template:
    metadata:
      labels:
        app: mockb
        version: base
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: base
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

    application-canary.yaml -- Canary version deployments

    apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-canary
  labels:
    app: mocka
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: canary
  template:
    metadata:
      labels:
        app: mocka
        version: canary
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-canary
  labels:
    app: mockb
    version: canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: canary
  template:
    metadata:
      labels:
        app: mockb
        version: canary
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/go-http-sample:tracing
        imagePullPolicy: Always
        env:
        - name: version
          value: canary
        - name: app
          value: mockb
        ports:
        - containerPort: 8000

    Step 1: Deploy sample services

    1. Enable automatic sidecar proxy injection for the target namespace (default in this example). See Enable automatic sidecar proxy injection.

    2. Connect to the Container Service for Kubernetes (ACK) cluster with kubectl and deploy the sample services:

      kubectl apply -n default -f application-base.yaml
kubectl apply -n default -f application-canary.yaml

    Step 2: Set up initial routing rules

    Set up the Gateway, DestinationRules, and VirtualServices to establish basic routing without traffic labels.

    1. Create a file named routing.yaml with the following content:

      Content of routing.yaml

      apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: simple-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mocka
spec:
  host: mocka
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: dr-mockb
spec:
  host: mockb
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - labels:
        version: base
      name: version-base
    - labels:
        version: canary
      name: version-canary
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb

      This configuration:

      • Creates a Gateway (simple-gateway) that accepts all HTTP traffic on port 80

      • Defines DestinationRules for both services with version-base and version-canary subsets

      • Creates VirtualServices that route traffic without label-based logic

    2. Connect to the ASM instance with kubectl and apply the routing rules:

      kubectl apply -n default -f routing.yaml

    3. Verify that both service versions are reachable.

      1. Get the public IP address of the ingress gateway from the ASM console. See Obtain the IP address of the ASM ingress gateway.

      2. Set the gateway IP as an environment variable. Replace <gateway-ip> with the actual IP address:

        export ASM_GATEWAY_IP=<gateway-ip>

      3. Send test requests:

        for i in {1..200};  do curl   -H'my-asm-prefer-tag: version-base'  -H'my-trace-id: x000'$i  http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- traffic is randomly distributed across all versions of both services:

      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)

      At this point, the my-asm-prefer-tag header has no effect on routing. Requests follow round-robin load balancing regardless of the header value.

    Step 3: Configure traffic labels

    Create TrafficLabel resources to extract routing labels from request headers. ASM uses these labels to determine whether traffic is version-base or version-canary. For background on traffic labeling, see Label traffic.

    1. Create a file named trafficlabel.yaml with the following content:

      Content of trafficlabel.yaml

      apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: trafficlabel-ns
  namespace: default
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getExternalInboundRequestHeader(my-asm-prefer-tag, my-trace-id)
---
apiVersion: istio.alibabacloud.com/v1
kind: TrafficLabel
metadata:
  name: ingressgateway
  namespace: istio-system
spec:
  rules:
  - labels:
    - name: asm-labels-sample
      valueFrom:
      - $getInboundRequestHeader(my-asm-prefer-tag)
  workloadSelector:
    labels:
      istio: ingressgateway

      This defines two TrafficLabel resources:

      ResourceScopeBehavior
      trafficlabel-nsAll services in the default namespace (mocka, mockb)Uses $getExternalInboundRequestHeader to extract the version tag from the my-asm-prefer-tag header. The second parameter, my-trace-id, specifies the trace header that the application must propagate between services.
      ingressgatewayThe ingress gateway only (istio-system namespace)Reads the version tag from the my-asm-prefer-tag header of the inbound client request.
      Note

      The header names my-asm-prefer-tag and my-trace-id are used in this example. Replace them with headers that match your application.

    2. Connect to the ASM instance with kubectl and apply the traffic labels:

      kubectl apply -f trafficlabel.yaml

    Step 4: Apply label-based routing and verify

    This step progressively verifies the end-to-end canary release in three stages:

    1. Single-hop routing -- Apply label-based routing to Service B and confirm that the traffic tag correctly routes requests to the matching version

    2. Full-chain routing -- Apply label-based routing to Service A at the gateway so canary traffic flows through canary versions of all services end to end

    3. Weighted routing -- Combine weight-based distribution with label-based routing to control the canary rollout percentage

    Verify label-based routing for Service B

    Apply label-based routing to Service B first. This verifies that the traffic tag from Service A correctly routes requests to the matching version of Service B.

    Service provider side canary release verification

    1. Create a file named vs-tf-mockb.yaml with the following content:

      apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-mockb
spec:
  hosts:
    - mockb
  http:
    - route:
        - destination:
            host: mockb
            subset: $asm-labels-sample

      The $asm-labels-sample subset dynamically resolves to the traffic label value -- either version-base or version-canary.

    2. Apply the routing rule:

      kubectl apply -n default -f vs-tf-mockb.yaml

    3. Send canary traffic and verify that Service B receives it on the canary version:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-canary' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- Service A still receives random traffic, but Service B consistently routes to the canary version:

      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: canary, ip: 172.17.0.54)

    4. Send base traffic and verify that Service B receives it on the base version:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-base' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- Service B consistently routes to the base version:

      -> mocka(version: canary, ip: 172.17.0.129)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)
-> mocka(version: base, ip: 172.17.0.132)-> mockb(version: base, ip: 172.17.0.70)

    Service B routing now works as expected. Service A still receives traffic randomly because no label-based routing rule has been applied to the gateway yet.

    Verify end-to-end canary routing

    Apply label-based routing to Service A at the gateway level so that the entire call chain respects the traffic label.

    End-to-end canary release verification

    1. Create a file named vs-tf-mocka.yaml with the following content:

      apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: $asm-labels-sample

    2. Apply the routing rule to the ingress gateway:

      kubectl apply -n default -f vs-tf-mocka.yaml

    3. Send canary traffic and verify that both services route to canary versions:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-canary' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- all traffic flows through canary versions end to end:

      -> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

    4. Send base traffic and verify that both services route to base versions:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-base' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- all traffic flows through base versions end to end:

      -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

    End-to-end canary routing is now active. Canary traffic consistently hits canary versions of all services, and base traffic stays on base versions.

    Verify weighted traffic distribution with labels

    In production, you typically start by routing a small percentage of traffic to the canary version. Combine weight-based distribution with label-based routing to control the rollout.

    Weight-based and label-based routing verification

    1. Create a file named vs-tf-mocka-90-10.yaml with the following content:

      apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-gateway-mocka
spec:
  gateways:
  - simple-gateway
  hosts:
    - "*"
  http:
  - route:
      - destination:
          host: mocka
          subset: version-base
        weight: 90
      - destination:
          host: mocka
          subset: $asm-labels-sample
        weight: 10

      This sends 90% of traffic to the base version of Service A. The remaining 10% is routed based on the traffic label -- if the my-asm-prefer-tag header is set to version-canary, that 10% goes to the canary version.

    2. Apply the weighted routing rule:

      kubectl apply -n default -f vs-tf-mocka-90-10.yaml

    3. Send canary traffic and check the distribution:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-canary' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- approximately 90% of requests reach the base version of Service A, while 10% reach the canary version. Service B consistently routes to the canary version because the label propagates through the call chain:

      -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: canary, ip: 172.17.0.130)
...
-> mocka(version: canary, ip: 172.17.0.69)-> mockb(version: canary, ip: 172.17.0.130)

      About 9 out of 10 requests hit mocka(version: base), while roughly 1 out of 10 hits mocka(version: canary). In both cases, mockb routes to the canary version.

    4. Send base traffic and verify that all requests reach base versions:

      for i in {1..200}; do curl -H'my-asm-prefer-tag: version-base' -H'my-trace-id: x000'$i http://${ASM_GATEWAY_IP}/; echo; sleep 1; done;

      Expected output -- 90% of traffic goes directly to the base version. The remaining 10% also routes to the base version because the label resolves to version-base:

      -> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)
-> mocka(version: base, ip: 172.17.0.90)-> mockb(version: base, ip: 172.17.0.93)

      With my-asm-prefer-tag set to version-base, 90% goes to the base version by fixed weight and the remaining 10% also resolves to base by label. The result is 100% base traffic.

    What to do next