Configure ASM Traffic Lanes to Propagate Baggage Headers Across Traces - Alibaba Cloud Service Mesh

When you run multiple service versions in traffic lanes (permissive mode), each request must carry a header that identifies its target lane throughout the entire call chain. Without end-to-end header propagation, the mesh cannot route inter-service calls to the correct lane. Baggage headers, propagated automatically by OpenTelemetry auto-instrumentation, solve this problem without requiring application code changes. When a service does not exist in a lane, the request falls back to the baseline lane.

Important

Before you begin, read Use traffic lanes in permissive mode to manage end-to-end traffic to understand lane groups, permissive mode, and baseline lanes.

How it works

This scenario uses three services and three lanes to demonstrate end-to-end traffic isolation with baggage header pass-through.

Services: mocka, mockb, and mockc, connected in a chain:

mocka --> mockb --> mockc

Lanes and service versions:

Lane	Role	Services
s1	Baseline lane	mocka (v1), mockb (v1), mockc (v1)
s2	Feature lane	mocka (v2), mockc (v2)
s3	Feature lane	mockb (v3)

Because s2 does not contain mockb and s3 does not contain mocka or mockc, requests for missing services fall back to the s1 baseline lane. For example, when a request enters lane s3, it reaches mockb (v3), but since mockc does not exist in s3, the call to mockc routes to mockc (v1) in s1.

Traffic distribution: Weight-based routing sends approximately 60% of traffic to s1, 20% to s2, and 20% to s3.

Prerequisites

Before you begin, make sure that you have:

An ASM instance with a data plane cluster. For more information, see Manage global namespaces
Automatic sidecar proxy injection enabled for the default namespace. For more information, see Enable automatic sidecar proxy injection
The OpenTelemetry Operator installed in the cluster to support auto-instrumentation

Step 1: Deploy sample services

Create a file named mock.yaml with the following content.

View the complete mock.yaml file

apiVersion: v1
kind: Service
metadata:
  name: mocka
  labels:
    app: mocka
    service: mocka
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mocka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v1
  labels:
    app: mocka
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mocka
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockb
  labels:
    app: mockb
    service: mockb
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v1
  labels:
    app: mockb
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mockb
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: mockc
  labels:
    app: mockc
    service: mockc
spec:
  ports:
  - port: 8000
    name: http
  selector:
    app: mockc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v1
  labels:
    app: mockc
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v1
      ASM_TRAFFIC_TAG: v1
  template:
    metadata:
      labels:
        app: mockc
        version: v1
        ASM_TRAFFIC_TAG: v1
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v1
        - name: app
          value: mockc
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v2
  labels:
    app: mocka
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mocka
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v2
  labels:
    app: mockb
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mockb
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v2
  labels:
    app: mockc
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v2
      ASM_TRAFFIC_TAG: v2
  template:
    metadata:
      labels:
        app: mockc
        version: v2
        ASM_TRAFFIC_TAG: v2
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v2
        - name: app
          value: mockc
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mocka-v3
  labels:
    app: mocka
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mocka
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mocka
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mocka
        - name: upstream_url
          value: "http://mockb:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockb-v3
  labels:
    app: mockb
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockb
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mockb
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mockb
        - name: upstream_url
          value: "http://mockc:8000/"
        ports:
        - containerPort: 8000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mockc-v3
  labels:
    app: mockc
    version: v3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mockc
      version: v3
      ASM_TRAFFIC_TAG: v3
  template:
    metadata:
      labels:
        app: mockc
        version: v3
        ASM_TRAFFIC_TAG: v3
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
        instrumentation.opentelemetry.io/container-names: "default"
    spec:
      containers:
      - name: default
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/asm-mock:v0.1-java
        imagePullPolicy: IfNotPresent
        env:
        - name: version
          value: v3
        - name: app
          value: mockc
        ports:
        - containerPort: 8000

Two annotations on each pod enable automatic baggage header propagation:

Annotation	Purpose
`instrumentation.opentelemetry.io/inject-java: "true"`	Declares that the service is a Java application and enables OpenTelemetry auto-instrumentation
`instrumentation.opentelemetry.io/container-names: "default"`	Specifies which container to instrument

The ASM_TRAFFIC_TAG label on each pod identifies the service version for lane-based routing.

Deploy the services:
```
kubectl apply -f mock.yaml
```
With these annotations in place, OpenTelemetry auto-instrumentation automatically propagates baggage headers through the entire call chain (mocka --> mockb --> mockc). No application code changes are required.

Step 2: Create a lane group and lanes

Create a lane group

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > Traffic Lane.

On the Traffic Lane page, click Create Swimlane Group. In the Create Swimlane Group panel, configure the following parameters and click OK.

Parameter	Value
Name of swim lane group	test
Istio gateway for an ingress gateway	ingressgateway
Lane Mode	Permissive Mode
Pass-through Mode of Trace Context	Pass Through Baggage Header
Routing Request Header	x-asm-prefer-tag
Swimlane Services	Select the cluster that contains mocka, mockb, and mockc from the Kubernetes Clusters drop-down list, select default from the Namespace drop-down list, then select mocka, mockb, and mockc and click the icon to add them to the selected section.

Create lanes

In the Traffic Rule Definition section of the Traffic Lane page, click Create swimlanes.

In the Create swimlanes dialog box, create three lanes with the following configurations and click OK. The following figure shows the configuration for the s1 lane. After creating all three lanes, the Traffic Rule Definition section displays them as shown below.

Note

The first lane created in a lane group is automatically set as the baseline lane. To change the baseline lane, see Change the baseline lane in permissive mode.

Parameter	s1	s2	s3
Swimlane Name	s1	s2	s3
Configure Service Tag -- Label Key	ASM_TRAFFIC_TAG	ASM_TRAFFIC_TAG	ASM_TRAFFIC_TAG
Configure Service Tag -- Label Value	v1	v2	v3
Add Service	mocka(default), mockb(default), mockc(default)	mocka(default), mockc(default)	mockb(default)

s1 lane configuration

All three lanes

Configure weight-based traffic routing

In the Traffic Rule Definition section, click Weight-based Routing under Traffic Routing Rule.
In the Set Unified Routing Rules dialog box, configure the following parameters and click OK.
Parameter Value
realm name *****
Matching request URI Method: Prefix, Content: /
Set the traffic routing weights for each lane. In the Traffic Rule Definition section, click the button next to the number in the Traffic Routing Weight column. In the Edit Traffic Routing Weight dialog box, configure the following parameters and click OK.
Parameter s1 s2 s3
Ingress service mocka.default.svc.cluster.local mocka.default.svc.cluster.local mocka.default.svc.cluster.local
Weight Value 60 20 20

Parameter	Value
realm name	*****
Matching request URI	Method: Prefix, Content: /

Parameter	s1	s2	s3
Ingress service	mocka.default.svc.cluster.local	mocka.default.svc.cluster.local	mocka.default.svc.cluster.local
Weight Value	60	20	20

Auto-generated routing resources

After you create the lane group, ASM automatically generates a DestinationRule and a VirtualService for each service. To view these resources, in the left-side navigation pane, choose Traffic Management Center > DestinationRule or VirtualService.

The following example shows the auto-generated resources for mocka:

View the DestinationRule and VirtualService for mocka

DestinationRule -- Defines subsets that map to lane-specific pod labels:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-dr-test-default-mocka
  namespace: istio-system
spec:
  host: mocka.default.svc.cluster.local
  subsets:
    - labels:
        ASM_TRAFFIC_TAG: v1
      name: s1
    - labels:
        ASM_TRAFFIC_TAG: v2
      name: s2

Each subset maps a lane name (s1, s2) to the corresponding ASM_TRAFFIC_TAG label value (v1, v2). The mesh uses this mapping to identify which pods belong to which lane.

VirtualService -- Routes requests based on the x-asm-prefer-tag header and defines fallback targets:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    asm-system: 'true'
    provider: asm
    swimlane-group: test
  name: trafficlabel-vs-test-default-mocka
  namespace: istio-system
spec:
  hosts:
    - mocka.default.svc.cluster.local
  http:
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s1
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s1
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s2
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s2
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1
    - match:
        - headers:
            x-asm-prefer-tag:
              exact: s3
      route:
        - destination:
            host: mocka.default.svc.cluster.local
            subset: s3
          fallback:
            target:
              host: mocka.default.svc.cluster.local
              subset: s1

The fallback.target field defines the routing behavior when a service version does not exist in the target lane:

Header value	Primary destination	Fallback target	Behavior
`s1`	subset s1	subset s1	Routes to s1 (baseline, always has all services)
`s2`	subset s2	subset s1	Routes to s2 if available, otherwise falls back to s1
`s3`	subset s3	subset s1	Routes to s3 if available, otherwise falls back to s1

This fallback mechanism is how permissive mode maintains trace integrity even when lanes contain only a subset of services.

Step 3: Verify the end-to-end canary release

Get the public IP address of the ingress gateway. For more information, see Obtain the IP address of the ASM ingress gateway.
Set the gateway IP as an environment variable. Replace <gateway-ip> with the actual IP address:
```
export ASM_GATEWAY_IP=<gateway-ip>
```

Send 100 requests to observe traffic distribution across lanes:

for i in {1..100}; do curl http://${ASM_GATEWAY_IP}/; echo ''; sleep 1; done;

Expected output (representative lines):

# Lane s1 -- all services at v1 (baseline)
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v1, ip: 192.168.0.190)

# Lane s2 -- mocka and mockc at v2, mockb falls back to v1 in s1
-> mocka(version: v2, ip: 192.168.0.184)-> mockb(version: v1, ip: 192.168.0.1)-> mockc(version: v2, ip: 192.168.0.189)

# Lane s3 -- mockb at v3, mocka and mockc fall back to v1 in s1
-> mocka(version: v1, ip: 192.168.0.193)-> mockb(version: v3, ip: 192.168.0.2)-> mockc(version: v1, ip: 192.168.0.190)

The output confirms two behaviors:

Weight-based distribution: Approximately 60% of requests go to s1, 20% to s2, and 20% to s3, matching the configured 60:20:20 ratio.
Baseline fallback: When a service version does not exist in a lane (for example, mockb in s2 or mocka/mockc in s3), the request falls back to the corresponding service in the s1 baseline lane.

Usage notes

OpenTelemetry requirement: The auto-instrumentation annotations (instrumentation.opentelemetry.io/inject-java and instrumentation.opentelemetry.io/container-names) require a working OpenTelemetry Operator in the cluster. Without it, baggage headers are not propagated and lane routing does not work correctly.
Baseline lane selection: The first lane created in a lane group is the default baseline. All fallback routing targets this lane, so make sure the baseline contains all services in the call chain.

Alibaba Cloud Service Mesh:Pass through baggage headers in traffic lanes