Use traffic mirroring for services within a cluster - Alibaba Cloud Service Mesh

Traffic mirroring copies live production requests to a test service version in real time, without affecting the production request path. Use it to validate new releases, debug issues, or run simulation tests against real traffic -- with zero risk to production.

How it works

Traffic mirroring -- also called traffic shadowing -- sends a copy of each inbound request to a mirrored service alongside the primary destination.

Out-of-band processing: Mirrored traffic runs separately from the critical request path. Failures in the mirrored service do not affect production responses.
Fire and forget: Responses from the mirrored service are discarded. The primary service returns its response to the client normally.
Header tagging: Requests to the mirrored service have -shadow appended to their Host or Authority header, so you can distinguish mirrored requests from production requests in logs.

Use cases

Use case	Description
Pre-release validation	Mirror production traffic to a new service version before release. Compare outputs between the current and new versions to catch regressions with real traffic.
System migration	Mirror traffic from a legacy system to its replacement for a trial run before cutover.
Database isolation testing	Route mirrored production traffic to a separate test database. Verify data processing performance without contaminating production data.
Live debugging	Mirror traffic from a problematic service to a temporary instance for debugging, without disrupting the running service.
User behavior capture	Log real user traffic for recommendation algorithms, user profiling, or big data analytics where synthetic test data falls short.

Sample VirtualService configuration

This VirtualService routes all traffic to the v1 subset and mirrors 100% of it to v1-mirroring:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-traffic-mirroring
spec:
  hosts:
    - myapp
  http:
    - route:
        - destination:
            host: myapp.default.svc.cluster.local
            port:
              number: 8000
            subset: v1
          weight: 100
      mirror:
        host: myapp.default.svc.cluster.local
        port:
          number: 8000
        subset: v1-mirroring
      mirrorPercentage:
        value: 100.0

Field	Description
`mirror`	Specifies the destination for mirrored traffic.
`mirrorPercentage.value`	The percentage of traffic to mirror. Set to `100.0` to mirror all traffic. Set a lower value such as `50.0` to mirror only a portion. If you omit the `mirrorPercentage` field entirely, all traffic (100%) is mirrored by default.

Tutorial: Mirror traffic between httpbin service versions

This tutorial deploys two versions of the httpbin service (v1 and v1-mirroring), then configures ASM to route all traffic to v1 while mirroring 50% to v1-mirroring.

Traffic mirroring architecture within a cluster

Prerequisites

Before you begin, make sure that you have:

An ASM instance with a Kubernetes cluster added
Sidecar proxy injection enabled for the target namespace
kubectl configured to connect to your cluster

Step 1: Deploy sample services

Create a file named httpbin.yaml with the following content:

httpbin.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1-mirroring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1-mirroring
  template:
    metadata:
      labels:
        app: httpbin
        version: v1-mirroring
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Deploy the v1 and v1-mirroring versions of the httpbin service:
```
   kubectl apply -f httpbin.yaml
```

Create a file named sleep.yaml with the following content:

sleep.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Deploy the sleep client service:
```
   kubectl apply -f sleep.yaml
```

Verify that all pods reach Running status:

   kubectl get pods -l app=httpbin
   kubectl get pods -l app=sleep

Step 2: Create routing rules

Create a destination rule and a virtual service through the ASM console.

Create a destination rule

Log on to the ASM console.
In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance.
In the left-side navigation pane, choose Traffic Management Center > DestinationRule.
Click Create and configure the destination rule to define the v1 and v1-mirroring subsets.

Create a virtual service

In the left-side navigation pane of the ASM instance details page, choose Traffic Management Center > VirtualService.
Click Create and configure the virtual service to route all traffic to v1 and mirror 50% of traffic to v1-mirroring.

Step 3: Send test requests and verify results

Export the sleep pod name and send a request to httpbin:

   export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
   kubectl exec -it $SLEEP_POD -c sleep -- curl http://httpbin:8000/headers

Check pod logs for v1 to confirm it received the request:

   kubectl logs -l app=httpbin,version=v1 -c httpbin

Check pod logs for v1-mirroring to confirm it received the mirrored request:
```
   kubectl logs -l app=httpbin,version=v1-mirroring -c httpbin
```
The v1 logs show all incoming requests. The v1-mirroring logs show mirrored copies. Mirrored requests have a larger packet size because -shadow is appended to the authority header.

Clean up

Remove sample resources after testing:

kubectl delete -f httpbin.yaml
kubectl delete -f sleep.yaml

Delete the destination rule and virtual service in the ASM console:

In the left-side navigation pane of the ASM instance details page, choose Traffic Management Center > VirtualService.
Delete the virtual service created in this tutorial.
Choose Traffic Management Center > DestinationRule and delete the destination rule.

What's next

Check the log of a pod: View pod logs in Container Service for Kubernetes (ACK).