You can use the traffic mirroring feature to mirror production traffic to a test cluster or test service version. Testing that uses the mirrored production traffic mitigates risks involved in version changes without affecting the production environment. This topic describes what traffic mirroring is and how to use this feature for services within a cluster.
What is traffic mirroring?
The microservice architecture makes application development and deployment faster, but risks exist in changing service versions. Alibaba Cloud Service Mesh (ASM) provides the traffic mirroring feature to mitigate the risks. The feature, also called traffic shadowing, sends production traffic to a mirrored service in real time. The mirrored traffic happens out of band of the critical request path for the production service. When traffic is mirrored, the requests that are sent to the mirrored service version have their Host/Authority headers appended with -shadow. This distinguishes production traffic and mirrored traffic. You can use the feature to mirror production traffic to a test cluster or test service version before the cluster or service version is running in the production environment. This mitigates risks involved in version changes.
Benefits
Benefit | Description |
---|---|
Less-risky version deployment with more production-like test environment | You can copy production traffic to a test cluster or test service version and perform tests by using the mirrored use cases and traffic. More accurate test results mitigate deployment risks in the production environment. |
Unaffected production environment |
|
Scenarios
Traffic mirroring allows you to test a service that is running in the production environment without affecting the end users. You can perform benchmark testing for two versions of a service to determine whether the new version can process inbound requests in the same way as the existing version.
The following table describes typical scenarios where you can use traffic mirroring.
Scenario | Description |
---|---|
Production traffic mirroring for trial runs and simulation tests | You can mirror traffic from a production cluster to a test cluster for testing. This does not affect the critical request path in the production environment. Assume that you want to replace or transform an old system with or into a new system. You can mirror and import the production traffic in the old system to the new system for a trial run. If you want to perform an experimental architecture adjustment, you can also mirror the production traffic for simulation tests. |
New version verification | You can compare the output results of production traffic and mirrored traffic in real time. You can use the mirrored traffic in drills before you release a new service. All the production traffic can be mirrored. Traditional manual drills are performed based on sample data. (It would be hard to predict how a service will respond to production traffic.) With mirrored production traffic, you can simulate all the situations in the production environment, such as exceptional special characters and tokens that suffer malicious attacks. This helps you understand the processing and troubleshooting capabilities of the service to be released. |
Isolation of database data from test data | If you want to test data processing performance, you can import test data to an empty database and then mirror production traffic to this test database. This isolates test data and data in the production database. |
Running service troubleshooting | When an unexpected issue occurs to a running service, it is hard to reproduce the issue on an on-premises network. In this case, you can start a temporary service, and mirror traffic from the running service to the temporary service for debugging. This troubleshooting way does not affect the running service. |
User behavior logging | Samples and data are critical for recommendation system algorithms. The biggest challenge of traditional automated testing for algorithms-dependent applications is the lack of real-world user behavior data. Traffic mirroring allows you to store user behavior data in logs. The log data can be used in simulation tests for building recommendation system algorithms. It can also be used as a big data source for user profile analysis. |
Sample code for using traffic mirroring
The following example YAML file shows how to use traffic mirroring in Istio. In the example, VirtualService routes all traffic to the v1 subset and mirrors the traffic to the v1-mirroring subset. When requests are sent to the v1 subset, the requests are copied and sent to the v1-mirroring subset.
After the v1-mirroring subset sends requests to the v1 version of the application, you can view the application logs. You can see that when the application is invoked, the response is from the v1 subset. You can also see that requests are mirrored to the v1-mirroring subset.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapp-traffic-mirroring
spec:
hosts:
- myapp
http:
- route:
- destination:
host: myapp.default.svc.cluster.local
port:
number: 8000
subset: v1
weight: 100
mirror:
host: myapp.default.svc.cluster.local
port:
number: 8000
subset: v1-mirroring
Enable traffic mirroring for services within a cluster

Step 1: Deploy sample application services
- Create an httpbin.yaml file that contains the following content:
apiVersion: v1 kind: ServiceAccount metadata: name: httpbin --- apiVersion: v1 kind: Service metadata: name: httpbin labels: app: httpbin service: httpbin spec: ports: - name: http port: 8000 targetPort: 80 selector: app: httpbin --- apiVersion: apps/v1 kind: Deployment metadata: name: httpbin-v1 spec: replicas: 1 selector: matchLabels: app: httpbin version: v1 template: metadata: labels: app: httpbin version: v1 spec: serviceAccountName: httpbin containers: - image: docker.io/kennethreitz/httpbin imagePullPolicy: IfNotPresent name: httpbin ports: - containerPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: httpbin-v1-mirroring spec: replicas: 1 selector: matchLabels: app: httpbin version: v1-mirroring template: metadata: labels: app: httpbin version: v1-mirroring spec: serviceAccountName: httpbin containers: - image: docker.io/kennethreitz/httpbin imagePullPolicy: IfNotPresent name: httpbin ports: - containerPort: 80
- Run the following command to deploy the v1 and v1-mirroring versions of the
httpbin
service:kubectl apply -f httpbin.yaml
- Create a sleep.yaml file that contains the following content:
apiVersion: v1 kind: ServiceAccount metadata: name: sleep --- apiVersion: v1 kind: Service metadata: name: sleep labels: app: sleep service: sleep spec: ports: - port: 80 name: http selector: app: sleep --- apiVersion: apps/v1 kind: Deployment metadata: name: sleep spec: replicas: 1 selector: matchLabels: app: sleep template: metadata: labels: app: sleep spec: terminationGracePeriodSeconds: 0 serviceAccountName: sleep containers: - name: sleep image: curlimages/curl command: ["/bin/sleep", "infinity"] imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /etc/sleep/tls name: secret-volume volumes: - name: secret-volume secret: secretName: sleep-secret optional: true ---
- Run the following command to deploy the client application service
sleep
:kubectl apply -f httpbin.yaml
Step 2: Create routing rules
- Create a destination rule that specifies v1 and v1-mirroring versions.
- Log on to the ASM console. In the left-side navigation pane, choose .
- On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose . On the page that appears, click Create.
- On the Create page, configure parameters and click Create. The following figure shows you how to configure the parameters.
- Create a virtual service to route all traffic to v1 and mirror half of the traffic to v1-mirroring.
Step 3: Send requests
- When the application services are running properly, run the following commands to send requests from
sleep
tohttpbin
.export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name}) kubectl exec -it $SLEEP_POD -c sleep -- curl http://httpbin:8000/headers
- View the pod logs of v1 and v1-mirroring. The logs show that all traffic is sent to v1 and half of the traffic is mirrored to v1-mirroring. The result meets expectations.The following figure compares the pod logs of v1 and v1-mirroring. It indicates that traffic sent to v1 is mirrored to v1-mirroring. The mirrored traffic packet size is larger because the requests that are mirrored to v1-mirroring have their
authority
header appended with-shadow
.