Use traffic mirroring for services within a cluster - Alibaba Cloud Service Mesh

You can use the traffic mirroring feature to mirror production traffic to a test cluster or test service version. Testing that uses the mirrored production traffic mitigates risks involved in version changes without affecting the production environment. This topic describes what traffic mirroring is and how to use this feature for services within a cluster.

What is traffic mirroring?

The microservice architecture makes application development and deployment faster, but risks exist in changing service versions. Alibaba Cloud Service Mesh (ASM) provides the traffic mirroring feature to mitigate the risks. The feature, also called traffic shadowing, sends production traffic to a mirrored service in real time. The mirrored traffic happens out of band of the critical request path for the production service. When traffic is mirrored, the requests that are sent to the mirrored service version have their Host/Authority headers appended with -shadow. This distinguishes production traffic and mirrored traffic. You can use the feature to mirror production traffic to a test cluster or test service version before the cluster or service version is running in the production environment. This mitigates risks involved in version changes.

Benefits


Benefit	Description
Less-risky version deployment with more production-like test environment	You can copy production traffic to a test cluster or test service version and perform tests by using the mirrored use cases and traffic. More accurate test results mitigate deployment risks in the production environment.
Unaffected production environment	The mirrored traffic happens out of band of the critical request path for the production service. Any issues caused by the mirrored traffic do not affect the production environment. Requests are mirrored as "fire and forget", which means that the responses are discarded.

Scenarios

Traffic mirroring allows you to test a service that is running in the production environment without affecting the end users. You can perform benchmark testing for two versions of a service to determine whether the new version can process inbound requests in the same way as the existing version.

The following table describes typical scenarios where you can use traffic mirroring.


Scenario	Description
Production traffic mirroring for trial runs and simulation tests	You can mirror traffic from a production cluster to a test cluster for testing. This does not affect the critical request path in the production environment. Assume that you want to replace or transform an old system with or into a new system. You can mirror and import the production traffic in the old system to the new system for a trial run. If you want to perform an experimental architecture adjustment, you can also mirror the production traffic for simulation tests.
New version verification	You can compare the output results of production traffic and mirrored traffic in real time. You can use the mirrored traffic in drills before you release a new service. All the production traffic can be mirrored. Traditional manual drills are performed based on sample data. (It would be hard to predict how a service will respond to production traffic.) With mirrored production traffic, you can simulate all the situations in the production environment, such as exceptional special characters and tokens that suffer malicious attacks. This helps you understand the processing and troubleshooting capabilities of the service to be released.
Isolation of database data from test data	If you want to test data processing performance, you can import test data to an empty database and then mirror production traffic to this test database. This isolates test data and data in the production database.
Running service troubleshooting	When an unexpected issue occurs to a running service, it is hard to reproduce the issue on an on-premises network. In this case, you can start a temporary service, and mirror traffic from the running service to the temporary service for debugging. This troubleshooting way does not affect the running service.
User behavior logging	Samples and data are critical for recommendation system algorithms. The biggest challenge of traditional automated testing for algorithms-dependent applications is the lack of real-world user behavior data. Traffic mirroring allows you to store user behavior data in logs. The log data can be used in simulation tests for building recommendation system algorithms. It can also be used as a big data source for user profile analysis.

Sample code for using traffic mirroring

The following example YAML file shows how to use traffic mirroring in Istio. In the example, VirtualService routes all traffic to the v1 subset and mirrors the traffic to the v1-mirroring subset. When requests are sent to the v1 subset, the requests are copied and sent to the v1-mirroring subset.

After the v1-mirroring subset sends requests to the v1 version of the application, you can view the application logs. You can see that when the application is invoked, the response is from the v1 subset. You can also see that requests are mirrored to the v1-mirroring subset.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-traffic-mirroring
spec:
  hosts:
    - myapp
  http:
    - route:
        - destination:
            host: myapp.default.svc.cluster.local
            port:
              number: 8000
            subset: v1
          weight: 100
      mirror:
        host: myapp.default.svc.cluster.local
        port:
          number: 8000
        subset: v1-mirroring

Enable traffic mirroring for services within a cluster

In this example, all traffic is routed to v1 and half of the traffic is mirrored to v1-mirroring. v1 and v1-mirroring are two versions of the httpbin service. Use traffic mirroring for services within a cluster

Use traffic mirroring for services within a cluster

Step 1: Deploy sample application services

Create an httpbin.yaml file that contains the following content:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-v1-mirroring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1-mirroring
  template:
    metadata:
      labels:
        app: httpbin
        version: v1-mirroring
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Run the following command to deploy the v1 and v1-mirroring versions of the httpbin service:
```
kubectl apply -f httpbin.yaml
```

Create a sleep.yaml file that contains the following content:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---

Run the following command to deploy the client application service sleep:
```
kubectl apply -f httpbin.yaml
```

Step 2: Create routing rules

Create a destination rule that specifies v1 and v1-mirroring versions.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > DestinationRule. On the page that appears, click Create.
3. On the Create page, configure parameters and click Create. The following figure shows you how to configure the parameters.

Create a virtual service to route all traffic to v1 and mirror half of the traffic to v1-mirroring.

On the details page of the ASM instance, choose Traffic Management Center > VirtualService in the left-side navigation pane.

Use one of the following methods to create a virtual service.

Method 1:
On the VirtualService page, click Create. On the Create page, configure parameters and click Create. The following figures show you how to configure the parameters.

Method 2:

On the VirtualService page, click Create from YAML. On the Create page, select the default namespace, select a template, replace the content in the YAML code editor with the following content, and then click Create.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v1-mirroring
    labels:
      version: v1-mirroring
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-traffic-mirroring
spec:
  hosts:
    - httpbin
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000
            subset: v1
          weight: 100
      mirror:
        host: httpbin
        port:
          number: 8000
        subset: v1-mirroring
      mirrorPercentage:
        value: 50

Step 3: Send requests

When the application services are running properly, run the following commands to send requests from sleep to httpbin.

export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl exec -it $SLEEP_POD -c sleep -- curl http://httpbin:8000/headers

View the pod logs of v1 and v1-mirroring.
The logs show that all traffic is sent to v1 and half of the traffic is mirrored to v1-mirroring. The result meets expectations.
The following figure compares the pod logs of v1 and v1-mirroring. It indicates that traffic sent to v1 is mirrored to v1-mirroring. The mirrored traffic packet size is larger because the requests that are mirrored to v1-mirroring have their authority header appended with -shadow.