Alibaba Cloud Service Mesh:Use traffic mirroring across clusters at the service mesh layer

When to use traffic mirroring

How traffic mirroring works

Two behaviors to understand before you configure mirroring:

• Host header modification: Mirrored requests have -shadow appended to the Host/Authority header. For example, if the original Host is httpbin, the mirrored request Host becomes httpbin-shadow. The host field in the VirtualService mirror section only determines the destination address -- it does not overwrite the original Host header.

• Mirror percentage: The mirrorPercentage.value field controls what fraction of traffic to mirror. If omitted, 100% of traffic is mirrored.

Sample VirtualService configuration

The following VirtualService routes all traffic to the v1 subset and mirrors it to the v1-mirroring subset:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-traffic-mirroring
spec:
  hosts:
    - myapp
  http:
    - route:
        - destination:
            host: myapp.default.svc.cluster.local
            port:
              number: 8000
            subset: v1
          weight: 100
      mirror:
        host: myapp.default.svc.cluster.local
        port:
          number: 8000
        subset: v1-mirroring

After you apply this configuration, every request to myapp is served by the v1 subset, while a copy of each request is sent to the v1-mirroring subset.

Prerequisites

Before you begin, make sure that you have:

• Two Kubernetes clusters (Cluster A and Cluster B) with ASM configured

• kubectl configured to access both clusters

• Automatic sidecar injection enabled in the target namespaces

Step 1: Deploy the httpbin service in Cluster B

1. Create a file named httpbin.yaml with the following content:

   Expand to view httpbin.yaml

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: httpbin
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: httpbin
     labels:
       app: httpbin
       service: httpbin
   spec:
     ports:
     - name: http
       port: 8000
       targetPort: 80
     selector:
       app: httpbin
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin-v1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         serviceAccountName: httpbin
         containers:
         - image: docker.io/kennethreitz/httpbin
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 80

2. Deploy the httpbin v1 application:

   kubectl apply -f httpbin.yaml

3. Verify the Pod is running:

   kubectl get pods -l app=httpbin

   The expected output is similar to:

   NAME                         READY   STATUS    RESTARTS   AGE
   httpbin-v1-xxxxxxxxx-xxxxx   2/2     Running   0          30s

   A 2/2 in the READY column confirms the sidecar proxy was injected alongside the application container.

Step 2: Configure a routing rule for the ingress gateway in Cluster B

This step exposes the httpbin service through the Cluster B ingress gateway so that mirrored traffic from Cluster A can reach it.

1. Create a file named httpbin-gateway.yaml with the following content:

   Expand to view httpbin-gateway.yaml

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: httpbin-gateway
   spec:
     selector:
       istio: ingressgateway
     servers:
     - port:
         number: 80
         name: http
         protocol: HTTP
       hosts:
       - "*"
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: httpbin
   spec:
     hosts:
     - "*"
     gateways:
     - httpbin-gateway
     http:
     - match:
       - uri:
           prefix: /headers
       route:
       - destination:
           host: httpbin
           port:
             number: 8000

2. Apply the Gateway and VirtualService:

   kubectl apply -f httpbin-gateway.yaml

3. Send a test request to verify the service is reachable through the ingress gateway:

   curl http://<cluster-b-ingress-gateway-ip>/headers

   Replace <cluster-b-ingress-gateway-ip> with the external IP of the ingress gateway in Cluster B.

   The expected output is similar to:

   {
     "headers": {
       "Accept": "*/*",
       "Host": "47.99.XX.XX",
       "User-Agent": "curl/7.79.1",
       "X-Envoy-Attempt-Count": "1",
       "X-Envoy-External-Address": "120.244.XXX.XXX",
       "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=158e4ef69876550c34d10e3bfbd8d43f5ab481b16ba0e90b4e38a2d53ac****;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
     }
   }

Step 3: Configure traffic mirroring in Cluster A

Deploy the same httpbin application in Cluster A if you have not already done so. Use the same httpbin.yaml from Step 1:

kubectl apply -f httpbin.yaml

Next, create four resources in Cluster A:

• A ServiceEntry that registers the Cluster B ingress gateway as an external service

• A DestinationRule that defines the v1 subset for the local httpbin service

• A Gateway and VirtualService that route traffic to the local httpbin service and mirror it to Cluster B

Step 4: Verify that traffic mirroring works

"routes": [
  {
    "match": {
      "prefix": "/headers",
      "case_sensitive": true
    },
    "route": {
      "cluster": "outbound|8000|v1|httpbin.default.svc.cluster.local",
      "timeout": "0s",
      "retry_policy": {
        "retry_on": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
        "num_retries": 2,
        "retry_host_predicate": [
          {
            "name": "envoy.retry_host_predicates.previous_hosts",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
            }
          }
        ],
        "host_selection_retry_max_attempts": "5",
        "retriable_status_codes": [503]
      },
      "request_mirror_policies": [
        {
          "cluster": "outbound|80||httpbin.mirror.cluster-b",
          "runtime_fraction": {
            "default_value": {
              "numerator": 500000,
              "denominator": "MILLION"
            }
          },
          "trace_sampled": false
        }
      ]
    }
  }
]

Clean up

To remove the resources created in this tutorial:

On Cluster B:

kubectl delete -f httpbin-gateway.yaml
kubectl delete -f httpbin.yaml

On Cluster A:

kubectl delete -f httpbin-gateway.yaml
kubectl delete -f httpbin-destinationrule.yaml
kubectl delete -f httpbin-cluster-b.yaml
kubectl delete -f httpbin.yaml