Container Service for Kubernetes:Use an Ingress controller to mirror network traffic

Prerequisites

Before you begin, ensure that you have:

• Two ACK clusters — one for production (K8s Product Cluster) and one for staging (K8s Stage Cluster). See Create an ACK managed cluster

• The NGINX Ingress controller v1.2.1 or later installed. See Manage the NGINX Ingress controller

• kubectl configured with the kubeconfig file for each cluster. See Connect to ACK clusters by using kubectl

Use cases

• Stress testing before release: Mirror production traffic to staging to simulate real-world workloads — including abnormal traffic — before releasing a new system. This gives you a more accurate load profile than synthetic benchmarks.

• Troubleshooting production issues: When you can't locate a performance bottleneck in production, mirror traffic to staging to reproduce and diagnose the issue safely.

How it works

All requests destined for example.com on the production cluster are replicated and sent to example1.com on the staging cluster. The staging cluster receives the mirrored traffic without any changes to the production cluster's routing.

The NGINX Ingress controller uses the nginx.ingress.kubernetes.io/mirror-target annotation to duplicate requests to a specified destination. Use the mirror-host annotation to set the exact Host header value for the mirrored request.

Mirroring annotations

Step 1: Deploy an application in the staging cluster

Deploy a receiving application in K8s Stage Cluster. This deployment gives the mirrored traffic a destination. Don't change any production cluster configuration in this step.

1. Create my-nginx.yaml with the following content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
           imagePullPolicy: Always
           name: nginx
           ports:
           - containerPort: 80
             protocol: TCP
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
   spec:
     ports:
     - port: 80
       protocol: TCP
       targetPort: 80
     selector:
       app: nginx
     type: NodePort
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: nginx-ingress
   spec:
     rules:
     - host: example1.com
       http:
         paths:
         - path: /
           backend:
             service:
               name: nginx-service
               port:
                 number: 80
           pathType: ImplementationSpecific

2. Deploy the application:

   kubectl apply -f my-nginx.yaml

3. Verify the Ingress is assigned an external IP address:

   kubectl get ing nginx-ingress

   Expected output:

   NAME            HOSTS         ADDRESS       PORTS   AGE
   nginx-ingress   example1.com  47.**.**.53   80      8m

   Note the ADDRESS value (47.**.**.53). You'll use this IP address when configuring traffic mirroring in Step 2.

4. Confirm the staging application is reachable:

   curl http://example1.com

Step 2: Configure traffic mirroring in the production cluster

Deploy the application and Ingress in K8s Product Cluster, then add the mirroring annotations to redirect a copy of all traffic to the staging cluster.

1. Create my-nginx.yaml with the following content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
           imagePullPolicy: Always
           name: nginx
           ports:
           - containerPort: 80
             protocol: TCP
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
   spec:
     ports:
     - port: 80
       protocol: TCP
       targetPort: 80
     selector:
       app: nginx
     type: NodePort

2. Deploy the application:

   kubectl apply -f my-nginx.yaml

3. Create my-ingress.yaml with the mirroring annotations. Replace 47.**.**.53 with the ADDRESS from Step 1, and example1.com with the staging cluster's HOSTS value:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: nginx-ingress
     annotations:
       nginx.ingress.kubernetes.io/mirror-target: "http://47.**.**.53$request_uri"
       nginx.ingress.kubernetes.io/mirror-request-body: "off"
       nginx.ingress.kubernetes.io/mirror-host: "example1.com"
   spec:
     rules:
     - host: example.com
       http:
         paths:
         - path: /
           backend:
             service:
               name: nginx-service
               port:
                 number: 80
           pathType: ImplementationSpecific

4. Deploy the Ingress:

   kubectl apply -f my-ingress.yaml

5. Verify the production Ingress is active:

   kubectl get ing nginx-ingress

   Expected output:

   NAME            HOSTS        ADDRESS       PORTS   AGE
   nginx-ingress   example.com  39.**.**.54   80      1m

6. Confirm the production application is reachable:

   curl http://example.com

Verify traffic mirroring

Send a request to the production domain and check the NGINX Ingress controller logs to confirm traffic is being duplicated to the staging cluster.

1. Send a request to the production application:

   curl http://example.com

2. Check the NGINX Ingress controller logs in the kube-system namespace of K8s Product Cluster:

   kubectl -n kube-system logs --tail=0 -f nginx-ingress-controller-674c96ffbc-9mc8n

   The log output shows duplicate entries — one for the original request to example.com and one for the mirrored request forwarded to the staging cluster.