Alibaba Cloud Service Mesh:Accelerate pod network communications in ASM with eRDMA

Step 1: Initialize nodes

SMC accelerates network communications through the elastic RDMA interface (ERI). Prepare the nodes before enabling SMC.

1. Upgrade the Alibaba Cloud Linux 3 kernel to version 5.10.134-17.3 or later. See Change the kernel version.

2. Install the erdma-controller and enable Shared Memory Communication over RDMA (SMC-R). See Use eRDMA to accelerate container networking.

Step 2: Deploy test workloads

The following steps use Fortio, a load testing tool, to generate traffic between a client pod and a server pod. Both pods require sidecar proxy injection.

1. Enable automatic sidecar proxy injection for the default namespace.

2. Create a file named fortioserver.yaml with the following content:

   fortioserver.yaml

   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: fortioserver
   spec:
     ports:
     - name: http-echo
       port: 8080
       protocol: TCP
     - name: tcp-echoa
       port: 8078
       protocol: TCP
     - name: grpc-ping
       port: 8079
       protocol: TCP
     selector:
       app: fortioserver
     type: ClusterIP
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: fortioserver
     name: fortioserver
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: fortioserver
     template:
       metadata:
         labels:
           app: fortioserver
         annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/proxyCPULimit: 2000m
           proxy.istio.io/config: |
             concurrency: 2
       spec:
         shareProcessNamespace: true
         containers:
         - name: captured
           image: fortio/fortio:latest_release
           ports:
           - containerPort: 8080
             protocol: TCP
           - containerPort: 8078
             protocol: TCP
           - containerPort: 8079
             protocol: TCP
         - name: anolis
           securityContext:
             runAsUser: 0
           image: openanolis/anolisos:latest
           args:
           - /bin/sleep
           - 3650d
   ---
   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off"
     name: fortioclient
   spec:
     ports:
     - name: http-report
       port: 8080
       protocol: TCP
     selector:
       app: fortioclient
     type: LoadBalancer
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: fortioclient
     name: fortioclient
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: fortioclient
     template:
       metadata:
         annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/proxyCPULimit: 4000m
           proxy.istio.io/config: |
             concurrency: 4
         labels:
           app: fortioclient
       spec:
         shareProcessNamespace: true
         affinity:
           podAntiAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                 matchExpressions:
                 - key: app
                   operator: In
                   values:
                   - fortioserver
               topologyKey: "kubernetes.io/hostname"
         containers:
         - name: captured
           volumeMounts:
           - name: shared-data
             mountPath: /var/lib/fortio
           image: fortio/fortio:latest_release
           ports:
           - containerPort: 8080
             protocol: TCP
         - name: anolis
           securityContext:
             runAsUser: 0
           image: openanolis/anolisos:latest
           args:
           - /bin/sleep
           - 3650d
         volumes:
         - name: shared-data
           emptyDir: {}

   Key details in this manifest:

   • The fortioclient and fortioserver Deployments each include an anolis sidecar container for diagnostic access.

   • Pod anti-affinity ensures the client and server pods land on different nodes, so traffic crosses the network.

   • The sidecar proxy CPU limits and concurrency values are tuned for the benchmark (2 workers for the server, 4 for the client).

3. Connect to the ACK cluster with kubectl using the kubeconfig file, then deploy the workloads:

      kubectl apply -f fortioserver.yaml

4. Verify that both pods are running: Expected output: Both pods show 3/3 under READY (application container + anolis container + sidecar proxy).

      kubectl get pods | grep fortio

      NAME                            READY   STATUS    RESTARTS
      fortioclient-8569b98544-9qqbj   3/3     Running   0
      fortioserver-7cd5c46c49-mwbtq   3/3     Running   0

Step 3: Run a baseline benchmark

Measure latency and throughput before enabling SMC. Fortio exposes a web console on port 8080.

1. Forward the fortioclient port to your local machine:

      kubectl port-forward service/fortioclient 8080:8080

2. Open http://localhost:8080/fortio in a browser. Configure the following test parameters:

   Parameter	Value
   URL	http://fortioserver:8080/echo
   QPS	100000
   Duration	30s
   Threads/Simultaneous connections	64
   Payload	xhsyL4ELNoUUbC3WEyvaz0qoHcNYUh0j2YHJTpltJueyXlSgf7xkGqc5RcSJBtqUENNjVHNnGXmoMyILWsrZL1O2uordH6nLE7fY6h5TfTJCZtff3Wib8YgzASha8T8g (128 bytes)

   

3. Click Start. Wait for the progress bar to complete.

   

4. Review the results. The histogram displays request latency distribution. The x-axis represents latency, the y-axis shows the number of processed requests, and the purple curve traces throughput across response times. The top of the chart reports P50, P75, P90, P99, and P99.9 latency percentiles. Record these values to compare against the SMC-enabled results.

   Test results vary depending on the test environment. The figures shown here are for reference only.

   

Step 4: Enable SMC for the ASM instance and workloads

SMC requires two configuration changes: one at the mesh level and one at the workload level. Both client-side and server-side pods must carry the SMC annotation.

kubectl edit asmmeshconfig

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMMeshConfig
metadata:
  name: default
spec:
  ambientConfiguration:
    redirectMode: ""
    waypoint: {}
    ztunnel: {}
  cniConfiguration:
    enabled: true
    repair: {}
  smcEnabled: true

Step 5: Benchmark with SMC enabled

After the pods restart, re-establish the port-forward (as in Step 3) and run the same test with identical parameters.

kubectl port-forward service/fortioclient 8080:8080

Open http://localhost:8080/fortio, enter the same settings, and click Start.

Compare the results against the baseline. With SMC enabled, expect lower request latencies and higher QPS.

Why does database access fail after enabling SMC?

A load balancer in the connection path may misidentify the TCP-kind 254 option used by the SMC protocol. This happens when the load balancer does not fully comply with TCP specifications.

To diagnose:

1. Install the aliyun-smc-extensions toolkit on the node:

      sudo yum install -y aliyun-smc-extensions

2. Check the communication link to the service URL: Replace <url> with the database service URL. The tool reports TCP option issues such as replay, conflict, and overlength.

      aliyunsmc-check syn_check --url <url>

   

If the tool detects TCP option processing problems on the path to the service, disable SMC for the affected pod by removing the smc.asm.alibabacloud.com/enabled: "true" annotation from its Deployment.