Use ASM to achieve gRPC load balancing - Alibaba Cloud Service Mesh

gRPC uses persistent HTTP/2 connections, so a standard Kubernetes TCP load balancer pins all requests from one client to a single pod. Service Mesh (ASM) solves this by injecting sidecar proxies that balance gRPC traffic at the request level, distributing requests evenly across pods or at custom ratios you define through traffic rules.

How it works

gRPC is an HTTP/2-based RPC framework that serializes data with protocol buffers into a compact binary format. Because gRPC generates client and server code from .proto files in multiple languages, it is widely used in polyglot microservices architectures.

The load balancing problem stems from a fundamental difference between HTTP/1.1 and HTTP/2:

HTTP/1.1 creates a new TCP connection per request. A TCP load balancer distributes these connections across pods, naturally spreading the load.
HTTP/2 (used by gRPC) maintains a persistent TCP connection. Once a client connects to a pod, every subsequent request reuses that same connection and bypasses the load balancer entirely.

When a gRPC client sends 100 requests, all 100 go to the same pod. As pods scale up or restart, the cluster becomes imbalanced.

ASM fixes this by injecting an Envoy sidecar proxy into each pod. The sidecar intercepts outbound gRPC calls and distributes individual requests across all available pods, regardless of the underlying HTTP/2 connection.

Reproduce the problem: unbalanced gRPC load without ASM

This section deploys a gRPC server and client without sidecar injection. All requests land on a single pod, demonstrating the problem.

Prerequisites

Before you begin, make sure that you have:

A Container Service for Kubernetes (ACK) cluster. See Create an ACK managed cluster
A namespace named grpc-nosidecar. See Create a namespace
kubectl connected to the cluster. See Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster

Deploy and test

Deploy the gRPC server istio-grpc-server in the grpc-nosidecar namespace.

Create an istio-grpc-server.yaml file with the following content:

istio-grpc-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v1
  labels:
    app: istio-grpc-server
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v1
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v1
    spec:
      containers:
      - args:
        - --address=0.0.0.0:8080
        image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
        name: istio-grpc-server
        ports:
        - containerPort: 8080
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v2
  labels:
    app: istio-grpc-server
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v2
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v2
    spec:
      containers:
        - args:
            - --address=0.0.0.0:8080
          image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
          imagePullPolicy: Always
          livenessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
          name: istio-grpc-server
          ports:
            - containerPort: 8080
          readinessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-server
  labels:
    app: istio-grpc-server
spec:
  ports:
  - name: grpc-backend
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-server
  type: ClusterIP
---

Apply the manifest:

kubectl apply -n grpc-nosidecar -f istio-grpc-server.yaml

Deploy the gRPC client istio-grpc-client in the grpc-nosidecar namespace.

Create an istio-grpc-client.yaml file with the following content:

istio-grpc-client.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-client
  labels:
    app: istio-grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-client
  template:
    metadata:
      labels:
        app: istio-grpc-client
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-client
        imagePullPolicy: Always
        command: ["/bin/sleep", "3650d"]
        name: istio-grpc-client
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-client
spec:
  ports:
  - name: grpc
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-client
  type: ClusterIP
---

Apply the manifest:

kubectl apply -n grpc-nosidecar -f istio-grpc-client.yaml

Verify that all pods are running:

kubectl get pod -n grpc-nosidecar

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-grpc-client-dd56bcb45-hvmjt       1/1     Running   0          95m
istio-grpc-server-v1-546d9876c4-j2p9r   1/1     Running   0          95m
istio-grpc-server-v2-66d9b8847-276bd    1/1     Running   0          95m

Each pod shows 1/1 in the READY column, which means no sidecar proxy is injected.

Open a shell in the gRPC client pod:

kubectl exec -it -n grpc-nosidecar <istio-grpc-client-pod> sh

Send 100 gRPC requests:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
...

All 100 requests go to the same pod. Without a service mesh, the gRPC client reuses a single HTTP/2 connection and every request is handled by one pod.

Balance gRPC load with ASM

This section deploys the same gRPC server and client in a sidecar-injected namespace, then uses traffic rules to control request distribution.

Prerequisites

Before you begin, make sure that you have:

An ASM instance with an ACK cluster added to it. See Add a cluster to an ASM instance
A namespace named grpc-sidecar with the istio-injection:enabled label. See Create a namespace
kubectl connected to the cluster. See Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster

Step 1: Deploy the gRPC server and client

Create an istio-grpc-server.yaml file with the following content:

istio-grpc-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v1
  labels:
    app: istio-grpc-server
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v1
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v1
    spec:
      containers:
      - args:
        - --address=0.0.0.0:8080
        image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
        name: istio-grpc-server
        ports:
        - containerPort: 8080
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v2
  labels:
    app: istio-grpc-server
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v2
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v2
    spec:
      containers:
        - args:
            - --address=0.0.0.0:8080
          image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
          imagePullPolicy: Always
          livenessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
          name: istio-grpc-server
          ports:
            - containerPort: 8080
          readinessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-server
  labels:
    app: istio-grpc-server
spec:
  ports:
  - name: grpc-backend
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-server
  type: ClusterIP
---

Create an istio-grpc-client.yaml file with the following content:

istio-grpc-client.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-client
  labels:
    app: istio-grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-client
  template:
    metadata:
      labels:
        app: istio-grpc-client
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-client
        imagePullPolicy: Always
        command: ["/bin/sleep", "3650d"]
        name: istio-grpc-client
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-client
spec:
  ports:
  - name: grpc
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-client
  type: ClusterIP
---

Deploy both resources:

kubectl apply -n grpc-sidecar -f istio-grpc-server.yaml
kubectl apply -n grpc-sidecar -f istio-grpc-client.yaml

Verify that all pods are running with sidecar injection:

kubectl get pod -n grpc-sidecar

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-grpc-client-dd56bcb45-zhfsg       2/2     Running   0          1h15m
istio-grpc-server-v1-546d9876c4-tndsm   2/2     Running   0          1h15m
istio-grpc-server-v2-66d9b8847-99v62    2/2     Running   0          1h15m

Each pod shows 2/2 in the READY column. The second container is the injected sidecar proxy.

Step 2: Verify default round-robin balancing

Without any traffic rules, the sidecar proxy distributes gRPC requests evenly across pods using round-robin scheduling.

Open a shell in the gRPC client pod:

kubectl exec -it -n grpc-sidecar <istio-grpc-client-pod> -c istio-grpc-client sh

Send 100 gRPC requests:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
...

Requests alternate between the v1 and v2 pods at roughly a 50:50 ratio. The sidecar proxy intercepts each gRPC call and routes it independently, solving the sticky-connection problem.

Step 3: Control traffic distribution with destination rules and virtual services

Create a DestinationRule and a VirtualService on the ASM control plane to route requests at a custom ratio -- for example, 90% to v1 and 10% to v2.

Create a namespace named grpc-sidecar in the ASM instance. See Create a namespace.

Create a DestinationRule in the grpc-sidecar namespace with the following content. See Manage destination rules.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dr-istio-grpc-server
spec:
  host: istio-grpc-server
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - name: v1
      labels:
        version: "v1"
    - name: v2
      labels:
        version: "v2"

Create a VirtualService in the grpc-sidecar namespace with the following content. See Manage virtual services.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs-istio-grpc-server
spec:
  hosts:
    - "istio-grpc-server"
  http:
    - match:
        - port: 8080
      route:
        - destination:
            host: istio-grpc-server
            subset: v1
          weight: 90
        - destination:
            host: istio-grpc-server
            subset: v2
          weight: 10

Open a shell in the gRPC client pod and send 100 requests:

kubectl exec -it -n grpc-sidecar <istio-grpc-client-pod> -c istio-grpc-client sh

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:56:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
...

Requests are distributed at approximately 90:10 between v1 and v2, matching the weights in the VirtualService.

Step 4: Adjust traffic weights

Modify the VirtualService weights to shift all traffic to v2.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > VirtualService.
On the VirtualService page, select the grpc-sidecar namespace from the Namespace drop-down list. Find vs-istio-grpc-server and click YAML in the Actions column.

In the Edit dialog box, update the weights and click OK:

route:
  - destination:
      host: istio-grpc-server
      subset: v1
    weight: 0
  - destination:
      host: istio-grpc-server
      subset: v2
    weight: 100

Open a shell in the gRPC client pod and send 100 requests:

kubectl exec -it -n grpc-sidecar <istio-grpc-client-pod> -c istio-grpc-client sh

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
...

All requests go to the v2 pod, confirming that the weight change takes effect immediately without redeploying pods.