Use ASM to achieve gRPC load balancing - Alibaba Cloud Service Mesh

When you use HTTP 2-based gRPC to call services in Kubernetes, a connection is always pointed to a specific pod. If the gRPC client sends multiple requests, all of the requests are routed to the pod for processing, which causes an unbalanced load. This topic describes the unbalanced load issue between gRPC service calls. This topic also shows you how to use Service Mesh (ASM) to achieve gRPC load balancing.

Background information

gRPC is an HTTP 2-based communication protocol for services. You can specify service definitions in a format that is called protocol buffers. The data of service calls is serialized into a small binary format for transmission. gRPC allows you to generate boilerplate code from .proto files into multiple programming languages. This makes gRPC an ideal choice for polyglot microservices.

HTTP 1.1-based remote procedure calls (RPCs) build temporary connections between the client and pods. In this case, a TCP load balancer is enough to balance the load between RPCs. However, for an HTTP 2-based gRPC service call, the TCP connection between the client and a pod is persistent. If the pod expires, the load in the cluster becomes unbalanced.

Unbalanced load between gRPC service calls

The following example shows unbalanced load between gRPC service calls.

Prerequisites

A Container Service for Kubernetes (ACK) cluster is created. For more information, see Create an ACK managed cluster.
A namespace named grpc-nosidecar is created. For more information, see Create a namespace.
The cluster is connected by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Procedure

Deploy a gRPC server that is named istio-grpc-server in the grpc-nosidecar namespace.

Create an istio-grpc-server.yaml file that contains the following content:

Show the istio-grpc-server.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v1
  labels:
    app: istio-grpc-server
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v1
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v1
    spec:
      containers:
      - args:
        - --address=0.0.0.0:8080
        image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
        name: istio-grpc-server
        ports:
        - containerPort: 8080
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v2
  labels:
    app: istio-grpc-server
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v2
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v2
    spec:
      containers:
        - args:
            - --address=0.0.0.0:8080
          image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
          imagePullPolicy: Always
          livenessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
          name: istio-grpc-server
          ports:
            - containerPort: 8080
          readinessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-server
  labels:
    app: istio-grpc-server
spec:
  ports:
  - name: grpc-backend
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-server
  type: ClusterIP
---

Run the following command to deploy the gRPC server istio-grpc-server:
```
kubectl apply -n grpc-nosidecar -f istio-grpc-server.yaml
```

Deploy a gRPC client that is named istio-grpc-client in the grpc-nosidecar namespace.

Create an istio-grpc-client.yaml file that contains the following content:

Show the istio-grpc-client.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-client
  labels:
    app: istio-grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-client
  template:
    metadata:
      labels:
        app: istio-grpc-client
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-client
        imagePullPolicy: Always
        command: ["/bin/sleep", "3650d"]
        name: istio-grpc-client
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-client
spec:
  ports:
  - name: grpc
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-client
  type: ClusterIP
---

Run the following command to deploy the gRPC client istio-grpc-client:
```
kubectl apply -n grpc-nosidecar -f istio-grpc-client.yaml
```

Run the following command to query the pod status:

kubectl get pod -n grpc-nosidecar

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-grpc-client-dd56bcb45-hvmjt       1/1     Running   0          95m
istio-grpc-server-v1-546d9876c4-j2p9r   1/1     Running   0          95m
istio-grpc-server-v2-66d9b8847-276bd    1/1     Running   0          95m

Run the following command to log on to the default container of the pod of the gRPC client:
```
kubectl exec -it -n grpc-nosidecar istio-grpc-client-dd56bcb45-hvmjt sh
```

After you log on to the container, run the following command:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd
2024/02/07 14:37:14 Hello world from istio-grpc-server-v2-66d9b8847-276bd

The command output indicates that all requests are routed to the same pod. All requests from the gRPC client are processed by the same pod, which causes unbalanced load.

Use ASM to balance the load between gRPC service calls

The following example describes how to use ASM to achieve gRPC load balancing:

Prerequisites

An ASM instance is created and an ACK cluster is added to the instance. For more information, see Add a cluster to an ASM instance.
A namespace named grpc-sidecar is created and the istio-injection:enabled label is added to the namespace. For more information, see Create a namespace.
You are connected to the cluster by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Procedure

Deploy a gRPC server that is named istio-grpc-server in the grpc-sidecar namespace.

Create an istio-grpc-server.yaml file that contains the following content:

Show the istio-grpc-server.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v1
  labels:
    app: istio-grpc-server
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v1
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v1
    spec:
      containers:
      - args:
        - --address=0.0.0.0:8080
        image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
        name: istio-grpc-server
        ports:
        - containerPort: 8080
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:8080
          initialDelaySeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-server-v2
  labels:
    app: istio-grpc-server
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-server
      version: v2
  template:
    metadata:
      labels:
        app: istio-grpc-server
        version: v2
    spec:
      containers:
        - args:
            - --address=0.0.0.0:8080
          image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-server
          imagePullPolicy: Always
          livenessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
          name: istio-grpc-server
          ports:
            - containerPort: 8080
          readinessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:8080
            initialDelaySeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-server
  labels:
    app: istio-grpc-server
spec:
  ports:
  - name: grpc-backend
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-server
  type: ClusterIP
---

Run the following command to deploy the gRPC server istio-grpc-server:
```
kubectl apply -n grpc-sidecar -f istio-grpc-server.yaml
```

Deploy a gRPC client that is named istio-grpc-client in the grpc-sidecar namespace.

Create an istio-grpc-client.yaml file that contains the following content:

Show the istio-grpc-client.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-grpc-client
  labels:
    app: istio-grpc-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: istio-grpc-client
  template:
    metadata:
      labels:
        app: istio-grpc-client
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/aliacs-app-catalog/istio-grpc-client
        imagePullPolicy: Always
        command: ["/bin/sleep", "3650d"]
        name: istio-grpc-client
---
apiVersion: v1
kind: Service
metadata:
  name: istio-grpc-client
spec:
  ports:
  - name: grpc
    port: 8080
    protocol: TCP
  selector:
    app: istio-grpc-client
  type: ClusterIP
---

Run the following command to deploy the gRPC client istio-grpc-client:
```
kubectl apply -n grpc-sidecar -f istio-grpc-client.yaml
```

Run the following command to query the pod status:

kubectl get pod -n grpc-sidecar

Expected output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-grpc-client-dd56bcb45-zhfsg       2/2     Running   0          1h15m
istio-grpc-server-v1-546d9876c4-tndsm   2/2     Running   0          1h15m
istio-grpc-server-v2-66d9b8847-99v62    2/2     Running   0          1h15m

The command output indicates that each pod contains two containers, one of which runs the injected sidecar proxy.

Run the following command to log on to the default container of the pod of the gRPC client:
```
kubectl exec -it -n grpc-sidecar istio-grpc-client-dd56bcb45-zhfsg -c istio-grpc-client sh
```

After you log on to the container, run the following command:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:53:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:53:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62

The command output indicates that requests are routed to the two pods at a ratio of almost 50 to 50 in round-robin scheduling mode:

Configure the control plane for the ASM instance.

Create a namespace named grpc-sidecar in the ASM instance. For more information, see Create a namespace.

Create a destination rule in the grpc-sidecar namespace by using the following content. For more information, see Manage destination rules.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: dr-istio-grpc-server
spec:
  host: istio-grpc-server
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
    - name: v1
      labels:
        version: "v1"
    - name: v2
      labels:
        version: "v2"

Create a virtual service in the grpc-sidecar namespace by using the following content. For more information, see Manage virtual services.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs-istio-grpc-server
spec:
  hosts:
    - "istio-grpc-server"
  http:
    - match:
        - port: 8080
      route:
        - destination:
            host: istio-grpc-server
            subset: v1
          weight: 90
        - destination:
            host: istio-grpc-server
            subset: v2
          weight: 10

Run the following command to log on to the default container of the pod of the gRPC client:
```
kubectl exec -it -n grpc-sidecar istio-grpc-client-dd56bcb45-zhfsg -c istio-grpc-client sh
```

After you log on to the container, run the following command:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:56:16 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm
2024/02/07 14:56:16 Hello world from istio-grpc-server-v1-546d9876c4-tndsm

The command output indicates that requests are routed to the two pods at a ratio of 90 to 10, instead of 50 to 50 in round-robin scheduling mode.

Adjust the values of the weight parameters in the virtual service and observe the distribution of requests.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > VirtualService.
In the upper part of the VirtualService page, select the grpc-sidecar namespace from the Namespace drop-down list. Find the vs-istio-grpc-server virtual service and click YAML in the Actions column. In the Edit dialog box, change the values of the weight parameters in the virtual service and click OK.
Change the values of the weight parameters according to the values shown in the following code block:
```
     route:
        - destination:
            host: istio-grpc-server
            subset: v1
          weight: 0
        - destination:
            host: istio-grpc-server
            subset: v2
          weight: 100
```
Run the following command to log on to the default container of the pod of the gRPC client:
```
kubectl exec -it -n grpc-sidecar istio-grpc-client-dd56bcb45-zhfsg -c istio-grpc-client sh
```

After you log on to the container, run the following command:

/bin/greeter-client --insecure=true --address=istio-grpc-server:8080 --repeat=100

Expected output:

2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62
2024/02/07 14:58:30 Hello world from istio-grpc-server-v2-66d9b8847-99v62

The command output indicates that all requests are routed to the pod of the version v2.