Reduce Sidecar Configuration Push Latency with ASM Recommendations - Alibaba Cloud Service Mesh - Alibaba Cloud - Alibaba Cloud Service Mesh

Prerequisites

The cluster is added to the ASM instance.
You are connected to an ACK cluster by using kubectl. For more information, see Obtain the KubeConfig file of a cluster and use kubectl to connect to the cluster.
You use Log Service to collect access logs from the data plane. For more information, see Use Log Service to collect access logs from data-plane clusters.
Control-plane log collection is enabled for the ASM instance. The test measures the configuration push efficiency of the control plane from the content of the control plane logs. For more information, see Enable control-plane log collection and log-based alerting.

Test environment

The test deploys multiple sleep applications and httpbin applications in one namespace that is managed by the ASM instance. A Sidecar is injected into every test pod, and the cluster contains a large number of services with only a few call dependencies between them. This topic uses the default namespace, which is referenced by the sample manifests, commands, and control plane logs. For more information about namespaces in an ASM instance, see Manage global namespaces.

httpbin applications — After the httpbin application starts, it exposes an HTTP service on port 8000. The application simulates the large number of services called inside the cluster. The test generates 200 httpbin application deployments and deploys 400 httpbin application pods in the cluster.
sleep applications — The sleep application contains a curl container. Modify the command field of the application deployment so that the sleep application calls the services provided by multiple httpbin containers before it sleeps. The application simulates a service in the cluster that depends on other services. The test gives each sleep application a dependency on 10 httpbin applications. Therefore, 20 sleep application deployments and 20 sleep pods are deployed in total.
Cluster scale — A total of 420 pods run in the test cluster.

Deploy the test applications in the cluster

Deploy multiple httpbin applications in the cluster.

Use the following YAML content to create a YAML file named httpbin-{i}.yaml.

Note

In httpbin-{i}, replace {i} with a specific number to generate multiple numbered httpbin services. Use this template to generate any number of httpbin applications. The maximum number of applications depends on the size of your cluster.

The following YAML content defines the ServiceAccount, Service, and Deployment of one numbered httpbin application:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: httpbin-{i}
    service: httpbin-{i}
  name: httpbin-{i}
  namespace: default
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin-{i}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: httpbin-{i}
  name: httpbin-{i}
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin-{i}
      version: v1
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: httpbin-{i}
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
      serviceAccountName: httpbin

Run the following command to create the httpbin-{i} application.
```
kubectl apply -f httpbin-{i}.yaml -n default
```
Expected output:
```
deployment.apps/httpbin-{i} created
```

Deploy sleep applications in the cluster.

Use the following YAML content to create a YAML file named sleep-{i}.yaml.

Note

In sleep-{i}, replace {i} with a specific number to generate multiple numbered sleep services. In this template, you add the curl httpbin-{i*10}:8000 command parameter to the args field of the sleep application deployment to simulate calls to different httpbin applications. The number of the httpbin service that you call must not exceed the number of the httpbin services that you deployed earlier. Otherwise, the call is invalid.

The following YAML content defines the ServiceAccount, Service, and Deployment of one numbered sleep application:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: sleep-{i}
    service: sleep-{i}
  name: sleep-{i}
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    targetPort: 0
  selector:
    app: sleep-{i}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: sleep-{i}
  name: sleep-{i}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep-{i}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: sleep-{i}
    spec:
      containers:
      - args:
        - curl httpbin-{i*10}:8000; curl httpbin-{i*10+1}:8000; curl httpbin-{i*10+2}:8000; curl httpbin-{i*10+3}:8000;
          curl httpbin-{i*10+4}:8000; curl httpbin-{i*10+5}:8000; curl httpbin-{i*10+6}:8000; curl httpbin-{i*10+7}:8000;
          curl httpbin-{i*10+8}:8000; curl httpbin-{i*10+9}:8000; sleep 3650d
        command:
        - /bin/sh
        - -c
        image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: sleep
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      serviceAccountName: sleep
      terminationGracePeriodSeconds: 0
      volumes:
      - name: secret-volume
        secret:
          optional: true
          secretName: sleep-secret

Run the following command to create the sleep-{i} application.
```
kubectl apply -f sleep-{i}.yaml -n default
```
Expected output:
```
deployment.apps/sleep-{i} created
```

Test the control plane configuration push before a Sidecar resource is applied

Before you apply a Sidecar resource, measure two baseline values in the test cluster: the size of the Sidecar configuration in a single pod, and the time that the ASM control plane needs to complete one configuration push.

Sidecar configuration size before a Sidecar resource is applied

Run the following command to identify the pod name of the httpbin-0 application.

kubectl get pod -n default | grep httpbin-0

Expected output:

NAME                         READY   STATUS    RESTARTS   AGE
httpbin-0-756995d867-jljgp   2/2     Running   0          9m15s
httpbin-0-756995d867-whstr   2/2     Running   0          9m15s

Run the following command to download the Sidecar configuration of the pod that runs the httpbin-0 application to your local machine. Replace httpbin-0-756995d867-jljgp with a pod name returned in the previous step.
```
kubectl exec -it httpbin-0-756995d867-jljgp -c istio-proxy -n default -- curl -s localhost:15000/config_dump > config_dump.json
```
Run the following command to view the size of the Sidecar configuration file.
```
du -sh config_dump.json
```
Expected output:
```
1.2M    config_dump.json
```

The expected output shows that the Sidecar configuration is about 1.2 MB in the test scenario where 420 pods are deployed in the cluster. If a Sidecar is deployed for every pod in the cluster, the large amount of Sidecar configuration increases the push load on the control plane.

Control plane push efficiency before a Sidecar resource is applied

Apply a new virtual service rule to the httpbin-0 service in ASM to trigger one configuration push from the control plane to the Sidecars on the data plane. The content of the control plane logs shows how efficiently the control plane completes a single push. Control-plane log collection must already be enabled, as described in Prerequisites.

Use the following YAML content to create a virtual service in ASM that applies timeout handling to the httpbin-0 service. For more information about how to create a virtual service, see Manage virtual services.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-0-timeout
  namespace: default
spec:
  hosts:
    - httpbin-0.default.svc.cluster.local
  http:
    - route:
        - destination:
            host: httpbin-0.default.svc.cluster.local
      timeout: 5s

View the logs that the control plane generates. The console navigation path depends on the version of your ASM instance.

ASM instance version 1.17.2.35 or later

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
On the Log Center page, click the Control-Plane Logs tab to view the logs.

ASM instance version earlier than 1.17.2.35

Log on to the ASM console.
In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the target instance.
On the mesh details page, choose Instance Information > Base Information.
On the Basic Information page, click View log to the right of Control-plane log collection.

The following snippet shows sample logs:

2021-12-01T10:20:09.708673Z info  ads CDS: PUSH for node:httpbin-27-7dd8578b46-nkmvg.default resources:227 size:169.3kB
2021-12-01T10:20:09.710469Z info  ads CDS: PUSH for node:httpbin-184-65d97797db-njst5.default resources:227 size:169.3kB
2021-12-01T10:20:09.713567Z info  ads CDS: PUSH for node:httpbin-86-5b64586bbf-jv92w.default resources:227 size:169.3kB
2021-12-01T10:20:09.714514Z info  ads LDS: PUSH for node:httpbin-86-5b64586bbf-jv92w.default resources:16 size:70.7kB
2021-12-01T10:20:09.792732Z info  ads LDS: PUSH for node:httpbin-27-7dd8578b46-nkmvg.default resources:16 size:70.7kB
2021-12-01T10:20:09.792982Z info  ads LDS: PUSH for node:httpbin-184-65d97797db-njst5.default resources:16 size:70.7kB
2021-12-01T10:20:09.796430Z info  ads RDS: PUSH for node:httpbin-86-5b64586bbf-jv92w.default resources:8 size:137.4kB
...
2021-12-01T10:20:13.405850Z info  ads RDS: PUSH for node:httpbin-156-68b85b4f79-2znmp.default resources:8 size:137.4kB
2021-12-01T10:20:13.406154Z info  ads RDS: PUSH for node:httpbin-121-7c4cff97b9-sn5g4.default resources:8 size:137.4kB
2021-12-01T10:20:13.406420Z info  ads CDS: PUSH for node:httpbin-161-7bc74c5fb5-ldgn4.default resources:227 size:169.3kB
2021-12-01T10:20:13.407230Z info  ads LDS: PUSH for node:httpbin-161-7bc74c5fb5-ldgn4.default resources:16 size:70.7kB
2021-12-01T10:20:13.410147Z info  ads RDS: PUSH for node:httpbin-161-7bc74c5fb5-ldgn4.default resources:8 size:137.4kB
2021-12-01T10:20:13.494840Z info  ads RDS: PUSH for node:httpbin-57-69b756f779-db7vv.default resources:8 size:137.4kB

In the test environment where 420 pods are deployed, adding one virtual service causes the control plane to push the change to every Sidecar on the data plane. This generates a large number of push logs, and each push transfers a large amount of data. Applying one virtual service rule in ASM requires the control plane to push for up to about 4 seconds, which indicates low push efficiency on the control plane.

Test the control plane configuration push after a Sidecar resource is applied

Apply a recommended Sidecar resource in the same test cluster, and then repeat the two measurements to compare the results with the baseline values.

Apply the recommended Sidecar resource

Use the feature that automatically recommends Sidecar resources based on access log analysis to recommend and apply a Sidecar resource for each workload in the test cluster. For more information, see Use automatically recommended Sidecar resources based on access log analysis.

Sidecar configuration size after a Sidecar resource is applied

Run the following command to download the Sidecar configuration of the pod that runs the httpbin-0 application to your local machine. Replace httpbin-0-756995d867-jljgp with the name of a pod that currently runs the httpbin-0 application.
```
kubectl exec -it httpbin-0-756995d867-jljgp -c istio-proxy -n default -- curl -s localhost:15000/config_dump > config_dump.json
```
Run the following command to view the size of the Sidecar configuration file.
```
du -sh config_dump.json
```
Expected output:
```
105k    config_dump.json
```

The expected output shows that in the scenario where 420 pods are deployed in the cluster, the feature that automatically recommends Sidecar resources based on access log analysis reduces the Sidecar configuration by a factor of more than 10, to 105 KB. This greatly improves the efficiency of configuration pushes from the control plane to the Sidecars on the data plane.

Control plane push efficiency after a Sidecar resource is applied

Apply a new virtual service rule to the httpbin-0 service in ASM again to trigger one configuration push from the control plane to the Sidecars on the data plane.

Delete the httpbin-0-timeout virtual service that you created in ASM before the optimization. Then, create the virtual service again by using the same YAML content as in Control plane push efficiency before a Sidecar resource is applied, so that the control plane performs one more configuration push for the httpbin-0 service. For more information about how to create a virtual service, see Manage virtual services.
View the logs that the control plane generates. The console navigation path depends on the version of your ASM instance.
ASM instance version 1.17.2.35 or later
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Log Center.
3. On the Log Center page, click the Control-Plane Logs tab to view the logs.
ASM instance version earlier than 1.17.2.35
1. Log on to the ASM console.
2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
3. On the Mesh Management page, click the name of the target instance.
4. On the mesh details page, choose Instance Information > Base Information.
5. On the Basic Information page, click View log to the right of Control-plane log collection.
The following snippet shows sample logs:
```
2021-12-01T12:12:43.498048Z info  ads Push debounce stable[750] 1: 100.03379ms since last change, 100.033692ms since last push, full=true
2021-12-01T12:12:43.504270Z info  ads XDS: Pushing:2021-12-01T12:12:43Z/493 Services:230 ConnectedEndpoints:421  Version:2021-12-01T12:12:43Z/493
2021-12-01T12:12:43.507451Z info  ads CDS: PUSH for node:sleep-0-b68c8c5d9-5kww5.default resources:14 size:7.8kB
2021-12-01T12:12:43.507739Z info  ads LDS: PUSH for node:sleep-0-b68c8c5d9-5kww5.default resources:3 size:15.5kB
2021-12-01T12:12:43.508029Z info  ads RDS: PUSH for node:sleep-0-b68c8c5d9-5kww5.default resources:1 size:6.3kB
```

After the Sidecar resource recommended by ASM is applied, each workload on the data plane no longer receives changes that are related to services it does not depend on. After a virtual service rule is applied to the httpbin-0 service, the control plane pushes the configuration change only to the Sidecar in the pod of the sleep-0 application, because only the sleep-0 application depends on the httpbin-0 service. The configuration push that the virtual service rule triggers lasts only about 0.01 seconds, which is about 400 times faster than before the optimization. The data volume of the change is also reduced by a factor of about 10. This greatly improves the efficiency of configuration pushes from the control plane to the data plane.

Summary of the comparison

The following table compares the values that are measured in this test environment before and after a recommended Sidecar resource is applied.

Metric	Without Sidecar resource optimization	With Sidecar resource optimization
Sidecar configuration size	1.2 MB	105 KB
Pushes to services without dependencies	Yes	No
Control plane configuration push duration	About 4 seconds	About 0.01 seconds