If you have deployed a large number of services in a single namespace, you can use the sidecar recommendation feature to reduce the size of sidecar configurations to the maximum extent. In this topic, a large-scale cluster that has 420 pods deployed is used to test and analyze the effects of sidecar recommendation on configuration push optimization.

Prerequisites

Deploy applications in the test cluster

In this test, multiple sleep and httpbin applications are used to simulate a large number of services that have few call dependencies in a cluster.

  • After an httpbin application is started, the httpbin application exposes an HTTP service on port 8000. These HTTP services simulate a large number of services that are called in a cluster.
  • Each sleep application contains a curl container. You can modify the command field in the configuration file of a sleep application. This way, you can configure the sleep application to call the services provided by multiple httpbin containers before the sleep application enters the sleeping state. The services provided by these sleep applications simulate the services that depend on other services in a cluster.
  1. Deploy httpbin applications in the cluster.
    1. Create a YAML file named httpbin-{i} based on the following template:
      Note Replace {i} in httpbin-{i} with a specific number. This way, you can create multiple httpbin applications that have different IDs. You can use this template to generate as many httpbin applications as required. The maximum number of applications that you can generate depends on the size of the cluster. In this test, 200 httpbin applications are generated by using this template, and 400 pods are deployed in the cluster to run the httpbin applications. 
      apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: httpbin-{i}
    service: httpbin-{i}
  name: httpbin-{i}
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin-0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: httpbin-{i}
  name: httpbin-{i}
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin-{i}
      version: v1
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: httpbin-{i}
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
      serviceAccountName: httpbin
    2. Run the following command to create the httpbin-{i} application: 
      kubectl apply -f httpbin-{i}.yaml

      Expected output:

      deployment.apps/httpbin-{i} created
  2. Deploy sleep applications in the cluster.
    1. Create a YAML file named sleep-{i} based on the following template:
      Note Replace {i} in sleep-{i} with a specific number. This way, you can create multiple sleep applications that have different IDs. In this template, the curl httpbin-{i*10}:8000 command parameters that are added to the args field simulate call dependencies on different httpbin applications. The IDs of the httpbin applications that you specify in the command parameters must not exceed the maximum ID of the httpbin applications that are deployed. Otherwise, valid calls cannot be made. In this test, each sleep application depends on 10 httpbin applications. Therefore, 20 sleep applications are created, and 20 pods are deployed to run the sleep applications. 
      apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: sleep-{i}
    service: sleep-{i}
  name: sleep-{i}
spec:
  ports:
  - name: http
    port: 80
    targetPort: 0
  selector:
    app: sleep-{i}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: sleep-{i}
  name: sleep-{i}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep-{i}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: sleep-{i}
    spec:
      containers:
      - args:
        - curl httpbin-{i*10}:8000; curl httpbin-{i*10+1}:8000; curl httpbin-{i*10+2}:8000; curl httpbin-{i*10+3}:8000;
          curl httpbin-{i*10+4}:8000; curl httpbin-{i*10+5}:8000; curl httpbin-{i*10+6}:8000; curl httpbin-{i*10+7}:8000;
          curl httpbin-{i*10+8}:8000; curl httpbin-{i*10+9}:8000; sleep 3650d
        command:
        - /bin/sh
        - -c
        image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: sleep
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      serviceAccountName: sleep
      terminationGracePeriodSeconds: 0
      volumes:
      - name: secret-volume
        secret:
          optional: true
          secretName: sleep-secret
    2. Run the following command to create the sleep-{i} application: 
      kubectl apply -f sleep-{i}.yaml

      Expected output:

      deployment.apps/sleep-{i} created

Test the configuration push efficiency of the control plane before recommended sidecars are applied

Test 1: Test the configuration size of each sidecar before recommended sidecars are applied
  1. Run the following command to obtain the names of the pods that run the httpbin-0 application: 
    kubectl get pod -n ns-in-mesh | grep httpbin-0

    Expected output:

    httpbin-0-756995d867-j****   2/2     Running   0          9m15s
httpbin-0-756995d867-w****   2/2     Running   0          9m15s
  2. Run the following command to dump the configurations of the sidecar that runs in one of the preceding pods to a local file: 
    kubectl exec -it httpbin-0-756995d867-j**** -c istio-proxy -n ns-in-mesh -- curl -s localhost:15000/config_dump > config_dump.json
  3. Run the following command to view the size of the configuration file: 
    du -sh config_dump.json

    Expected output:

    1.2M    config_dump.json

    The preceding output shows that the configuration size of a sidecar is about 1.2 MB when 420 pods are deployed in the cluster. If a sidecar is created for each pod in the cluster, the configurations of these sidecars reach a considerable size and increase the load of the control plane.

Test 2: Test the configuration push efficiency of the control plane before recommended sidecars are applied

Create a virtual service for the httpbin-0 service in the ASM console. This triggers the control plane to push configurations to the sidecars on the data plane. You can view the logs of the control plane to determine the efficiency of the control plane in one configuration push. For more information about how to enable log collection on the control plane, see Enable collection of control plane logs and control plane alerting.

  1. Use a YAML file that contains the following content to create a virtual service in the ASM instance to handle the timeout of the httpbin-0 service. For more information, see Manage virtual services. 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-0-timeout
  namespace: default
spec:
  hosts:
    - httpbin-0.default.svc.cluster.local
  http:
    - route:
        - destination:
            host: httpbin-0.default.svc.cluster.local
      timeout: 5s
  2. View the newly generated logs of the control plane.
    1. Log on to the ASM console.
    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
    3. On the Mesh Management page, click the name of the ASM instance.
    4. On the details page of the ASM instance, choose ASM Instance > Basic Information in the left-side navigation pane.
    5. On the Basic Information page, click View log on the right of Control-plane log collection.

      Expected output:

      2021-12-01T10:20:09.708673Z info  ads CDS: PUSH for node:httpbin-27-7dd8578b46-n****.default resources:227 size:169.3kB
2021-12-01T10:20:09.710469Z info  ads CDS: PUSH for node:httpbin-184-65d97797db-n****.default resources:227 size:169.3kB
2021-12-01T10:20:09.713567Z info  ads CDS: PUSH for node:httpbin-86-5b64586bbf-j****.default resources:227 size:169.3kB
2021-12-01T10:20:09.714514Z info  ads LDS: PUSH for node:httpbin-86-5b64586bbf-j****.default resources:16 size:70.7kB
2021-12-01T10:20:09.792732Z info  ads LDS: PUSH for node:httpbin-27-7dd8578b46-n****.default resources:16 size:70.7kB
2021-12-01T10:20:09.792982Z info  ads LDS: PUSH for node:httpbin-184-65d97797db-n****.default resources:16 size:70.7kB
2021-12-01T10:20:09.796430Z info  ads RDS: PUSH for node:httpbin-86-5b64586bbf-j****.default resources:8 size:137.4kB
...
2021-12-01T10:20:13.405850Z info  ads RDS: PUSH for node:httpbin-156-68b85b4f79-2****.default resources:8 size:137.4kB
2021-12-01T10:20:13.406154Z info  ads RDS: PUSH for node:httpbin-121-7c4cff97b9-s****.default resources:8 size:137.4kB
2021-12-01T10:20:13.406420Z info  ads CDS: PUSH for node:httpbin-161-7bc74c5fb5-l****.default resources:227 size:169.3kB
2021-12-01T10:20:13.407230Z info  ads LDS: PUSH for node:httpbin-161-7bc74c5fb5-l****.default resources:16 size:70.7kB
2021-12-01T10:20:13.410147Z info  ads RDS: PUSH for node:httpbin-161-7bc74c5fb5-l****.default resources:8 size:137.4kB
2021-12-01T10:20:13.494840Z info  ads RDS: PUSH for node:httpbin-57-69b756f779-d****.default resources:8 size:137.4kB

      The preceding output shows that the control plane pushes configuration changes to all sidecars on the data plane after a virtual service is added. A large number of push logs are generated in the cluster that has 420 pods deployed. The amount of data that is pushed to each sidecar is large. To apply a virtual service in the ASM instance, the control plane takes about 4 seconds to push the configurations to the data plane. The configuration push efficiency of the control plane is low.

Test the configuration push efficiency of the control plane after recommended sidecars are applied

Use the sidecar recommendation feature to recommend a sidecar for each workload in the test cluster based on access log analysis. This helps improve the configuration push efficiency of the control plane. For more information, see Use the sidecars that are automatically recommended based on access log analysis.

Test 1: Test the configuration size of each sidecar after recommended sidecars are applied
  1. Run the following command to dump the configurations of the sidecar in a pod that runs the httpbin-0 application to a local file: 
    kubectl exec -it httpbin-0-756995d867-j**** -c istio-proxy -n ns-in-mesh -- curl -s localhost:15000/config_dump > config_dump.json
  2. Run the following command to view the size of the configuration file: 
    du -sh config_dump.json

    Expected output:

    105k    config_dump.json

    The preceding output shows that the configuration size of a sidecar is reduced by more than 10 times after the recommended sidecars are applied in the cluster that has 420 pods deployed.

Test 2: Test the configuration push efficiency of the control plane after recommended sidecars are applied

Create a virtual service for the httpbin-0 service in the ASM console again. This triggers the control plane to push configurations to the sidecars on the data plane.

  1. Delete the virtual service that you created earlier. Use a YAML file that contains the following content to create a virtual service in the ASM instance to handle the timeout of the httpbin-0 service. For more information, see Manage virtual services. 
    apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin-0-timeout
  namespace: default
spec:
  hosts:
    - httpbin-0.default.svc.cluster.local
  http:
    - route:
        - destination:
            host: httpbin-0.default.svc.cluster.local
      timeout: 5s
  2. View the newly generated logs of the control plane.
    1. Log on to the ASM console.
    2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
    3. On the Mesh Management page, click the name of the ASM instance.
    4. On the details page of the ASM instance, choose ASM Instance > Basic Information in the left-side navigation pane.
    5. On the Basic Information page, click View log on the right of Control-plane log collection.

      Expected output:

      2021-12-01T12:12:43.498048Z info  ads Push debounce stable[750] 1: 100.03379ms since last change, 100.033692ms since last push, full=true
2021-12-01T12:12:43.504270Z info  ads XDS: Pushing:2021-12-01T12:12:43Z/493 Services:230 ConnectedEndpoints:421  Version:2021-12-01T12:12:43Z/493
2021-12-01T12:12:43.507451Z info  ads CDS: PUSH for node:sleep-0-b68c8c5d9-5****.default resources:14 size:7.8kB
2021-12-01T12:12:43.507739Z info  ads LDS: PUSH for node:sleep-0-b68c8c5d9-5****.default resources:3 size:15.5kB
2021-12-01T12:12:43.508029Z info  ads RDS: PUSH for node:sleep-0-b68c8c5d9-5****A.default resources:1 size:6.3kB

      The preceding output shows that each workload on the data plane will not receive changes related to services with which the workload has no dependencies after recommended sidecars are applied. After you create the virtual service for the httpbin-0 service, the control plane pushes configuration changes only to the sidecar in the pod that runs the sleep-0 application. This is because only the sleep-0 application has dependencies with the httpbin-0 service. The configuration push takes about 0.01 seconds to complete. Compared with the time taken before optimization, the configuration push efficiency is improved by about 400 times. In addition, the amount of pushed data is reduced by about 10 times. This indicates that the sidecar recommendation feature can significantly improve the efficiency of the control plane in pushing configurations to the data plane.

Test summary

This test uses multiple sleep and httpbin applications to simulate a large number of services that have few call dependencies in a cluster. A total of 200 httpbin applications, 400 pods that run the httpbin applications, 20 sleep applications, and 20 pods that run the sleep applications are deployed. The following table describes the comparison before and after recommended sidecars are applied.

Item Before recommended sidecars are applied After recommended sidecars are applied
Sidecar configuration size 1.2 MB 105 KB
Whether the information about unrelated services is pushed Yes No
Time taken by the control plane to push configurations About 4 seconds About 0.01 seconds