Inject sidecar containers into pods on virtual nodes - Container Service for Kubernetes

You can use the OpenKruise SidecarSet feature to automatically inject sidecar containers into pods that are scheduled to virtual nodes. This decouples application containers from auxiliary function containers. This topic describes how to create a SidecarSet to define sidecar containers and automatically inject them into pods on virtual nodes.

Terms

Sidecar: A design pattern that separates auxiliary functions from the application itself as separate processes or containers. This design pattern allows you to extend the functions of an application without business intrusions. You can use third-party components without the need to modify the configurations of your applications.
Sidecar container: An additional container added to a pod to extend and enhance the main container without modifying it.
SidecarSet: One of the core features of OpenKruise, an open-source cloud-native application automation engine from Alibaba Cloud. SidecarSet automatically injects sidecar containers into pods that meet specific conditions, decoupling the definition and lifecycle of sidecar containers, such as monitoring and logging agents, from application containers.

Preparations

The cluster version is 1.22 or later, and the cluster type is ACK Serverless Pro cluster.
The ack-virtual-node component is installed, and the version is v2.10.0 or later. For more information, see ACK Virtual Node.
The ack-kruise component is installed, and the version is v1.3.0 or later. For more information, see OpenKruise.
Important
In virtual node scenarios, all SidecarSet features of OpenKruise v1.3.0 and earlier versions are supported, but new SidecarSet features in versions later than 1.3.0 are not supported.
You have customized the parameters of the Kube API server component and set SidecarSetServerlessPod=true in featureGates to enable the feature gate for the SidecarSet feature. For more information, see Customize the parameters of a control plane component.

Introduction

SidecarSet

You can use the same method as the default SidecarSet to match all pods scheduled to virtual nodes by specifying the serverless.alibabacloud.com/virtual-node: "true" label. This label is added after the pod is determined to be scheduled to a virtual node. For information about how to use the default SidecarSet, see SidecarSet.

Another commonly used feature of SidecarSet is referencing ConfigMaps and Secrets across namespaces. DaemonSet core containers often depend on ConfigMaps, such as configuration parameters. When you inject DaemonSet core containers into application pods, the application pods and ConfigMaps are usually in different namespaces. Because virtual node scenarios do not support DaemonSet, you need to use sidecar containers instead. In the volume of a sidecar container, you can reference ConfigMaps and Secrets across namespaces by using the Namespace/Name method.

Note

To reference ConfigMaps and Secrets across namespaces, you must first complete authorization. For more information, see SidecarSetResourceBinding.

View a YAML example of a SidecarSet that mounts a ConfigMap

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:      
      serverless.alibabacloud.com/virtual-node: "true" # dd the label to select pods that are scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent    
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000", # In this example, only the filebeat configuration file is printed.
    ]    
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml    
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config # Use a namespace or name to reference a ConfigMap in another namespace.

SidecarSetResourceBinding

For security reasons, when you reference ConfigMaps and Secrets from other namespaces in the volume of a sidecar container, you must grant the SidecarSetResourceBinding permission.

Note

A SidecarSetResourceBinding grants the read-only permissions to perform the following operations on ConfigMaps and Secrets: Get, List, and Watch.

View a YAML example of a SidecarSetResourceBinding

# Grant permissions to filebeat-sidecarset so that pods matched by the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # The SidecarSetResourceBinding grants permissions only on resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap # Only the permissions to perform the following read-only operations are granted: Get, List, and Watch.
      name: filebeat-config

Container startup and exit order

Sidecar containers often need to start before application containers and exit after application containers. You can achieve this by configuring the startup and exit priorities of containers.

Automatically terminate sidecar containers

For job pods, sidecar containers may prevent the job from exiting after the application containers complete. You can solve this problem by forcibly terminating the sidecar container and ignoring the container exit code.

Upgrade sidecar containers

You may need to update sidecar containers after you inject them into pods. You can use the existing Sidecar hot upgrade feature of OpenKruise. This method allows you to seamlessly update sidecar containers without affecting pod availability and is fully compatible with the current virtual node approach.

Collect standard output logs

You can collect the standard output logs of application containers by mounting the standard output logs of virtual node pods (in the form of stdlog volumes) to a specified directory in the sidecar container. For more information, see Mount stdlog to a pod.

View a YAML example of a SidecarSet that mounts stdlog

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # dd the label to select pods that are scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  # In this example, only the log of the sidecar container is printed.
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /var/log/std/filebeat/0.log && sleep 36000",
    ]    
    volumeMounts:    
    - name: stdlog # Mount the pod stdout to the /var/log/std directory in the sidecar container to collect logs.
      mountPath: /var/log/std
      readOnly: true
  volumes:  
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Example

The following example demonstrates how to inject a sidecar container (filebeat container) into an application pod (echo-server). You can refer to this example to design your actual business applications.

Deploy a SidecarSet to automatically inject sidecar containers into application pods on virtual nodes.

Deploy a ConfigMap to be mounted to the sidecar container.

kubectl apply -f filebeat-config.yaml

The following is an example of the filebeat-config.yaml file. This example only mounts the configuration file (filebeat.yml) to the sidecar container and prints it. The related variables do not take effect and do not need to be replaced.

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/std/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME} # This does not take effect and does not need to be modified. Please use it directly.
            matchers:
            - logs_path:
                logs_path: "/var/log/std/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}  # This does not take effect and does not need to be modified. Please use it directly.
    cloud.auth: ${ELASTIC_CLOUD_AUTH}  # This does not take effect and does not need to be modified. Please use it directly.

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}  # This does not take effect and does not need to be modified. Please use it directly.
      password: ${ELASTICSEARCH_PASSWORD}  # This does not take effect and does not need to be modified. Please use it directly.

Deploy a SidecarSet to describe the sidecar container.

kubectl apply -f sidecarset.yaml

The following is an example of the sidecarset.yaml file. In this example, the filebeat container as a sidecar container will print the contents of the configuration file and has mounted stdlog, which can collect the standard output logs of the application pod.

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Add the label to select pods that are scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  # This example does not actually run filebeat, but replaces it with bosybox cat.
  #- name: filebeat
  #  image: docker.elastic.co/beats/filebeat:8.6.1
  #  args: [
  #    "-c", "/etc/filebeat.yml",
  #    "-e",
  #  ]
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000",
    ]
    env:
    - name: ECI_SIDECAR_CONTAINER         # Indicates that the sidecar container exits after the application container.
      value: "true"
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml
    - name: stdlog                        # Mount the pod standard output logs for the sidecar container to read.
      mountPath: /var/log/std
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config  # Use the Namespace/Name format to reference a ConfigMap in another namespace.
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Grant permissions for the sidecar container to access the ConfigMap.

If the application pod and the ConfigMap are in different namespaces, the sidecar container injected into the application pod needs explicit authorization through SidecarSetResourceBinding to access the ConfigMap.

kubectl apply -f sidecarset-resourcebinding.yaml

The following is an example of the sidecarset-resourcebinding.yaml file. In this example, the application pod is planned to be deployed in the default namespace, while the ConfigMap is in the kube-system namespace, so authorization is required.

# Grant permissions to filebeat-sidecarset so that pods matched by the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # The SidecarSetResourceBinding grants permissions only on resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap
      name: filebeat-config

Deploy an application pod scheduled to a virtual node.

kubectl apply -f echo-server.yaml

The following is an example of the echo-server.yaml file. This Deployment contains one replica, and the pod contains one container. After adding the alibabacloud.com/eci: "true" label, the pod will be scheduled to a virtual node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server        
        alibabacloud.com/eci: "true"
    spec:
      containers:
        - name: echo-server
          image: hashicorp/http-echo
          imagePullPolicy: IfNotPresent
          args:
            - -listen=:8080
            - -text="hello world"

Confirm that the sidecar container has been automatically injected into the application pod and verify the mounting results.

View the business pod.
```
kubectl get pod
```
The following expected output shows that the application pod contains 2 containers, which indicates that the sidecar container has been successfully injected.
```
NAME                          READY   STATUS    RESTARTS   AGE
echo-server-f8bdc5844-r44nj   2/2     Running   0          14m
```

Verify that the sidecar container has mounted the standard output logs of the application pod.

kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /var/log/std/echo-server/0.log

The following expected output shows the standard output logs of the application pod.

2025-04-29T11:26:06.783205694+08:00 stderr F 2025/04/29 03:26:06 Server is listening on :8080

Verify that the sidecar container has mounted the configuration file from across namespaces.

kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /etc/filebeat.yml

The following expected output indicates that the mounting is normal.

View the output example

filebeat.inputs:
- type: log
  paths:
    - /var/log/std/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME} # This does not take effect and does not need to be modified. Please use it directly.
        matchers:
        - logs_path:
            logs_path: "/var/log/std/"

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      node: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}  # This does not take effect and does not need to be modified. Please use it directly.
cloud.auth: ${ELASTIC_CLOUD_AUTH}  # This does not take effect and does not need to be modified. Please use it directly.

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}  # This does not take effect and does not need to be modified. Please use it directly.
  password: ${ELASTICSEARCH_PASSWORD}  # This does not take effect and does not need to be modified. Please use it directly.