Use a SidecarSet to inject a sidecar container into a virtual node pod - Container Service for Kubernetes

The OpenKruise SidecarSet feature automatically injects sidecar containers into pods that are scheduled to virtual nodes. This decouples application containers from containers that provide auxiliary functions. This topic describes how to use a SidecarSet to define and automatically inject a sidecar container into pods on virtual nodes.

Terms

Sidecar: A design pattern that separates auxiliary functions from an application into a separate process or container. This pattern lets you add multiple features to an application non-intrusively. It avoids adding extra configuration code to the application to accommodate third-party components.
Sidecar container: An additional container added to a pod to extend and enhance the application container without changing it.
SidecarSet is one of the core features of OpenKruise, an open-source cloud-native application automation engine from Alibaba Cloud. A SidecarSet can automatically inject sidecar containers, such as monitoring and logging agents, into eligible pods to decouple their definition and lifecycle from application containers.

Scenarios

ACK virtual nodes support ECI and ACS computing power. When you use ACS computing power, only CPU pods of the general-purpose and compute-optimized instance types are supported.

Prerequisites

Your cluster is an ACK managed cluster (Pro Edition) or an ACK dedicated cluster, and the cluster version is 1.22 or later.
The ack-virtual-node component of version v2.10.0 or later is installed. For more information, see ACK Virtual Node.
The ack-kruise component of version v1.3.0 or later is installed. For more information, see OpenKruise.
Important
In virtual node scenarios, all SidecarSet features of OpenKruise v1.3.0 and earlier are supported. However, new SidecarSet features added in versions of OpenKruise later than v1.3.0 are not supported.
You have customized the parameter settings for the Kube API Server component. In featureGates, set SidecarSetServerlessPod=true to enable the feature gate for the SidecarSet feature. For more information, see Customize control plane component parameters.

Features

SidecarSet

You can match all pods that are scheduled to virtual nodes using the label serverless.alibabacloud.com/virtual-node: "true". This label is added after the pod is scheduled to a virtual node. For more information about how to use the default SidecarSet, see SidecarSet.

Another common feature of SidecarSet is the ability to reference ConfigMaps and Secrets across namespaces. In scenarios that use DaemonSets, the core container often depends on a ConfigMap for parameter settings. Because virtual node scenarios do not support DaemonSets, you must use a sidecar container as a replacement. When this sidecar container is injected into an application pod, the pod and the ConfigMap may be in different namespaces. In the volume of the sidecar container, you can reference ConfigMaps and Secrets across namespaces using the Namespace/Name format.

Note

Authorization is required to access ConfigMaps and Secrets across namespaces. For more information, see SidecarSetResourceBinding.

Expand to view a YAML example of a SidecarSet that mounts a ConfigMap

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:      
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent    
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000", # This example only prints the filebeat configuration content.
    ]    
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml    
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config # Use the namespace/name format to reference a ConfigMap in another namespace.

SidecarSetResourceBinding

For security reasons, you must use a SidecarSetResourceBinding to explicitly grant authorization when a sidecar container volume references a ConfigMap or Secret from another namespace.

Note

This authorization grants only read-only permissions (Get, List, and Watch) for the ConfigMap and Secret.

Expand to view a SidecarSetResourceBinding YAML example

# Authorize filebeat-sidecarset so that pods matched by the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # This SidecarSetResourceBinding can only authorize resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap # Grants only read-only permissions (Get, List, Watch).
      name: filebeat-config

Container startup and shutdown order

Sidecar containers often need to start before the application container and shut down after the application container. You can achieve this by setting the container startup and shutdown order.

Automatically terminate sidecar containers

For Job pods, a sidecar container can prevent the Job from exiting after the application container finishes. To resolve this issue, you can forcefully terminate the sidecar container and ignore its exit code.

Upgrade sidecar containers

When you use the sidecar pattern, you may have operations and maintenance (O&M) requirements, such as upgrading sidecar containers. You can use the Sidecar hot upgrade feature of OpenKruise. This feature lets you seamlessly upgrade sidecar containers without affecting pod availability and is fully compatible with the virtual node implementation.

Collect standard output logs

You can collect the standard output logs of an application container by mounting the stdlog volume of the virtual node pod to a specified directory in the sidecar container. For more information, see Mount a stdlog volume to mount the standard output logs of a container.

Expand to view a SidecarSet YAML example for mounting stdlog

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  # This example only prints the sidecar container log content.
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /var/log/std/filebeat/0.log && sleep 36000",
    ]    
    volumeMounts:    
    - name: stdlog # Mount the pod's standard output log volume /var/log/std so the sidecar container can read it.
      mountPath: /var/log/std
      readOnly: true
  volumes:  
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Example

The following example shows how to inject a sidecar container (filebeat) into an application pod (echo-server). You can use this example as a reference for your own applications.

Deploy a SidecarSet to automatically inject a sidecar container into application pods on virtual nodes.

Deploy a ConfigMap to mount to the sidecar container.

kubectl apply -f filebeat-config.yaml

The following is an example of the `filebeat-config.yaml` file. This example mounts the configuration file (`filebeat.yml`) to the sidecar container and prints its content. The variables in the file do not take effect and do not need to be replaced.

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/std/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME} # This value does not take effect. No modification is needed.
            matchers:
            - logs_path:
                logs_path: "/var/log/std/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}  # This value does not take effect. No modification is needed.
    cloud.auth: ${ELASTIC_CLOUD_AUTH}  # This value does not take effect. No modification is needed.

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}  # This value does not take effect. No modification is needed.
      password: ${ELASTICSEARCH_PASSWORD}  # This value does not take effect. No modification is needed.

Deploy a SidecarSet to define the sidecar container.

kubectl apply -f sidecarset.yaml

The following is an example of the `sidecarset.yaml` file. In this example, the filebeat sidecar container prints the content of the configuration file. It also mounts the stdlog volume to collect the standard output logs of the application pod.

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  # This example does not actually run filebeat. It is replaced with busybox cat.
  #- name: filebeat
  #  image: docker.elastic.co/beats/filebeat:8.6.1
  #  args: [
  #    "-c", "/etc/filebeat.yml",
  #    "-e",
  #  ]
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000",
    ]
    env:
    - name: ECI_SIDECAR_CONTAINER         # Indicates that the sidecar container shuts down after the application container.
      value: "true"
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml
    - name: stdlog                        # Mount the pod's standard output log so the sidecar container can read it.
      mountPath: /var/log/std
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config  # Use the Namespace/Name format to reference a ConfigMap in another namespace.
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Grant the sidecar container access to the ConfigMap.

If the application pod and the ConfigMap are in different namespaces, you must use a SidecarSetResourceBinding to grant the injected sidecar container access to the ConfigMap.

kubectl apply -f sidecarset-resourcebinding.yaml

The following example `sidecarset-resourcebinding.yaml` file deploys an application pod to the default namespace. Because the ConfigMap is in the kube-system namespace, authorization is required.

# Authorize filebeat-sidecarset so that pods matched by the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # This SidecarSetResourceBinding only authorizes resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap
      name: filebeat-config

Deploy an application pod to a virtual node.

kubectl apply -f echo-server.yaml

The following is an example of the echo-server.yaml file. This Deployment contains one replica, and the pod contains one container. The pod is scheduled to a virtual node because it has the alibabacloud.com/eci: "true" label.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server        
        alibabacloud.com/eci: "true"
    spec:
      containers:
        - name: echo-server
          image: hashicorp/http-echo
          imagePullPolicy: IfNotPresent
          args:
            - -listen=:8080
            - -text="hello world"

Confirm that the sidecar container is automatically injected into the application pod and verify that the volumes are mounted correctly.

Check the application pod.
```
kubectl get pod
```
The expected output is as follows. The application pod contains two containers, which indicates that the sidecar container was injected successfully.
```
NAME                          READY   STATUS    RESTARTS   AGE
echo-server-f8bdc5844-r44nj   2/2     Running   0          14m
```

Verify that the sidecar container mounted the standard output logs of the application pod.

kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /var/log/std/echo-server/0.log

The expected output is as follows. It shows the standard output logs of the application pod.

2025-04-29T11:26:06.783205694+08:00 stderr F 2025/04/29 03:26:06 Server is listening on :8080

Verify that the sidecar container mounted the configuration file from another namespace.

kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /etc/filebeat.yml

The expected output is as follows. This indicates that the mount was successful.

Expand to view the sample output

filebeat.inputs:
- type: log
  paths:
    - /var/log/std/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME} # This value does not take effect. No modification is needed.
        matchers:
        - logs_path:
            logs_path: "/var/log/std/"

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      node: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}  # This value does not take effect. No modification is needed.
cloud.auth: ${ELASTIC_CLOUD_AUTH}  # This value does not take effect. No modification is needed.

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}  # This value does not take effect. No modification is needed.
  password: ${ELASTICSEARCH_PASSWORD}  # This value does not take effect. No modification is needed.