Container Service for Kubernetes:Inject sidecar containers into virtual node pods

Match virtual node pods

Use the label serverless.alibabacloud.com/virtual-node: "true" as the SidecarSet selector to match all pods on virtual nodes:

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.

For full SidecarSet selector options, see SidecarSet.

Reference ConfigMaps and Secrets across namespaces

Because virtual nodes do not support DaemonSets, sidecar containers replace DaemonSet core containers. These containers often need access to ConfigMaps — such as agent configuration files — that live in a different namespace from the application pod.

Reference a ConfigMap or Secret from another namespace using the Namespace/Name format in the volume definition:

volumes:
- name: config
  configMap:
    name: kube-system/filebeat-config # Use the Namespace/Name format to reference a ConfigMap in another namespace.

Cross-namespace access requires a SidecarSetResourceBinding authorization. See Authorize cross-namespace access below.

Authorize cross-namespace access

Create a SidecarSetResourceBinding to grant the SidecarSet read-only access to ConfigMaps or Secrets in another namespace. Create the resource in the namespace that owns the ConfigMap or Secret:

# Authorizes filebeat-sidecarset. Pods matching the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # This SidecarSetResourceBinding only authorizes resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap # Only grants read-only permission (Get, List, Watch).
      name: filebeat-config

Control container startup and shutdown order

Sidecar containers often need to start before application containers and exit after them. Configure startup and shutdown ordering for:

• ECI: Configure container startup and shutdown order

• ACS: Configure the startup and exit priorities of sidecar containers

Terminate sidecar containers for Job pods

For Job-type pods, a sidecar container that keeps running after the application container exits blocks the Job from completing. Set the ECI_SIDECAR_CONTAINER environment variable to "true" to make the sidecar container exit automatically when the application container finishes:

containers:
- name: filebeat
  image: busybox
  env:
  - name: ECI_SIDECAR_CONTAINER  # Causes the sidecar container to exit after the application container exits.
    value: "true"

For complete configuration details, see Forcibly terminate the sidecar container and ignore the container exit code.

Upgrade sidecar containers without downtime

OpenKruise supports hot upgrades for sidecar containers, enabling seamless upgrades without affecting pod availability. This mechanism is fully compatible with virtual nodes. See Sidecar hot upgrade in the OpenKruise documentation for configuration details.

Collect standard output logs

Mount the pod's standard output logs as a stdlog volume into the sidecar container to collect logs from the application container:

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /var/log/std/filebeat/0.log && sleep 36000",
    ]
    volumeMounts:
    - name: stdlog # Mounts the pod's standard output log volume for the sidecar container to read.
      mountPath: /var/log/std
      readOnly: true
  volumes:
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Logs are available at /var/log/std/<container-name>/0.log inside the sidecar container. For more information, see Mount container logs by using stdlog.

Step 1: Deploy the ConfigMap

Create filebeat-config.yaml with the following content, then apply it. This example mounts the configuration file into the sidecar container for demonstration — the variables in this file are placeholders and are not active.

kubectl apply -f filebeat-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      paths:
        - /var/log/std/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME} # Not effective. Do not modify. Use directly.
            matchers:
            - logs_path:
                logs_path: "/var/log/std/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}  # Not effective. Do not modify. Use directly.
    cloud.auth: ${ELASTIC_CLOUD_AUTH}  # Not effective. Do not modify. Use directly.

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}  # Not effective. Do not modify. Use directly.
      password: ${ELASTICSEARCH_PASSWORD}  # Not effective. Do not modify. Use directly.

Step 2: Deploy the SidecarSet

Create sidecarset.yaml with the following content, then apply it. The filebeat container uses busybox for demonstration instead of an actual filebeat binary, and prints the mounted configuration file.

kubectl apply -f sidecarset.yaml

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Matches all pods scheduled to virtual nodes.
  updateStrategy:
    type: NotUpdate
  containers:
  # This example does not actually run filebeat; it uses busybox cat instead.
  #- name: filebeat
  #  image: docker.elastic.co/beats/filebeat:8.6.1
  #  args: [
  #    "-c", "/etc/filebeat.yml",
  #    "-e",
  #  ]
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000",
    ]
    env:
    - name: ECI_SIDECAR_CONTAINER         # Causes the sidecar container to exit after the application container exits.
      value: "true"
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml
    - name: stdlog                        # Mounts the pod's standard output logs for the sidecar container to read.
      mountPath: /var/log/std
      readOnly: true
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config  # Uses the Namespace/Name format to reference a ConfigMap in another namespace.
  - name: stdlog
    csi:
      driver: stdlogplugin.csi.alibabacloud.com

Step 3: Authorize cross-namespace ConfigMap access

The application pod runs in the default namespace while the ConfigMap is in kube-system. Create a SidecarSetResourceBinding to authorize access.

Create sidecarset-resourcebinding.yaml with the following content, then apply it:

kubectl apply -f sidecarset-resourcebinding.yaml

# Authorizes filebeat-sidecarset. Pods matching the SidecarSet can access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # This SidecarSetResourceBinding only authorizes resources in the kube-system namespace.
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap
      name: filebeat-config

Step 4: Deploy the application pod

Create echo-server.yaml with the following content, then apply it. The alibabacloud.com/eci: "true" label schedules the pod to a virtual node.

kubectl apply -f echo-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  labels:
    app: echo-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
        alibabacloud.com/eci: "true"
    spec:
      containers:
        - name: echo-server
          image: hashicorp/http-echo
          imagePullPolicy: IfNotPresent
          args:
            - -listen=:8080
            - -text="hello world"

Step 5: Verify the injection

1. Check that the pod has two containers, confirming the sidecar was injected:

   kubectl get pod

   Expected output:

   NAME                          READY   STATUS    RESTARTS   AGE
   echo-server-f8bdc5844-r44nj   2/2     Running   0          14m

2. Verify the sidecar can read the application pod's standard output logs:

   kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /var/log/std/echo-server/0.log

   Expected output:

   2025-04-29T11:26:06.783205694+08:00 stderr F 2025/04/29 03:26:06 Server is listening on :8080

3. Verify the sidecar can read the cross-namespace ConfigMap:

   kubectl exec echo-server-f8bdc5844-r44nj -c filebeat -- cat /etc/filebeat.yml

   Expand to view example output

   filebeat.inputs:
   - type: log
     paths:
       - /var/log/std/*.log
     processors:
       - add_kubernetes_metadata:
           host: ${NODE_NAME} # Not effective. Do not modify. Use directly.
           matchers:
           - logs_path:
               logs_path: "/var/log/std/"
   
   # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
   #filebeat.autodiscover:
   #  providers:
   #    - type: kubernetes
   #      node: ${NODE_NAME}
   #      hints.enabled: true
   #      hints.default_config:
   #        type: container
   #        paths:
   #          - /var/log/containers/*${data.kubernetes.container.id}.log
   
   processors:
     - add_cloud_metadata:
     - add_host_metadata:
   
   cloud.id: ${ELASTIC_CLOUD_ID}  # Not effective. Do not modify. Use directly.
   cloud.auth: ${ELASTIC_CLOUD_AUTH}  # Not effective. Do not modify. Use directly.
   
   output.elasticsearch:
     hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
     username: ${ELASTICSEARCH_USERNAME}  # Not effective. Do not modify. Use directly.
     password: ${ELASTICSEARCH_PASSWORD}  # Not effective. Do not modify. Use directly.