Container Service for Kubernetes:Inject sidecar containers into pods on virtual nodes

Terms

• Sidecar: a design pattern that implements the functions of an application in separate processes. This design pattern allows you to extend the functions of an application without business intrusions. You can use third-party components without the need to modify the configurations of your applications.

• Sidecar container: an auxiliary container attached to a pod. Sidecar containers are used to enhance or extend the functionality of the primary containers of the pod without the need to modify the configurations of the primary containers.

• SidecarSet: a key controller provided by OpenKruise, an open source cloud-native application automation engine developed by Alibaba Cloud. This controller can automatically inject sidecar containers into each pod that matches the specified labels during pod creation. The sidecar containers can function as monitoring agents or logging agents. This way, the SidecarSet controller decouples the lifecycle management of sidecar containers from that of business containers.

Limits

The SidecarSet features of OpenKruise version 1.3 or earlier are supported by ack-virtual-node. The new SidecarSet features introduced in versions later than 1.3 are not supported.

Prerequisites

• An ACK Pro cluster, ACK Serverless Pro cluster, or ACK dedicated cluster that runs Kubernetes 1.22 or later is created. For more information, see Create an ACK managed cluster, Create an ACK Serverless cluster, or Create a ACK dedicated cluster.

• ack-kruise 1.3.0 or later is installed. For more information, see ack-kruise.

• ack-virtual-node 2.10.0 or later is installed. For more information, see ACK Virtual Node.

• The SidecarSetServerlessPod=true feature gate is enabled. Specify SidecarSetServerlessPod=true in the featureGates parameter of kube-apiserver. For more information, see Customize the parameters of a control plane component.

Feature description

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:      
      serverless.alibabacloud.com/virtual-node: "true" # Add the label to select pods that are scheduled to virtual nodes. 
  updateStrategy:
    type: NotUpdate
  containers:
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent    
    args: [
      "/bin/sh",
      "-c",
      "cat /etc/filebeat.yml && sleep 36000", # In this example, only the filebeat configuration file is printed. 
    ]    
    volumeMounts:
    - name: config
      mountPath: /etc/filebeat.yml
      readOnly: true
      subPath: filebeat.yml    
  volumes:
  - name: config
    configMap:
      name: kube-system/filebeat-config # Specify the ConfigMap name in the namespace/name format.

# Authorize the pod that matches the filebeat-sidecarset SidecarSet to access the filebeat-config ConfigMap in the kube-system namespace. 
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
  name: filebeat-sidecarset-resourcebinding
  namespace: kube-system # The SidecarSetResourceBinding grants permissions only on resources in the kube-system namespace. 
  labels:
spec:
  subjects:
    - kind: SidecarSet
      name: filebeat-sidecarset
  resourceRefs:
    - kind: ConfigMap # Only the permissions to perform the following operations are granted: Get, List, and Watch. 
      name: filebeat-config

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: filebeat-sidecarset
spec:
  selector:
    matchLabels:
      serverless.alibabacloud.com/virtual-node: "true" # Add the label to select pods that are scheduled to virtual nodes. 
  updateStrategy:
    type: NotUpdate
  containers:
  # In this example, only the log of the sidecar container is printed. 
  - name: filebeat
    image: busybox
    imagePullPolicy: IfNotPresent
    args: [
      "/bin/sh",
      "-c",
      "cat /var/log/std/filebeat/0.log && sleep 36000",
    ]    
    volumeMounts:    
    - name: stdlog # Mount the pod stdout to the /var/log/std directory in the sidecar container to collect logs. 
      mountPath: /var/log/std
      readOnly: true
  volumes:  
  - name: stdlog
    flexVolume:
      driver: alicloud/pod-stdlog # Mount the pod stdout as a volume.

Procedure

In the following example, filebeat is deployed in a sidecar container and injected into a business pod named echo-server.

1. Deploy the configuration file of the filebeat container.

   The configuration file is a ConfigMap in the kube-system namespace. The ConfigMap will be mounted to the filebeat container. In this example, the configuration file is mounted to the sidecar container and printed. You do not need to replace relevant variables because they do not take effect.

   Show sample code

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: filebeat-config
     namespace: kube-system
     labels:
       k8s-app: filebeat
   data:
     filebeat.yml: |-
       filebeat.inputs:
       - type: log
         paths:
           - /var/log/std/*.log
         processors:
           - add_kubernetes_metadata:
               host: ${NODE_NAME} # This variable does not take effect. You do not need to modify it. 
               matchers:
               - logs_path:
                   logs_path: "/var/log/std/"
   
       # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
       #filebeat.autodiscover:
       #  providers:
       #    - type: kubernetes
       #      node: ${NODE_NAME}
       #      hints.enabled: true
       #      hints.default_config:
       #        type: container
       #        paths:
       #          - /var/log/containers/*${data.kubernetes.container.id}.log
   
       processors:
         - add_cloud_metadata:
         - add_host_metadata:
   
       cloud.id: ${ELASTIC_CLOUD_ID}  # This variable does not take effect. You do not need to modify it. 
       cloud.auth: ${ELASTIC_CLOUD_AUTH}  # This variable does not take effect. You do not need to modify it. 
   
       output.elasticsearch:
         hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
         username: ${ELASTICSEARCH_USERNAME}  # This variable does not take effect. You do not need to modify it. 
         password: ${ELASTICSEARCH_PASSWORD}  # This variable does not take effect. You do not need to modify it. 
   EOF

2. Deploy the SidecarSet used to inject the filebeat container.

   In this example, only the configuration of the filebeat container is printed. The stdout of the business pod is mounted as a volume to the filebeat container to collect logs.

   Show sample code

   kubectl apply -f - << EOF
   apiVersion: apps.kruise.io/v1alpha1
   kind: SidecarSet
   metadata:
     name: filebeat-sidecarset
   spec:
     selector:
       matchLabels:
         serverless.alibabacloud.com/virtual-node: "true" # Add the label to select pods that are scheduled to virtual nodes. 
     updateStrategy:
       type: NotUpdate
     containers:
     # In this example, filebeat is replaced by busybox cat. 
     #- name: filebeat
     #  image: docker.elastic.co/beats/filebeat:8.6.1
     #  args: [
     #    "-c", "/etc/filebeat.yml",
     #    "-e",
     #  ]
     - name: filebeat
       image: busybox
       imagePullPolicy: IfNotPresent
       args: [
         "/bin/sh",
         "-c",
         "cat /etc/filebeat.yml && sleep 36000",
       ]
       env:
       - name: ECI_SIDECAR_CONTAINER # The sidecar container exits after the business container is terminated. 
         value: "true"
       volumeMounts:
       - name: config
         mountPath: /etc/filebeat.yml
         readOnly: true
         subPath: filebeat.yml
       - name: stdlog # Mount the pod stdout to the /var/log/std directory in the sidecar container to collect logs. 
         mountPath: /var/log/std
         readOnly: true
     volumes:
     - name: config
       configMap:
         name: kube-system/filebeat-config # Specify the ConfigMap name in the namespace/name format. 
     - name: stdlog
       flexVolume:
         driver: alicloud/pod-stdlog # Mount the pod stdout as a volume. 
   EOF

3. Authorize the filebeat container to access the configuration file in the kube-system namespace.

   The echo-server pod is deployed in the default namespace. You need to explicitly authorize the filebeat container to access the configuration file in another namespace.

   Show sample code

   kubectl apply -f - << EOF
   # Authorize the pod that matches the filebeat-sidecarset SidecarSet to access the filebeat-config ConfigMap in the kube-system namespace. 
   apiVersion: sidecarset.alibabacloud.com/v1alpha1
   kind: SidecarSetResourceBinding
   metadata:
     name: filebeat-sidecarset-resourcebinding
     namespace: kube-system # The SidecarSetResourceBinding grants permissions only on resources in the kube-system namespace. 
     labels:
   spec:
     subjects:
       - kind: SidecarSet
         name: filebeat-sidecarset
     resourceRefs:
       - kind: ConfigMap
         name: filebeat-config
   EOF

4. Deploy the echo-server pod.

   Show sample code

   kubectl apply -f - << EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: echo-server
     labels:
       app: echo-server
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: echo-server
     template:
       metadata:
         labels:
           app: echo-server        
           alibabacloud.com/eci: "true"
       spec:
         containers:
           - name: echo-server
             image: hashicorp/http-echo
             imagePullPolicy: IfNotPresent
             args:
               - -listen=:8080
               - -text="hello world"
   EOF

5. View the business pod.

   kubectl get pod
   NAME                           READY   STATUS    RESTARTS   AGE
   echo-server-54f5cf556c-8ms5z   2/2     Running   0          41s

   The output shows that two containers run in the pod. This means that the injection is successful.

6. Verify that the stdout of the business pod is mounted to the filebeat container.

   kubectl exec echo-server-54f5cf556c-8ms5z -c filebeat -- cat /var/log/std/echo-server/0.log
   2023-07-07T00:40:07.927378153+08:00 stderr F 2023/07/06 16:40:07 Server is listening on :8080

   The output shows that the stdout of the business pod is visible in the filebeat container.

7. Verify that the filebeat-config configuration file in another namespace is mounted to the filebeat container.

   Show sample code

   kubectl exec echo-server-54f5cf556c-8ms5z -c filebeat -- cat /etc/filebeat.yml
   filebeat.inputs:
   - type: log  
     paths:
       - /var/log/std/*.log
   
   # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
   #filebeat.autodiscover:
   #  providers:
   #    - type: kubernetes
   #      node: ${NODE_NAME}
   #      hints.enabled: true
   #      hints.default_config:
   #        type: container
   #        paths:
   #          - /var/log/containers/*${data.kubernetes.container.id}.log
   
   processors:
     - add_cloud_metadata:
     - add_host_metadata:
   
   cloud.id: ${ELASTIC_CLOUD_ID}
   cloud.auth: ${ELASTIC_CLOUD_AUTH}
   
   output.elasticsearch:
     hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
     username: ${ELASTICSEARCH_USERNAME}
     password: ${ELASTICSEARCH_PASSWORD}

   The output shows that the filebeat-config configuration file in another namespace is mounted to the filebeat container.