Use ACK Virtual Node to automatically inject sidecar containers into pods on virtual nodes for observability and security. Combine with OpenKruise SidecarSet to customize injection rules and update sidecars independently.
How it works
ACK deploys observability and security agents as DaemonSets on physical nodes. Virtual nodes (ACS) do not support DaemonSets, so agents must run as sidecars. A SidecarSet injects a sidecar container into pods matching specific labels. Because a pod's destination node is unknown at creation, standard SidecarSet rules cannot target only virtual-node pods. ACK Virtual Node solves this by injecting sidecars after pods are scheduled to a virtual node, decoupling the sidecar lifecycle from the business container.
Key concepts
-
Sidecar container: A container added to a pod to extend the primary container without modifying it. Configure sidecar containers in ACS clusters as described in Feature description.
-
SidecarSet: A feature of OpenKruise, an open-source cloud-native application automation engine from Alibaba Cloud. SidecarSet automatically injects sidecar containers (such as monitoring or log collection agents) into matching pods, decoupling sidecar lifecycle from business containers.
Using SidecarSet with virtual nodes
Add the label serverless.alibabacloud.com/virtual-node: "true" to your SidecarSet to match all pods scheduled to a virtual node. This label is automatically added after scheduling. By default, elastic container instances are prioritized. Use a label like alibabacloud.com/compute-class: general-purpose to target a specific ACS compute class.
DaemonSet containers often depend on a ConfigMap. When the business pod and ConfigMap are in different namespaces, reference the ConfigMap in the sidecar volume using the namespace/name format. Cross-namespace access requires a SidecarSetResourceBinding.
The serverless.alibabacloud.com/virtual-node: "true" label is required only when you use ACS compute resources on virtual nodes in an ACK cluster. This label is not required in an ACK Serverless cluster.
Inject sidecar containers into specific pods
By default, a SidecarSet targets every pod on a virtual node. To target specific pods, modify the .spec.selector field:
apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
name: filebeat-sidecarset
spec:
containers:
...
selector:
matchLabels:
serverless.alibabacloud.com/virtual-node: "true"
alibabacloud.com/compute-class: general-purpose
app: nginx
|
Parameter |
Description |
|
serverless.alibabacloud.com/virtual-node |
Required. Matches all pods scheduled to a virtual node. |
|
alibabacloud.com/compute-class |
Optional. Targets pods of a specific compute class on virtual nodes. Compute class definition. |
|
app |
Optional. Add a custom label to target a specific workload. |
SidecarSetResourceBinding
When a sidecar volume references a ConfigMap or Secret from another namespace, create a SidecarSetResourceBinding to grant cross-namespace access.
This grants read-only (get, list, watch) permissions on the referenced ConfigMap and Secret.
# Authorize pods matching the filebeat-sidecarset SidecarSet to access the filebeat-config ConfigMap in the kube-system namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
name: filebeat-sidecarset-resourcebinding
namespace: kube-system # This SidecarSetResourceBinding can only grant permissions on resources in the kube-system namespace.
spec:
subjects:
- kind: SidecarSet
name: filebeat-sidecarset
resourceRefs:
- kind: ConfigMap
name: filebeat-config
- kind: Secret
name: elasticsearch-master-certs
Container startup, exit order, and job pods
Sidecar containers often have the following two requirements:
-
The sidecar container must start before the business container and exit after it.
-
For Job-type pods, the sidecar container must exit automatically after the business container finishes.
In ACS, set the __IS_SIDECAR__="true" environment variable in the sidecar container definition. Configure the startup and shutdown sequence of sidecar containers.
Update sidecar containers
Use the OpenKruise hot sidecar update feature to update injected sidecars without affecting pod availability. This feature is fully compatible with virtual nodes.
Scope
Cross-namespace ConfigMap access is supported only for CPU pods with the general-purpose and performance-optimized compute classes. For GPU compute classes, use the OpenKruise ResourceDistribution feature to distribute the ConfigMap to the target namespace.
Prerequisites
-
An ACK Pro cluster, ACK dedicated cluster, or ACK Serverless Pro cluster of version 1.22 or later. Create an ACK managed cluster, Create an ACK dedicated cluster (discontinued), or Create an ACK Serverless cluster.
-
The ACK Virtual Node component is installed (v2.13.0 or later). ACK Virtual Node.
-
The ack-kruise component is installed (v1.3.0 or later). ack-kruise.
-
The
SidecarSetServerlessPod=truefeature gate is enabled. SetSidecarSetServerlessPod=truein the featureGates parameter of kube-apiserver. Customize control plane component parameters.
Example
Inject a Filebeat container as a sidecar into an Nginx business pod.
-
Deploy the ConfigMap.
NoteThis ConfigMap is in the
kube-systemnamespace. In this example, the file is mounted to the sidecar container only to print its content. The variables do not take effect and do not need to be replaced.-
Create a file named
configmap.yamlwith the following content.apiVersion: v1 data: filebeat.yml: | filebeat.inputs: - type: log paths: - /var/log/* - /stdout/* output.elasticsearch: host: '${NODE_NAME}' hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]' username: '${ELASTICSEARCH_USERNAME}' password: '${ELASTICSEARCH_PASSWORD}' protocol: https ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ] kind: ConfigMap metadata: name: filebeat-config namespace: kube-system -
Run the following command to deploy the ConfigMap.
kubectl apply -f configmap.yaml
-
-
Deploy the SidecarSet for the filebeat container.
NoteIn this example, the filebeat container collects both file logs and stdout from the business container.
-
Create a file named
sidecarset.yamlwith the following content. -
Run the following command to deploy the SidecarSet.
kubectl apply -f sidecarset.yaml
-
-
Authorize the filebeat container to access the ConfigMap in the
kube-systemnamespace.NoteThe business pod is in the
defaultnamespace, so you must authorize the filebeat container to access the ConfigMap inkube-system.-
Create a file named
policy.yamlwith the following content.apiVersion: sidecarset.alibabacloud.com/v1alpha1 kind: SidecarSetResourceBinding metadata: name: filebeat-sidecarset-resourcebinding namespace: kube-system # This SidecarSetResourceBinding can only grant permissions on resources in the kube-system namespace. spec: subjects: - kind: SidecarSet name: filebeat-sidecarset resourceRefs: - kind: ConfigMap name: filebeat-config - kind: Secret name: elasticsearch-master-certs -
Run the following command to deploy the SidecarSetResourceBinding.
kubectl apply -f policy.yaml
-
-
Deploy the Nginx business pod. Create a stateless workload by using a Deployment.
apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx alibabacloud.com/compute-class: general-purpose alibabacloud.com/compute-qos: default spec: containers: - name: nginx image: mirrors-ssl.aliyuncs.com/nginx:latest resources: limits: cpu: "1" memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: # Share log directory with filebeat sidecar container via volumeMount - mountPath: /var/log/nginx name: varlog volumes: - name: varlog emptyDir: {} nodeSelector: type: virtual-kubelet tolerations: - key: virtual-kubelet.io/provider operator: Equal value: alibabacloud effect: NoSchedule -
Check the business pod.
kubectl get pods nginx-785d5xxxxx-xxxxxExpected output:
NAME READY STATUS RESTARTS AGE nginx-785d5xxxxx-xxxxx 2/2 Running 0 10mThe pod contains two containers (2/2 READY), confirming successful injection.
-
Verify that the file logs and stdout from the business pod are mounted to the filebeat container.
-
Run the following command to access the filebeat container.
kubectl exec -it deploy/nginx -c filebeat -- /bin/bash -
In the container, view the error logs.
cat /var/log/error.logExpected output:
2024/11/08 07:20:54 [notice] 1#1: using the "epoll" event method 2024/11/08 07:20:54 [notice] 1#1: nginx/1.27.2 2024/11/08 07:20:54 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14) 2024/11/08 07:20:54 [notice] 1#1: OS: Linux 5.10.134-17.2.1.lifsea8.x86_64 2024/11/08 07:20:54 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2024/11/08 07:20:54 [notice] 1#1: start worker processes 2024/11/08 07:20:54 [notice] 1#1: start worker process 29 -
In the container, view the stdout logs.
cat /stdout/nginx/0.logExpected output:
2024-11-08T15:20:53.99215101+08:00 stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration 2024-11-08T15:20:53.992173978+08:00 stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ 2024-11-08T15:20:54.003081339+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 2024-11-08T15:20:54.085010761+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 2024-11-08T15:20:54.276107913+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf 2024-11-08T15:20:54.276263126+08:00 stdout F /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh 2024-11-08T15:20:54.276842182+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 2024-11-08T15:20:54.345892283+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh 2024-11-08T15:20:54.347524813+08:00 stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
-
-
Verify that the cross-namespace
filebeat-configfile is mounted to the filebeat container.kubectl exec deploy/nginx -c filebeat -- cat /usr/share/filebeat/filebeat.ymlExpected output:
filebeat.inputs: - type: log paths: - /var/log/* - /stdout/* output.elasticsearch: host: '${NODE_NAME}' hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]' username: '${ELASTICSEARCH_USERNAME}' password: '${ELASTICSEARCH_PASSWORD}' protocol: https ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ]The ConfigMap is mounted correctly.