Virtual nodes do not support DaemonSets, so observability and security agents must run as sidecar containers within each pod. The ACK Virtual Node component works with OpenKruise SidecarSet to automatically inject sidecar containers into pods scheduled to virtual nodes. This decouples sidecar lifecycle management from the application container.
How sidecar injection works
On physical nodes, you deploy monitoring or security agents through a DaemonSet. Virtual nodes do not support DaemonSets, so agents must run as sidecar containers.
OpenKruise SidecarSet uses an admission webhook to inject a sidecar container into pods that match specified labels at creation time. However, if a pod's scheduling target is unknown until after the scheduling decision, SidecarSet alone cannot distinguish between pods destined for physical nodes and those destined for virtual nodes.
The ACK Virtual Node component solves this by adding the label serverless.alibabacloud.com/virtual-node: "true" to a pod after the pod is confirmed to be scheduled to a virtual node. By default, Elastic Container Instance (ECI) instances are prioritized. The SidecarSet then matches this label and injects the sidecar container only into virtual node pods.
Configure the serverless.alibabacloud.com/virtual-node: "true" label only when you use ACS computing power through virtual nodes in an ACK cluster. No extra configuration is needed when you use an ACS cluster directly.
Supported compute types
This feature supports only CPU pods of the following compute types:
general-purpose
compute-optimized instance
Prerequisites
An ACK Pro cluster, an ACK dedicated cluster, or an ACK Serverless cluster Pro is created. The cluster version must be 1.22 or later. For more information, see Create an ACK managed cluster, Create an ACK dedicated cluster, or Create an ACK Serverless cluster.
The ACK Virtual Node component is installed. The component version must be v2.13.0 or later. For more information, see ACK Virtual Node.
The ack-kruise component is installed. The component version must be v1.3.0 or later. For more information, see ack-kruise.
The
SidecarSetServerlessPod=truefeature gate is enabled. SetSidecarSetServerlessPod=truein the featureGates of the Kubernetes API server component. For more information, see Customize control plane component parameters.
Target specific pods with selector labels
By default, a SidecarSet that uses the virtual node label injects a sidecar container into all pods scheduled to virtual nodes. To target specific pods, add more labels to .spec.selector.
apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
name: filebeat-sidecarset
spec:
containers:
...
selector:
matchLabels:
serverless.alibabacloud.com/virtual-node: "true"
alibabacloud.com/compute-class: general-purpose
app: nginx| Label | Required | Description |
|---|---|---|
serverless.alibabacloud.com/virtual-node | Yes | Matches all pods scheduled to virtual nodes. |
alibabacloud.com/compute-class | No | Restricts injection to pods of a specific compute type, such as general-purpose. For more information, see Compute type definitions. |
Custom labels (for example, app) | No | Restricts injection to pods with a specific application label. |
Access cross-namespace resources with SidecarSetResourceBinding
A sidecar container often depends on a ConfigMap or Secret for its configuration. When the sidecar is injected into a pod in a different namespace, the pod cannot access resources in the sidecar's original namespace by default.
To reference a ConfigMap or Secret from another namespace, use the namespace/name format in the SidecarSet volume definition. Then create a SidecarSetResourceBinding to authorize cross-namespace access.
SidecarSetResourceBinding grants only read-only (Get, List, and Watch) permissions for the specified ConfigMap and Secret resources. The binding can only authorize resources in its own namespace.
apiVersion: sidecarset.alibabacloud.com/v1alpha1
kind: SidecarSetResourceBinding
metadata:
name: filebeat-sidecarset-resourcebinding
namespace: kube-system # Can only grant access to resources in the kube-system namespace
spec:
subjects:
- kind: SidecarSet
name: filebeat-sidecarset
resourceRefs:
- kind: ConfigMap
name: filebeat-config
- kind: Secret
name: elasticsearch-master-certsThis authorization grants only read-only (Get, List, and Watch) permissions for the ConfigMap and Secret.
Control container startup and exit order
Sidecar containers typically need to:
Start before the application container and exit after it.
For Job pods, actively exit after the application container exits.
In an ACS scenario, set the environment variable __IS_SIDECAR__="true" on the sidecar container to enable this behavior. For more information, see Configure the startup and shutdown order of sidecar containers.
Upgrade the sidecar container independently
After you deploy a sidecar container, you can upgrade it independently using the OpenKruise sidecar hot upgrade feature. This feature performs seamless upgrades without affecting pod availability and is fully compatible with the virtual node method.
Example: inject a Filebeat sidecar into an Nginx pod
The following walkthrough deploys a Filebeat sidecar container alongside an Nginx application pod. The Filebeat sidecar collects both file logs and standard output from the application container.
Step 1: Deploy the ConfigMap
This ConfigMap provides the Filebeat configuration. It resides in the kube-system namespace. In this example, the configuration is only mounted and printed. The environment variables in the configuration do not take effect and do not need to be replaced.
Create a file named
configmap.yaml:apiVersion: v1 data: filebeat.yml: | filebeat.inputs: - type: log paths: - /var/log/* - /stdout/* output.elasticsearch: host: '${NODE_NAME}' hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]' username: '${ELASTICSEARCH_USERNAME}' password: '${ELASTICSEARCH_PASSWORD}' protocol: https ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ] kind: ConfigMap metadata: name: filebeat-config namespace: kube-systemDeploy the ConfigMap:
kubectl apply -f configmap.yaml
Step 2: Deploy the SidecarSet
This SidecarSet defines the Filebeat sidecar container and its injection rules.
Create a file named
sidecarset.yaml:Deploy the SidecarSet:
kubectl apply -f sidecarset.yaml
Step 3: Create the SidecarSetResourceBinding
Because the application pod runs in the default namespace, the injected Filebeat container needs explicit authorization to access the ConfigMap and Secret in the kube-system namespace.
Create a file named
policy.yaml:apiVersion: sidecarset.alibabacloud.com/v1alpha1 kind: SidecarSetResourceBinding metadata: name: filebeat-sidecarset-resourcebinding namespace: kube-system # Can only grant access to resources in the kube-system namespace spec: subjects: - kind: SidecarSet name: filebeat-sidecarset resourceRefs: - kind: ConfigMap name: filebeat-config - kind: Secret name: elasticsearch-master-certsDeploy the SidecarSetResourceBinding:
kubectl apply -f policy.yaml
Step 4: Deploy the application pod
Deploy an Nginx Deployment that schedules pods to virtual nodes. For more information, see Create a stateless application using a Deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
alibabacloud.com/compute-class: general-purpose
alibabacloud.com/compute-qos: default
spec:
containers:
- name: nginx
image: mirrors-ssl.aliyuncs.com/nginx:latest
resources:
limits:
cpu: "1"
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
# Share log directory with filebeat sidecar container
- mountPath: /var/log/nginx
name: varlog
volumes:
- name: varlog
emptyDir: {}
nodeSelector:
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Equal
value: alibabacloud
effect: NoScheduleStep 5: Verify the injection
Check the pod status: Expected output: The
2/2in the READY column confirms that the sidecar container was injected successfully.kubectl get pods nginx-785d5xxxxx-xxxxxNAME READY STATUS RESTARTS AGE nginx-785d5xxxxx-xxxxx 2/2 Running 0 10mVerify that the Filebeat container can access file logs from the application container: Inside the container, view the error log: Expected output:
kubectl exec -it deploy/nginx -c filebeat -- /bin/bashcat /var/log/error.log2024/11/08 07:20:54 [notice] 1#1: using the "epoll" event method 2024/11/08 07:20:54 [notice] 1#1: nginx/1.27.2 2024/11/08 07:20:54 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14) 2024/11/08 07:20:54 [notice] 1#1: OS: Linux 5.10.134-17.2.1.lifsea8.x86_64 2024/11/08 07:20:54 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2024/11/08 07:20:54 [notice] 1#1: start worker processes 2024/11/08 07:20:54 [notice] 1#1: start worker process 29Verify that the Filebeat container can access the standard output of the application container: Expected output:
cat /stdout/nginx/0.log2024-11-08T15:20:53.99215101+08:00 stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration 2024-11-08T15:20:53.992173978+08:00 stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ 2024-11-08T15:20:54.003081339+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 2024-11-08T15:20:54.085010761+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 2024-11-08T15:20:54.276107913+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf 2024-11-08T15:20:54.276263126+08:00 stdout F /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh 2024-11-08T15:20:54.276842182+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 2024-11-08T15:20:54.345892283+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh 2024-11-08T15:20:54.347524813+08:00 stdout F /docker-entrypoint.sh: Configuration complete; ready for start upVerify that the Filebeat container has mounted the cross-namespace configuration file: Expected output: This output confirms that the cross-namespace ConfigMap is mounted correctly.
kubectl exec deploy/nginx -c filebeat -- cat /usr/share/filebeat/filebeat.ymlfilebeat.inputs: - type: log paths: - /var/log/* - /stdout/* output.elasticsearch: host: '${NODE_NAME}' hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]' username: '${ELASTICSEARCH_USERNAME}' password: '${ELASTICSEARCH_PASSWORD}' protocol: https ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ]