In addition to Simple Log Service, you can develop your own log collection system to collect logs from applications deployed in Container Compute Service (ACS). An example of such a system is the EFK stack, which consists of Elasticsearch, Filebeat, and Kibana. ACS does not support the deployment of DaemonSet components. Therefore, a log collection agent, such as Filebeat, must run in a sidecar container to collect logs. This topic describes how to deploy a custom log collection agent on ACS.
Prerequisites
An ACS cluster is created. For more information, see Create an ACS cluster.
Background information
In typical log collection solutions, an agent, such as Filebeat, is deployed as a sidecar container that runs alongside the application container in each pod instance. The agent collects logs generated by the application container.
In sidecar mode, the log collection agent must share a log directory with the application container. The application container writes logs to the shared directory. The agent listens for changes in the log files stored in the shared directory and collects new logs.
The following types of logs can be collected:
Standard output
Collecting standard output logs relies on an EmptyDir volume with its Medium set to Stdout. When you create a pod, you can mount this volume to the sidecar container. The sidecar container can then access standard output logs, which are collected by the basic components of ACS as files.
NoteEach pod can have a maximum of 1 EmptyDir volume with its Medium set to Stdout. The volume must be mounted to at least 1 container.
Collecting standard output logs adds performance overhead and increases storage costs.
Automatic rotation is configured for standard output logs. For more information, see log rotation. Log rotation is triggered when a log file reaches 10 MiB. A maximum of 5 log files are stored for each container.
Text files
To collect text files, you can use a shared volume in a pod. A volume can be mounted to multiple containers in a pod. This allows the sidecar container to collect text files that are stored by the application container in the shared volume.
Function Introduction
Container startup and exit order and pods created by Jobs
Sidecar containers usually have the following requirements:
Sidecar containers must start before application containers and exit after application containers.
For pods created by Jobs, sidecar containers must exit after the application containers exit.
To meet these requirements in ACS, you can set the __IS_SIDECAR__="true" environment variable in the sidecar container configuration. For more information, see Configure the startup and stop sequence of sidecar containers.
Collect container file logs in sidecar mode
If you use a sidecar container to collect the file logs of an application container, you can use a shared volume. For more information, see collecting application container file logs in sidecar mode.
Collect container stdout logs in sidecar mode
In ACS, you can set volumes.emptyDir.medium=Stdout to collect container stdout logs. For more information about the configuration, see Scenario 1.
Scenario 1: Collect application file logs and stdout logs
Use the following content to create a deployment that contains a Filebeat sidecar container.
You can configure a StatefulSet in the same way.
Run the following command to query information about the pod.
kubectl get pods -l app=nginxNAME READY STATUS RESTARTS AGE nginx-85776xxxxx-xxxxx 2/2 Running 0 13mView the logs.
Run the following command to enter the filebeat container.
kubectl exec -it deploy/nginx -c filebeat -- /bin/bashIn the filebeat container, view the application logs.
cat /var/log/error.logExpected output:
2024/10/30 10:38:28 [notice] 1#1: using the "epoll" event method 2024/10/30 10:38:28 [notice] 1#1: nginx/1.21.5 2024/10/30 10:38:28 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 2024/10/30 10:38:28 [notice] 1#1: OS: Linux 5.10.134-17.0.2.lifsea8.x86_64 2024/10/30 10:38:28 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2024/10/30 10:38:28 [notice] 1#1: start worker processes 2024/10/30 10:38:28 [notice] 1#1: start worker process 32In the filebeat container, view the standard output.
cat /stdout/nginx/0.logExpected output:
2024-10-30T18:38:28.37305498+08:00 stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration 2024-10-30T18:38:28.373072927+08:00 stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ 2024-10-30T18:38:28.379999685+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 2024-10-30T18:38:28.391562633+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 2024-10-30T18:38:28.405538758+08:00 stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf 2024-10-30T18:38:28.405657957+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh 2024-10-30T18:38:28.409014952+08:00 stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh 2024-10-30T18:38:28.410397615+08:00 stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
Scenario 2: Collect logs from task-based applications
Use the following content to deploy a task that is built based on busybox to print to the standard output. This task automatically exits after 100 s.
The following two YAML files show different ways to configure a sidecar container. The deployed tasks are identical.
Kubernetes 1.28 and earlier versions
Use the following content to deploy the busybox-stdout task.
Enter the Filebeat container. You can view the file logs and standard output from the nginx container.
% kubectl exec -it busybox-stdout-4fwnx -c filebeat -- /bin/bash $ cat /stdout/busybox/0.log 2024-11-07T18:46:06.77002301+08:00 stdout F Log message 1 2024-11-07T18:46:07.77143368+08:00 stdout F Log message 2 2024-11-07T18:46:08.772840307+08:00 stdout F Log message 3 2024-11-07T18:46:09.774047629+08:00 stdout F Log message 4 2024-11-07T18:46:10.776122173+08:00 stdout F Log message 5 2024-11-07T18:46:11.778271688+08:00 stdout F Log message 6
Kubernetes 1.29 and later versions
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
data:
filebeat.yml: |
filebeat.inputs:
- type: log
paths:
- /stdout/*
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: [ "/usr/share/filebeat/certs/ca.crt" ]
---
apiVersion: batch/v1
kind: Job
metadata:
name: busybox-stdout
spec:
template:
spec:
initContainers:
- args:
- -e
- -E
- http.enabled=true
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: ELASTICSEARCH_USERNAME
valueFrom:
secretKeyRef:
key: username
name: elasticsearch-master-credentials
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: elasticsearch-master-credentials
image: mirrors-ssl.aliyuncs.com/docker.elastic.co/beats/filebeat:8.5.1
imagePullPolicy: Always
restartPolicy: Always # Declare this container as a sidecar.
name: filebeat
resources:
limits:
cpu: 250m
ephemeral-storage: 30Gi
memory: 512Mi
requests:
cpu: 250m
ephemeral-storage: 30Gi
memory: 512Mi
volumeMounts:
- mountPath: /stdout
name: stdout-log
readOnly: true
- mountPath: /usr/share/filebeat/certs/
name: elasticsearch-master-certs
- mountPath: /usr/share/filebeat/filebeat.yml
name: filebeat-config
readOnly: true
subPath: filebeat.yml
containers:
- name: busybox
image: mirrors-ssl.aliyuncs.com/busybox:latest
command: [ "sh", "-c" ]
args:
- |
for i in $(seq 1 100); do
echo "Log message $i"
sleep 1
done
volumes:
## Collect stdout logs.
- emptyDir:
medium: Stdout
name: stdout-log
- name: elasticsearch-master-certs
secret:
secretName: elasticsearch-master-certs
- configMap:
name: filebeat-config
name: filebeat-config
restartPolicy: OnFailure
# The following configuration is for demonstration purposes only.
# In a production environment, replace it with the configuration that is automatically generated when you deploy EFK.
---
apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURJakNDQWdxZ0F3SUJBZ0lSQUl0SDZxR2YzRG9VNFBuVWRJOUdITlV3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVpXeGhjM1JwWTNObFlYSmphQzFqWVRBZUZ3MHlOREV3TWprd056RTNNamxhRncweQpOVEV3TWprd056RTNNamxhTUJzeEdUQVhCZ05WQkFNVEVHVnNZWE4wYVdOelpXRnlZMmd0WTJFd2dnRWlNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFET1Zxc0svVXAvWTVNRVY5T3hzTUk2TTZMTFhYMGsKNFBGSjE0SklVNStUVnBRVVNhVmx3N0REeGtJaUQ3RDVHZ3I3Snh4WHV0cFNjVlo0QWN0UzNrNFJvV3lqdzg0cApoYW4wY3JZL2VaQmJlWjVFUUhCSXprU0ZhMWd4bkpUcVErSSsvR3lKSlNHNkQyR21UVHRqRXZ2R2pWL1loSDNHCk1DMnRadVNXN1hPYWZBKzFqWUNkVFpIZkNpeDdBZURVNU0zcVplNzR4MjhTeitDNkM4WUFCQ0ZSTnJsVGNFQW8KaGQ5WGNnellPUGdJY2VZSUJWb25DTDdzVWFPZGVKa1hrbmdBR2ZzWjB6RnJhMm1qZGZtcHVIaWZFM21LbUZ1agpydGhXVFZTdE9oZGtIUnZTck52NDZaSFdtYlErNXZCb1RiODllTFZuNTNwbzhmSkJIWWpHZ24zdEFnTUJBQUdqCllUQmZNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQWRCZ05WSFNVRUZqQVVCZ2dyQmdFRkJRY0RBUVlJS3dZQkJRVUgKQXdJd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTU2dmVtcDRMem9QdVJiOUY3bjlmcU9JNQp2blF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUs2alBULzc2RnV2K0RLSmsxNG15b0ZzNThnRjRqbjlLWEUwCmFqOEMrZ1BUd2o2dUpUTjRLcWFmcnV0VGxlZWM5cXhabVZjQTgzanJhTEkySzlNN2ZyVE9pVE1vSnhmNnFrU1AKZ1ozazF2OG40Z0JGbzhsczZpc2YrankvL1dpMiswUVdWOElIU1lRbDlucklCT0lpb25rS1ljbDVQY2tKWVo4RQpkYVJUYW1xbi8zRTFGODFGaXFDT3dPc0NGRk5IRHhPRDRRb25NbU5ReFFvb1I3Mks1V2R6TmQrTG5BbjE4eHZlCjE2b3gybzZNQ3hjQS9RWDE0d3dDVi9lb1Y4KzIwWlJRY01LT1U5Y0djSjZNYm9TSW1odzJzS0NpNkpYcUQwQWMKSFZscWFzTGh4cHZBc3lJdXY3TjB5VnVhVVdZVDVMV1oyOFBGUVozcmwvYTIvNHZHNStJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpRENDQW5DZ0F3SUJBZ0lSQUp3R2cyNVR5YUNEVmlxNWJEbDliREF3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXhNUVpXeGhjM1JwWTNObFlYSmphQzFqWVRBZUZ3MHlOREV3TWprd056RTNNamxhRncweQpOVEV3TWprd056RTNNamxhTUI4eEhUQWJCZ05WQkFNVEZHVnNZWE4wYVdOelpXRnlZMmd0YldGemRHVnlNSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUE0L1QzMUp5VzJTSCtHZjhVTHJXVmhYZmwKdHhJRytMcGFGK0l6Q2lnSGh6QW5ZZnoyM0luQTZLOG9WWWtsejZLZzRzbEZjeXRHbStxd01ta3c3QzZXeHRVRwpSWlZiUXloZDZ6L1laRGhGUVpCQXZMWDVRVnIwWmIyRlN0NjdQM3dNa3Z1RE16TVZnRXoxZ2x0TmRyOVZiOXQyCjRUTXBka09GeFpPV24rZ0IyM0l3YnlNb0ZIMVNDZ1ZtcC9EQTNHZU1ENGErWURUcGowd1dSUFRRemdNcXh4YkUKd3FFdGN5R01yLzF6Sm4ycDZ6SWdmV3E0K1pwM2lRU2VOdjFUWWpHVm5xYzdWWUhDd29nb3pSRDI5TldTbC9BMApVSTVsbld3SDI3aU51QU1pVWFQRmx4eWtNbTlFbSs4SUcrT2VsRks4aDlBSEl2TDVYSFJjT3VOQzk2SjU1d0lECkFRQUJvNEhDTUlHL01BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUkKS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVdnQlRucTk2YW5ndk9nKzVGdjBYdQpmMStvNGptK2REQmZCZ05WSFJFRVdEQldnaFJsYkdGemRHbGpjMlZoY21Ob0xXMWhjM1JsY29JY1pXeGhjM1JwClkzTmxZWEpqYUMxdFlYTjBaWEl1WkdWbVlYVnNkSUlnWld4aGMzUnBZM05sWVhKamFDMXRZWE4wWlhJdVpHVm0KWVhWc2RDNXpkbU13RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQURwMVlXOFBmMm5YMldqeU5YZlVTSVJiamN0YQo2RTVpQzEwdENlbUxFZVpPZlpSRGtKOUVSaUNFQ2tUVUNCNy9QemFrTlE1UGNYQzVmcmYyWCtucGZ3RFFyREN4ClM0WkpEMWZFeHN0Yis3L29yQmgrWXNYcHJiUUJMbDJ6M0w0dm5tZ1kyb3V5bjdyT2NOdWQveENvWUdBVUd4a0YKdmVvUDNld0NUVzlaUVhGVWF0WUUzMno1bHRXTlRTOE5RU1hQRUtoSUlqYWNOL29SQ2ZhY1pRaTFoOUhTczdzQQpOcmludkRCTnE4bDl0b3g5NFZadCtXN3NmUXZvVU5hTTV1OXk0UU5Ib25rdUZ2enZMdkpGeEtvbWE0RmtFOHl5CmphR2RpUXh2NVFXdW1sTlBzZ3VOUUpSMnp3QzJEUkVVZUR1WC96Zk9xdDBucDFOZWpoWU11VTIyVk5zPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNC9UMzFKeVcyU0grR2Y4VUxyV1ZoWGZsdHhJRytMcGFGK0l6Q2lnSGh6QW5ZZnoyCjNJbkE2SzhvVllrbHo2S2c0c2xGY3l0R20rcXdNbWt3N0M2V3h0VUdSWlZiUXloZDZ6L1laRGhGUVpCQXZMWDUKUVZyMFpiMkZTdDY3UDN3TWt2dURNek1WZ0V6MWdsdE5kcjlWYjl0MjRUTXBka09GeFpPV24rZ0IyM0l3YnlNbwpGSDFTQ2dWbXAvREEzR2VNRDRhK1lEVHBqMHdXUlBUUXpnTXF4eGJFd3FFdGN5R01yLzF6Sm4ycDZ6SWdmV3E0CitacDNpUVNlTnYxVFlqR1ZucWM3VllIQ3dvZ296UkQyOU5XU2wvQTBVSTVsbld3SDI3aU51QU1pVWFQRmx4eWsKTW05RW0rOElHK09lbEZLOGg5QUhJdkw1WEhSY091TkM5Nko1NXdJREFRQUJBb0lCQUhSK3VEaDdYY3ZSUjE1WgpzU0s5d1kvWDJobFlxUjlyZktjLy9mMXV2NG9pM2IyQjNWYVBQM3FxS042dG5Ca2tienYyeC9zM1hucEgwWXV4Cm5rTFUvRkRZaE1BQ3VBVDJHQ2tsRTUwRDlNQ3d5NlNsQ3FDUHJ0NWZvRUxHMk1KMHpxZyt5S25kclZ0SCtSK0oKTVdsQ0ZwTjNnS1ZOMUI2UUcwa0JSN1NvaUdwd1ZhSEFOMEc2Q0NKTnR0eDByZHo5UVRpU3BCMmlzYmhaSVpRVgp2U1NwemY3R2ZzaU5pN1VaOTN0WkZ5S21PRE1CUW5yZm5xWkphVXdVeWhHYk02Y2h4YWpmY0dka1dadWJ6azZJCnUrTWFDcE5VOWsvSXpDUkhnNjVyZGhITlozeVdDYzh6UXRDdFNkaWFPRmFwclZ4aHhSeFU5WW9FT2hsaVlVS0wKNGhIN1VPRUNnWUVBNkkrL3MyMDN5WGx1R0tqb3VIRXc5Z29aNU9YNHk3WWFGb2dzTm9jMTBZdUxRSURjTEozTQpLWjc4aTZ1dlV3d2lVMVpKTmRYbHFhcFFDNFRBN2N0bktnTVpNRnhzYzFPQzZsTnJ6b3FXM3FST0NDNW9DRUJOCmVSQlR2V1FjOVhXUUxrK2wwcHBEZ3BaVFVjc0tYS2NJSldlakhVb0xuajMrcmo2ZjFsOHlhU01DZ1lFQSt1NXEKd1ZtU2NkQ1pqdXQzbktsZ3V6ZGlpbWpvZ0Jkc1VnRUg4WXhkK2QzTXgveVR0eGthdGNwMnYvTHNPQ3BMV1YzSQpNVTdCTUtVR3BpZGpOMzdYQ3FMREZHM3UzdHZhMHBjditsOENaYk45NWdVN3RNeHB2WjZyZjRwVFlKdG1Ya2RsCkVmUkNxMkhocU11UEpSUlhjMllOaDU2TjNSUW5CWnJRVmxLdjBtMENnWUVBb0Fvc2RpRjIvcU1kN01Kd1JGMUEKd0ZCN09WWTVQSmI0cFFEWXpEMkgvOGZ6OEZPOU1NYjJ0TDNBTmEzVVhXWkFTUEZjT0R3V2JBZlVSZGo1bTZzYQpONE1pVm5HRUFHazc4bDJ1RnRpd3NrNkhsSUc2L2RLaWZlbUtkdzdxRHREMGc2bzBCeFk1MXlmejlwbXZhOHRXCmc4Y3FMUUhEdFFZY3VYUkhNcE1ZY2RrQ2dZRUFrZHdxbys5OEo3cDR1Rkg1T2xCZWtSVFZxOXpsWVNlOGFFSi8KS3BKTVFpVUNsekVqY0NnZ2xaRjF5NGZhZFo5b0l5OVhZZ29FVkZGbzl3WW9MeWNFdXdMM1lKV3laMHJtL01pegpNOWNzWG8raVhDV29taVRFUmx2SUZxQUNiVUtIazcvdWFTeFI0S3RKNzhNN2x2TW5Ea1pCRVJkQ0lVTklsNEp4CkhleDhsVlVDZ1lCdlVMQUU3WkMyTkFnZUpYUmpRalFRd2laNU1jS09rTTBvVjlsYUlaR0xZUUVTOHVtYXA0THYKMVc5clN4UElxMTg3Q1haejBuOTdPN210aUpwYXhERDg1QVdZMEg5MGhFNitvZUltWmlPN3ZYbW5RTVRNZjNtWgp6dUcvNk84ckpsNWcyNnY4NTVBYjVUbC9ZQTNRcW1tOVdKdUt5eGw1bWxvMGkxNU14cWNia1E9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
name: elasticsearch-master-certs
type: kubernetes.io/tls
---
apiVersion: v1
data:
password: Z3BVMTFFZXZNWWFmMkVCUw==
username: ZWxhc3RpYw==
kind: Secret
metadata:
name: elasticsearch-master-credentials
type: OpaqueScenario 3: Automatically inject sidecar containers based on an OpenKruise SidecarSet
The preceding methods require you to add the sidecar container configuration to each workload. This increases the management overhead if you have many workloads. You can use an OpenKruise SidecarSet to manage sidecar containers. This lets you automatically inject and independently upgrade sidecar containers. This way, you can perform the configuration only once to achieve automatic injection. For more information, see Inject a sidecar container into a pod on a virtual node.