This topic describes how to create a Logtail configuration in the Log Service console and use the Logtail configuration to collect container text logs in DaemonSet mode.
Prerequisites
Features
- Allows you to specify a log file path in a container. You do not need to manually map the log file path to a path on the host.
- Uses the container label whitelist to specify containers from which text logs are collected.
- Uses the container label blacklist to specify containers from which text logs are not collected.
- Uses the environment variable whitelist to specify containers from which text logs are collected.
- Uses the environment variable blacklist to specify containers from which text logs are not collected.
- Collects multi-line logs. For example, Logtail can collect Java stack logs.
- Automatically associates container metadata that needs to be uploaded together with the collected container text logs. The metadata includes container names, image names, pod names, namespaces, and environment variables.
- If a container runs in a Kubernetes cluster, Logtail also supports the following features:
- Uses Kubernetes namespaces, pod names, and container names to specify containers from which text logs are collected.
- Uses the Kubernetes label whitelist to specify containers from which text logs are collected.
- Uses the Kubernetes label blacklist to specify containers from which text logs are not collected.
- Automatically associates Kubernetes labels that need to be uploaded together with the collected container text logs.
Limits
- If Logtail detects the
die
event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost. - For Docker containers, only overlay and overlay2 storage drivers are supported. If
other storage drivers are used, you must mount a volume to the directory of logs.
Then, a temporary directory is generated.
If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode.
- Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
- If a volume is mounted to the data directory of a container, Logtail cannot collect
data from the parent directory of the data directory. You must specify the complete
path of the data directory as the collection directory.
For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.
- By default, Kubernetes mounts the root directory of the host to the
/logtail_host
directory of the Logtail container. If you want to collect text logs from the host, you must specify/logtail_host
as the prefix of the log file path.For example, if you want to collect logs from the
/home/logs/app_log/
directory of the host, you must specify/logtail_host/home/logs/app_log/
as the log file path. - Logtail collects data from containers that use the Docker engine or containerd engine.
- Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
Create a Logtail configuration
Configuration examples
Example 1: Filter containers based on the environment variable whitelist and the environment variable blacklist
Collect text logs from the containers whose environment variable configurations include
NGINX_SERVICE_PORT=80
but exclude POD_NAMESPACE=kube-system
. The log file path is /var/log/nginx/access.log
. The logs are parsed in simple mode.
- Obtain environment variables.
To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.
- Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Example 2: Filter containers based on the container label whitelist and the container label blacklist
Collect text logs from the containers whose container label is io.kubernetes.container.name=nginx
. The log file path is /var/log/nginx/access.log
. The logs are parsed in simple mode.
- Obtain container labels.
To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.
- Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Example 3: Filter containers by using Kubernetes namespaces, pod names, and container names
Collect text logs from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.
- Obtain different levels of Kubernetes information.
- Obtain information about pods.
- Obtain information about namespaces.
- Obtain information about pods.
- Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Example 4: Filter containers by using Kubernetes labels
Collect text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.
- Obtain Kubernetes labels.
- Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Default fields
Log field | Description |
---|---|
_image_name_ | The name of the image. |
_container_name_ | The name of the container. |
_pod_name_ | The name of the pod. |
_namespace_ | The namespace of the pod. |
_pod_uid_ | The unique identifier of the pod. |
_container_ip_ | The IP address of the pod. |