Use the Log Service console to collect container text logs in DaemonSet mode - Log Service

This topic describes how to create a Logtail configuration in the Log Service console and use the Logtail configuration to collect container text logs in DaemonSet mode.

Prerequisites

The Logtail component is installed. For more information, see Install Logtail components in a Kubernetes cluster.
A Logstore is created in the project that you use to install the Logtail component. For more information, see Create a Logstore.
The container from which you want to collect logs continuously generates logs.
Important Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

Features

Logtail can collect container text logs and then upload the text logs together with container metadata to Log Service. Logtail supports the following features:

Allows you to specify a log file path in a container when you collect text logs. You do not need to manually map the log file path to a path on the host.
Uses the container label whitelist to specify containers from which logs are collected.
Uses the container label blacklist to specify containers from which logs are not collected.
Uses the environment variable whitelist to specify containers from which logs are collected.
Uses the environment variable blacklist to specify containers from which logs are not collected.
Collects multi-line logs. For example, Logtail can collect Java stack logs.
Automatically associates container metadata that needs to be uploaded together with the collected container logs. The metadata includes container names, image names, pod names, namespaces, and environment variables.
If a container runs in a Kubernetes cluster, Logtail also supports the following features:
- Uses Kubernetes namespaces, pod names, and container names to specify containers from which logs are collected.
- Uses the Kubernetes label whitelist to specify containers from which logs are collected.
- Uses the Kubernetes label blacklist to specify containers from which logs are not collected.
- Automatically associates Kubernetes labels that need to be uploaded together with the collected container logs.

Limits

If Logtail detects the die event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost.
For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.
If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode. For more information, see Use CRDs to collect container text logs in Sidecar mode and Use the Log Service console to collect container text logs in Sidecar mode.
Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
If a volume is mounted to the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.
For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.
By default, Kubernetes mounts the root directory of the host to the /logtail_host directory of the Logtail container. If you want to collect text logs from the host, you must specify /logtail_host as the prefix of the log file path.
For example, if you want to collect logs from the /home/logs/app_log/ directory of the host, you must specify /logtail_host/home/logs/app_log/ as the log file path.
Logtail collects data from containers that use the Docker engine or containerd engine.
- Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

Create a Logtail configuration

Important By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?.

Log on to the Log Service console.
In the Import Data section, click Kubernetes - Object.
Select a project and a Logstore. Then, click Next.
In this example, select the project that you use to install the Logtail component and the Logstore that you create.
Click Use Existing Machine Groups.
After you install the Logtail component, Log Service automatically creates a machine group named k8s-group-${your_k8s_cluster_id}. You can select this machine group.
Select the k8s-group-${your_k8s_cluster_id} machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
Important If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?

Configure the parameters for the Logtail configuration and click Next.

Configure basic settings, such as the name, log path, collection mode, filters, and plug-ins that are used to facilitate data processing.
The preceding configuration items are the same as the items that you can configure to collect server text logs. For more information, see Collect text logs.
Note
- When you collect container text logs, you can select a collection mode based on your business requirements. Collection modes include Simple Mode, Simple Mode - Multi-line, Full Regex Mode, JSON Mode, Delimiter Mode, and NGINX Configuration Mode. Logtail parses logs based on the collection mode that you select. If you want to collect multi-line logs, you can select Simple Mode - Multi-line or Full Regex Mode for the Mode parameter. If you want to collect JSON logs, you can select JSON Mode for the Mode parameter. For more information about each collection mode, see Collection modes.
- If your logs cannot be parsed in the collection mode that you select, you can turn on Enable Plug-in Processing and add Logtail plug-ins to parse logs. For example, you can add Logtail plug-ins to extract fields, extract log time, mask data, and filter logs. For more information, see Logtail plug-ins.
Turn on Docker File.

Optional:Specify conditions to filter containers.

For versions earlier than Logtail V1.0.34, you can filter containers only by using environment variables and container labels.

A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the LabelKey parameter for a namespace is io.kubernetes.pod.namespace. The value of the LabelKey parameter for a container name is io.kubernetes.container.name. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specify io.kubernetes.pod.namespace : backend-prod or io.kubernetes.container.name : worker-server in the container label whitelist.

Important

Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels.
Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values for the LabelKey parameter, only one of the values takes effect.


Parameter	Description
Label Whitelist	The container label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
Label Blacklist	The container label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to `io.kubernetes.container.name` and set the LabelValue parameter to `^(nginx\|cube)$`, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.
Environment Variable Whitelist	The environment variable whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are matched. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are matched. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
Environment Variable Blacklist	The environment variable blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are filtered out. If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are filtered out. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to `NGINX_SERVICE_PORT` and set the EnvValue parameter to `^(80\|6379)$`, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.

For Logtail V1.0.34 or later, we recommend that you use different levels of Kubernetes information, such as pod names, namespaces, container names, and labels to filter containers.

Turn on Deployed in K8s and configure the following parameters to filter containers.

Note If you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods.


Parameter	Description
K8s Pod Name Regular Matching	The pod name. The pod name specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify `^(nginx-log-demo.*)$`, all containers in the pod whose name starts with nginx-log-demo are matched.
K8s Namespace Regular Matching	The namespace. The namespace specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify `^(default\|nginx)$`, all containers in the nginx and default namespaces are matched.
K8s Container Name Regular Matching	The container name. The container name specifies the containers from which text logs are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify `^(container-test)$`, all containers whose name is container-test are matched.
K8s Label Whitelist	The Kubernetes label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is matched.
K8s Label Blacklist	The Kubernetes label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out. If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (`^`) and ends with a dollar sign (`$`), regular expression matching is performed. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `^(test1\|test2)$`, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is filtered out.

Optional:Specify log labels.

For Logtail V1.0.34 or later, we recommend that you specify environment variables or Kubernetes labels for logs as log labels.


Parameter	Description
Environment Variable Log Tag	After you specify environment variables as log labels, Log Service adds environment variable-related fields to logs. For example, if you set the EnvKey parameter to `VERSION` and set the EnvValue parameter to `env_version`, Log Service adds the __tag__:__env_version__: v1.0.0 field to logs if the environment variable configurations of a container include `VERSION=v1.0.0`.
K8s Label Log Tag	After you specify Kubernetes labels as log labels, Log Service adds Kubernetes label-related fields to logs. For example, if you set the LabelKey parameter to `app` and set the LabelValue parameter to `k8s_label_app`, Log Service adds the __tag__:__k8s_label_app__: serviceA field to logs if the label configurations of a Kubernetes cluster include `app=serviceA`.

Preview data, configure indexes, and then click Next.
By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
Click Log Query. You are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Configuration examples

Filter containers based on the environment variable whitelist and the environment variable blacklist

Collect text logs from the containers whose environment variable configurations include NGINX_SERVICE_PORT=80 but exclude POD_NAMESPACE=kube-system. The log file path is /var/log/nginx/access.log. The logs are collected in simple mode.

Obtain environment variables.
To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.
Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Filter containers based on the container label whitelist and the container label blacklist

Collect text logs from the containers whose container label is io.kubernetes.container.name=nginx. The log file path is /var/log/nginx/access.log. The logs are collected in simple mode.

Obtain container labels.
To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.
Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Filter containers by using Kubernetes namespaces, pod names, and container names

Collect text logs from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.

Obtain different levels of Kubernetes information.
- Obtain information about pods.
- Obtain information about namespaces.
Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Filter containers by using Kubernetes labels

Collect text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.

Obtain Kubernetes labels.
Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Collect multi-line logs

Collect multi-line text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo. Set the Mode parameter to Simple Mode - Multi-line.

Obtain Kubernetes labels.
Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to collect multi-line logs in simple mode, see Collect logs in simple mode.

Default fields

The following table describes the fields that are included by default in each container text log.


Log field	Description
_image_name_	The name of the image.
_container_name_	The name of the container.
_pod_name_	The name of the pod.
_namespace_	The namespace of the pod.
_pod_uid_	The unique identifier of the pod.
_container_ip_	The IP address of the pod.

Troubleshooting

If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:

What do I do if an error occurs when I use Logtail to collect logs from containers?