This topic describes how to create a Logtail configuration in the Log Service console and use the Logtail configuration to collect container text logs in DaemonSet mode.
Prerequisites
- The Logtail component is installed. For more information, see Install Logtail components in a Kubernetes cluster.
- A Logstore is created in the project that you use to install the Logtail component. For more information, see Create a Logstore.
- The container from which you want to collect logs continuously generates logs. Important Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.
Features
- Allows you to specify a log file path in a container when you collect text logs. You do not need to manually map the log file path to a path on the host.
- Uses the container label whitelist to specify containers from which logs are collected.
- Uses the container label blacklist to specify containers from which logs are not collected.
- Uses the environment variable whitelist to specify containers from which logs are collected.
- Uses the environment variable blacklist to specify containers from which logs are not collected.
- Collects multi-line logs. For example, Logtail can collect Java stack logs.
- Automatically associates container metadata that needs to be uploaded together with the collected container logs. The metadata includes container names, image names, pod names, namespaces, and environment variables.
- If a container runs in a Kubernetes cluster, Logtail also supports the following features:
- Uses Kubernetes namespaces, pod names, and container names to specify containers from which logs are collected.
- Uses the Kubernetes label whitelist to specify containers from which logs are collected.
- Uses the Kubernetes label blacklist to specify containers from which logs are not collected.
- Automatically associates Kubernetes labels that need to be uploaded together with the collected container logs.
Limits
- If Logtail detects the
die
event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost. - For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.
If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode. For more information, see Use CRDs to collect container text logs in Sidecar mode and Use the Log Service console to collect container text logs in Sidecar mode.
- Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
- If a volume is mounted to the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.
For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.
- By default, Kubernetes mounts the root directory of the host to the
/logtail_host
directory of the Logtail container. If you want to collect text logs from the host, you must specify/logtail_host
as the prefix of the log file path.For example, if you want to collect logs from the
/home/logs/app_log/
directory of the host, you must specify/logtail_host/home/logs/app_log/
as the log file path. - Logtail collects data from containers that use the Docker engine or containerd engine.
- Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
Create a Logtail configuration
- Log on to the Log Service console.
- In the Import Data section, click Kubernetes - Object.
- Select a project and a Logstore. Then, click Next. In this example, select the project that you use to install the Logtail component and the Logstore that you create.
- Click Use Existing Machine Groups. After you install the Logtail component, Log Service automatically creates a machine group named
k8s-group-${your_k8s_cluster_id}
. You can select this machine group. - Select the
k8s-group-${your_k8s_cluster_id}
machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.Important If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats? - Configure the parameters for the Logtail configuration and click Next.
- Preview data, configure indexes, and then click Next. By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
- Click Log Query. You are redirected to the query and analysis page of your Logstore. You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.
Configuration examples
Filter containers based on the environment variable whitelist and the environment variable blacklist
Collect text logs from the containers whose environment variable configurations include NGINX_SERVICE_PORT=80
but exclude POD_NAMESPACE=kube-system
. The log file path is /var/log/nginx/access.log
. The logs are collected in simple mode.
- Obtain environment variables.
To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.
- Create a Logtail configuration. The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Filter containers based on the container label whitelist and the container label blacklist
Collect text logs from the containers whose container label is io.kubernetes.container.name=nginx
. The log file path is /var/log/nginx/access.log
. The logs are collected in simple mode.
- Obtain container labels.
To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.
- Create a Logtail configuration. The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Filter containers by using Kubernetes namespaces, pod names, and container names
Collect text logs from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.
- Obtain different levels of Kubernetes information.
- Obtain information about pods.
- Obtain information about namespaces.
- Obtain information about pods.
- Create a Logtail configuration.
The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Filter containers by using Kubernetes labels
Collect text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.
- Obtain Kubernetes labels.
- Create a Logtail configuration. The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.
Collect multi-line logs
Collect multi-line text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo. Set the Mode parameter to Simple Mode - Multi-line.
- Obtain Kubernetes labels.
- Create a Logtail configuration. The following figure shows an example of a Logtail configuration. For more information about how to collect multi-line logs in simple mode, see Collect logs in simple mode.
Default fields
Log field | Description |
---|---|
_image_name_ | The name of the image. |
_container_name_ | The name of the container. |
_pod_name_ | The name of the pod. |
_namespace_ | The namespace of the pod. |
_pod_uid_ | The unique identifier of the pod. |
_container_ip_ | The IP address of the pod. |
Troubleshooting
If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:
What do I do if an error occurs when I use Logtail to collect logs from containers?