Use the console to collect Kubernetes stdout and stderr logs in the DaemonSet mode

Features

Logtail can collect container stdout and stderr logs and upload these logs together with container metadata to Log Service. The stdout and stderr log collection provides multiple features by supporting the following operations:

• Use labels to specify containers for log collection.
• Use labels to exclude containers for log collection.
• Use environment variables to specify containers for log collection.
• Use environment variables to exclude containers for log collection.
• Supports multi-line logs such as Java Stack logs.
• Supports automatic labeling for Docker container log data.
• Supports automatic labeling for Kubernetes container log data.

Implementation

The Logtail container uses a Unix domain socket to communicate with the Docker daemon. The container queries other Docker containers that are running and locates the containers whose logs are to be collected based on the labels and environment variables configured for log collection. Logtail collects logs of specified containers by using the docker logs command.

Logtail periodically saves the checkpoint to the checkpoint file when collecting stdout and stderr logs of a container. If Logtail is restarted, it collects logs from the last saved checkpoint.

Limits

• This feature only applies to Logtail version 0.16.0 and later running on Linux. For more information about Logtail versions and version upgrades, see Install Logtail in Linux.
• By default, Logtail accesses the docker daemon through the /var/run/docker.sock socket. Ensure that a Unix domain socket is available and the Logtail container has permissions to access the docker daemon.
• Multi-line log entries: To ensure that a multi-line log entry is not split into multiple log entries due to output latency, the last collected multi-line log entry is cached for three seconds by default. You can specify the cache time by configuring the BeginLineTimeoutMs parameter. The parameter value cannot be less than 1,000 ms. Otherwise, an error may occur.
• Collection stop policy: When a container is stopped and Logtail detects the die event on the container, Logtail stops collecting stdout and stderr logs of the container. In this case, if a collection latency occurs, some stdout or stderr logs that are generated before the stop action may be lost.
• Docker logging driver: The logging driver collects stdout and stderr logs only in JSON files.
• Context: Logs of different containers collected by using a collection configuration file are in the same context by default. If you need logs of each container to be in different contexts, create a collection configuration file for each container.
• Data processing: The collected data is contained in the content field. The data can be processed by using a common processing method. .

Configure log collection

1. Log on to the Log Service console.
2. On the page that appears, click Kubernetes Standard Output in the Import Data section.
3. Select an existing project and Logstore.
   You can also click Create Now to create a project and then a Logstore. For more information, see Get started.
4. Create a server group. After the server group is created, click Complete Installation.
   If a server group is available, click Using Existing Server Groups.
5. Configure the server group. Click Next.
   Select a server group by moving the group from Source Server Groups to Applied Server Groups.
6. Configure a data source. Click Next.

   Enter log collection configurations in the Plug-in Config field, as shown in the following example:

   {
    "inputs": [
        {
            "type": "service_docker_stdout",
            "detail": {
                "Stdout": true,
                "Stderr": true,
                "IncludeLabel": {
                    "io.kubernetes.container.name": "nginx"
                },
                "ExcludeLabel": {
                    "io.kubernetes.container.name": "nginx-ingress-controller"
                },
                "IncludeEnv": {
                    "NGINX_SERVICE_PORT": "80"
                },
                "ExcludeEnv": {
                    "POD_NAMESPACE": "kube-system"
                }
            }
        }
    ]
   }

   The type of input data sources is service_docker_stdout.

   Configuration item	Type	Required	Description
   IncludeLabel	The parameter value is a map where keys and values are strings.	Yes	The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose label keys match the specified keys are collected. Note Key-value pairs are associated by the OR operator. If a label key-value pair of a container matches one of the specified key-value pairs, logs of the container are collected.
   ExcludeLabel	The parameter value is a map where keys and values are strings.	No	The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose label keys match the specified keys are not collected. Note Key-value pairs are associated by the OR operator. If a label key-value pair of a container matches one of the specified key-value pairs, logs of the container are not collected.
   IncludeEnv	The parameter value is a map where keys and values are strings.	No	The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose environment variable keys match the specified keys are collected. Note Key-value pairs are associated by the OR operator. When the environment variable of a container includes one of the key-value pairs, the container is excluded.
   ExcludeEnv	The parameter value is a map where keys and values are strings.	No	The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose environment variable keys match the specified keys are not collected. Note Key-value pairs are associated by the OR operator. When the environment variable of a container includes one of the key-value pairs, the container is excluded.
   Stdout	bool	No	Default value: true. If you set the value to false, stdout logs are not collected.
   Stderr	bool	No	Default value: true. If you set the value to false, stderr logs are not collected.
   BeginLineRegex	string	No	The regular expression used to match a line as the first line of a log entry. The parameter value is empty by default. If a line matches this regular expression, the line is considered the first line of a new log entry. Otherwise, the line is considered a part of the last log entry.
   BeginLineTimeoutMs	int	No	The timeout period for the regular expression to match a line. Default value: 3000. Unit: ms. If no new log entry appears within 3 seconds, the last log entry is uploaded.
   BeginLineCheckLength	int	No	The length of the first line of a log entry that matches the regular expression. Default value: 10 × 1,024. Unit: bytes. You can set this parameter to check whether the beginning part of a line can match the regular expression. This improves matching efficiency.
   MaxLogSize	int	No	The maximum size of a log entry. Default value: 512 × 1,024. Unit: bytes. If the size of a log entry exceeds the parameter value, the log entry is uploaded.

   Note

   • The preceding IncludeLabel and ExcludeLabel parameters are included in the label information retrieved by using the docker inspect command.
   • A namespace and a container name in Kubernetes can be mapped to Docker labels. The LabelKey parameter corresponding to a namespace is io.kubernetes.pod.namespace. The LabelKey parameter corresponding to a container name is io.kubernetes.container.name. For example, the namespace of a pod that you created is backend-prod and the container name is worker-server. In this case, you can configure the whitelist label io.kubernetes.pod.namespace : backend-prod to collect logs of containers in the pod, including the worker-server container. You can also configure the whitelist label io.kubernetes.container.name : worker-server to collect the logs of the worker-server container.
   • In a Kubernetes cluster, we recommend that you specify only the io.kubernetes.pod.namespace and io.kubernetes.container.name labels. If the two labels cannot satisfy your business needs, you can configure the IncludeEnv and ExcludeEnv parameters.
7. Configure search and analytics statements. Click Next.
   Indexes are created by default. You can modify the indexes based on your needs.

Default fields

Configuration example of single-line log collection

• Environment variable configuration

  Collect stdout and stderr logs of containers that match the following conditions: Environment variables include NGINX_PORT_80_TCP_PORT=80 but exclude POD_NAMESPACE=kube-system.

  
  The following script shows how to configure data collection based on the preceding conditions:

  {
      "inputs": [
          {
              "type": "service_docker_stdout",
              "detail": {
                  "Stdout": true,
                  "Stderr": true,
                  "IncludeEnv": {
                      "NGINX_PORT_80_TCP_PORT": "80"
                  },
                  "ExcludeEnv": {
                      "POD_NAMESPACE": "kube-system"
                  }
              }
          }
      ]
  }

• Label configuration

  Collect the stdout and stderr logs of containers whose labels include io.kubernetes.container.name=nginx but exclude type=pre.

  
  The following script shows how to configure data collection based on the preceding conditions:

  {
      "inputs": [
          {
              "type": "service_docker_stdout",
              "detail": {
                  "Stdout": true,
                  "Stderr": true,
                  "IncludeLabel": {
                      "io.kubernetes.container.name": "nginx"
                  },
                  "ExcludeLabel": {
                      "type": "pre"
                  }
              }
          }
      ]
  }

Configuration example of multi-line log collection

Data processing example