This topic describes how to configure Logtail in the console to collect Kubernetes stdout and stderr logs in the DaemonSet mode.

Prerequisites

The Helm package alibaba-log-controller is installed. For more information, see Install Logtail.

Features

Logtail can collect container stdout and stderr logs and upload these logs together with container metadata to Log Service. The stdout and stderr log collection provides multiple features by supporting the following operations:

  • Use labels to specify containers for log collection.
  • Use labels to exclude containers for log collection.
  • Use environment variables to specify containers for log collection.
  • Use environment variables to exclude containers for log collection.
  • Supports multi-line logs such as Java Stack logs.
  • Supports automatic labeling for Docker container log data.
  • Supports automatic labeling for Kubernetes container log data.
Note
  • The preceding labels are the labels retrieved by using the docker inspect command. They are not the labels configured in a Kubernetes cluster.
  • The preceding environment variables are the environment variables configured for launching containers.

Implementation

The Logtail container uses a Unix domain socket to communicate with the Docker daemon. The container queries other Docker containers that are running and locates the containers whose logs are to be collected based on the labels and environment variables configured for log collection. Logtail collects logs of specified containers by using the docker logs command.

Logtail periodically saves the checkpoint to the checkpoint file when collecting stdout and stderr logs of a container. If Logtail is restarted, it collects logs from the last saved checkpoint.Implementation

Limits

  • This feature only applies to Logtail version 0.16.0 and later running on Linux. For more information about Logtail versions and version upgrades, see Install Logtail in Linux.
  • By default, Logtail accesses the docker daemon through the /var/run/docker.sock socket. Ensure that a Unix domain socket is available and the Logtail container has permissions to access the docker daemon.
  • Multi-line log entries: To ensure that a multi-line log entry is not split into multiple log entries due to output latency, the last collected multi-line log entry is cached for three seconds by default. You can specify the cache time by configuring the BeginLineTimeoutMs parameter. The parameter value cannot be less than 1,000 ms. Otherwise, an error may occur.
  • Collection stop policy: When a container is stopped and Logtail detects the die event on the container, Logtail stops collecting stdout and stderr logs of the container. In this case, if a collection latency occurs, some stdout or stderr logs that are generated before the stop action may be lost.
  • Docker logging driver: The logging driver collects stdout and stderr logs only in JSON files.
  • Context: Logs of different containers collected by using a collection configuration file are in the same context by default. If you need logs of each container to be in different contexts, create a collection configuration file for each container.
  • Data processing: The collected data is contained in the content field. The data can be processed by using a common processing method. .

Configure log collection

  1. Log on to the Log Service console.
  2. On the page that appears, click Kubernetes Standard Output in the Import Data section.
  3. Select an existing project and Logstore.
    You can also click Create Now to create a project and then a Logstore. For more information, see Get started.
  4. Create a server group. After the server group is created, click Complete Installation.
    If a server group is available, click Using Existing Server Groups.
  5. Configure the server group. Click Next.
    Select a server group by moving the group from Source Server Groups to Applied Server Groups.
  6. Configure a data source. Click Next.

    Enter log collection configurations in the Plug-in Config field, as shown in the following example:

    {
     "inputs": [
         {
             "type": "service_docker_stdout",
             "detail": {
                 "Stdout": true,
                 "Stderr": true,
                 "IncludeLabel": {
                     "io.kubernetes.container.name": "nginx"
                 },
                 "ExcludeLabel": {
                     "io.kubernetes.container.name": "nginx-ingress-controller"
                 },
                 "IncludeEnv": {
                     "NGINX_SERVICE_PORT": "80"
                 },
                 "ExcludeEnv": {
                     "POD_NAMESPACE": "kube-system"
                 }
             }
         }
     ]
    }

    The type of input data sources is service_docker_stdout.

    Configuration item Type Required Description
    IncludeLabel The parameter value is a map where keys and values are strings. Yes The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose label keys match the specified keys are collected.
    Note Key-value pairs are associated by the OR operator. If a label key-value pair of a container matches one of the specified key-value pairs, logs of the container are collected.
    ExcludeLabel The parameter value is a map where keys and values are strings. No The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose label keys match the specified keys are not collected.
    Note Key-value pairs are associated by the OR operator. If a label key-value pair of a container matches one of the specified key-value pairs, logs of the container are not collected.
    IncludeEnv The parameter value is a map where keys and values are strings. No The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose environment variable keys match the specified keys are collected.
    Note Key-value pairs are associated by the OR operator. When the environment variable of a container includes one of the key-value pairs, the container is excluded.
    ExcludeEnv The parameter value is a map where keys and values are strings. No The parameter value is an empty map by default, which indicates that logs from all containers are collected. If keys are not empty and values are empty, logs of containers whose environment variable keys match the specified keys are not collected.
    Note Key-value pairs are associated by the OR operator. When the environment variable of a container includes one of the key-value pairs, the container is excluded.
    Stdout bool No Default value: true. If you set the value to false, stdout logs are not collected.
    Stderr bool No Default value: true. If you set the value to false, stderr logs are not collected.
    BeginLineRegex string No The regular expression used to match a line as the first line of a log entry. The parameter value is empty by default. If a line matches this regular expression, the line is considered the first line of a new log entry. Otherwise, the line is considered a part of the last log entry.
    BeginLineTimeoutMs int No The timeout period for the regular expression to match a line. Default value: 3000. Unit: ms. If no new log entry appears within 3 seconds, the last log entry is uploaded.
    BeginLineCheckLength int No The length of the first line of a log entry that matches the regular expression. Default value: 10 × 1,024. Unit: bytes. You can set this parameter to check whether the beginning part of a line can match the regular expression. This improves matching efficiency.
    MaxLogSize int No The maximum size of a log entry. Default value: 512 × 1,024. Unit: bytes. If the size of a log entry exceeds the parameter value, the log entry is uploaded.
    Note
    • The preceding IncludeLabel and ExcludeLabel parameters are included in the label information retrieved by using the docker inspect command.
    • A namespace and a container name in Kubernetes can be mapped to Docker labels. The LabelKey parameter corresponding to a namespace is io.kubernetes.pod.namespace. The LabelKey parameter corresponding to a container name is io.kubernetes.container.name. For example, the namespace of a pod that you created is backend-prod and the container name is worker-server. In this case, you can configure the whitelist label io.kubernetes.pod.namespace : backend-prod to collect logs of containers in the pod, including the worker-server container. You can also configure the whitelist label io.kubernetes.container.name : worker-server to collect the logs of the worker-server container.
    • In a Kubernetes cluster, we recommend that you specify only the io.kubernetes.pod.namespace and io.kubernetes.container.name labels. If the two labels cannot satisfy your business needs, you can configure the IncludeEnv and ExcludeEnv parameters.
  7. Configure search and analytics statements. Click Next.
    Indexes are created by default. You can modify the indexes based on your needs.

Default fields

The following table lists the fields that are uploaded by default for each Kubernetes log entry.
Field name Description
_time_ The data upload time, for example, 2018-02-02T02:18:41.979147844Z.
_source_ The type of input data sources. Valid values: stdout and stderr.
_image_name_ The name of an image.
_container_name_ The name of a container.
_pod_name_ The name of a pod.
_namespace_ The namespace where a pod is located.
_pod_uid_ The unique identifier of a pod.
_container_id_ The IP address assigned to a pod.

Configuration example of single-line log collection

  • Environment variable configuration

    Collect stdout and stderr logs of containers that match the following conditions: Environment variables include NGINX_PORT_80_TCP_PORT=80 but exclude POD_NAMESPACE=kube-system.

    Environment variable configuration
    The following script shows how to configure data collection based on the preceding conditions:
    {
        "inputs": [
            {
                "type": "service_docker_stdout",
                "detail": {
                    "Stdout": true,
                    "Stderr": true,
                    "IncludeEnv": {
                        "NGINX_PORT_80_TCP_PORT": "80"
                    },
                    "ExcludeEnv": {
                        "POD_NAMESPACE": "kube-system"
                    }
                }
            }
        ]
    }
  • Label configuration

    Collect the stdout and stderr logs of containers whose labels include io.kubernetes.container.name=nginx but exclude type=pre.

    Label configuration
    The following script shows how to configure data collection based on the preceding conditions:
    {
        "inputs": [
            {
                "type": "service_docker_stdout",
                "detail": {
                    "Stdout": true,
                    "Stderr": true,
                    "IncludeLabel": {
                        "io.kubernetes.container.name": "nginx"
                    },
                    "ExcludeLabel": {
                        "type": "pre"
                    }
                }
            }
        ]
    }

Configuration example of multi-line log collection

Configuring multi-line log collection is important for the collection of Java exception stack logs. The following section describes how to collect stdout and stderr logs of standard Java applications.
  • Sample log entry
    2018-02-03 14:18:41.968  INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start
    2018-02-03 14:18:41.969 ERROR [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : java.lang.NullPointerException
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    ...
    2018-02-03 14:18:41.968  INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done
  • Log collection configuration

    Collect logs of containers that match the following conditions: Labels include app=monitor and the specified first bytes of a line is of a fixed-format date type. The first 10 bytes of each line are checked in the following example to improve matching efficiency.

    {
    "inputs": [
      {
        "detail": {
          "BeginLineCheckLength": 10,
          "BeginLineRegex": "\\d+-\\d+-\\d+. *",
          "IncludeLabel": {
            "app": "monitor"
          }
        },
        "type": "service_docker_stdout"
      }
    ]
    }

Data processing example

Logtail can process the collected Docker stdout and stderr logs by using a . You can use regular expressions to parse log entries into the time, module, thread, class, and info fields.
  • Log collection configuration
    Collect logs of containers that match the following conditions: Labels include app=monitor and the specified first bytes of a line is of a fixed-format date type. The first 10 bytes of each line are checked in the following example to improve matching efficiency.
    {
    "inputs": [
      {
        "detail": {
          "BeginLineCheckLength": 10,
          "BeginLineRegex": "\\d+-\\d+-\\d+. *",
          "IncludeLabel": {
            "app": "monitor"
          }
        },
        "type": "service_docker_stdout"
      }
    ],
    "processors": [
        {
            "type": "processor_regex",
            "detail": {
                "SourceKey": "content",
                "Regex": "(\\d+-\\d+-\\d+ \\d+:\\d+:\\d+\\.\\d+)\\s+(\\w+)\\s+\\[([^]]+)]\\s+\\[([^]]+)]\\s+:\\s+([\\s\\S]*)",
                "Keys": [
                    "time",
                    "module",
                    "thread",
                    "class",
                    "info"
                ],
                "NoKeyError": true,
                "NoMatchError": true,
                "KeepSource": false
            }
        }
    ]
    }
  • Sample output

    The collected 2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done log entry is processed as follows:

    __tag__:__hostname__:logtail-dfgef
    _container_name_:monitor
    _image_name_:registry.cn-hangzhou.aliyuncs.xxxxxxxxxxxxxxx
    _namespace_:default
    _pod_name_:monitor-6f54bd5d74-rtzc7
    _pod_uid_:7f012b72-04c7-11e8-84aa-00163f00c369
    _source_:stdout
    _time_:2018-02-02T14:18:41.979147844Z
    time:2018-02-02 02:18:41.968
    level:INFO
    module:spring-cloud-monitor
    thread:nio-8080-exec-4
    class:c.g.s.web.controller.DemoController
    message:service start done