This topic describes how to configure Logtail in the Log Service console to collect Kubernetes stdout and stderr logs in DaemonSet mode.
Prerequisites
Features
Logtail can collect and upload container stdout and stderr logs with container metadata to Log Service. The collection of Kubernetes stdout and stderr logs has the following features:
- Collects stdout and stderr logs in real time.
- Uses labels to specify containers for log collection.
- Uses labels to exclude containers from log collection.
- Uses environment variables to specify containers for log collection.
- Uses environment variables to exclude containers from log collection.
- Supports multi-line logs such as Java Stack logs.
- Supports automatic labeling for Docker container logs.
- Supports automatic labeling for Kubernetes container logs.
- The preceding labels are retrieved by using the docker inspect command. These labels are not the labels that are specified in a Kubernetes cluster.
- The preceding environment variables are the environment variables that you have specified to start containers.
Implementation
A Logtail container uses a UNIX domain socket to communicate with the Docker daemon. The Logtail container queries all Docker containers and locates the specified Docker containers based on specified labels and environment variables. Logtail collects the logs of the specified Docker containers by using the docker logs command.

Limits
- Logtail version: Only Logtail 0.16.0 or later that runs on Linux can be used to collect stdout and stderr logs. For more information about Logtail versions and version updates, see Install Logtail in Linux.
- Permissions: By default, Logtail uses the
/var/run/docker.sock
socket to access the docker daemon. Make sure that a UNIX domain socket is available and the Logtail container has permissions to access the docker daemon. - Multi-line log entries: To ensure that a multi-line log entry is not split into multiple
log entries due to output latency, the last collected multi-line log entry is cached
for 3 seconds by default. You can set the cache time by specifying the
BeginLineTimeoutMs
parameter. The parameter value cannot be less than 1,000 ms. Otherwise, an error may occur. - Stop policy: When a container is stopped and Logtail detects the
die
event on the container, Logtail stops collecting stdout and stderr logs of the container. In this case, if a collection delay occurs, some stdout or stderr logs that are generated before the stop action may be lost. - Docker logging driver: The logging driver collects stdout and stderr logs only in JSON files.
- Context: By default, logs that are collected from different containers by using a collection configuration file are in the same context. If you want the logs of each container to be in different contexts, create a Logtail configuration file for each container.
- Data processing: The collected data is contained in the
content
field. The data can be processed by using a common processing method.
Procedure
Default fields
Field name | Description |
---|---|
_time_ |
The data upload time, for example, 2018-02-02T02:18:41.979147844Z .
|
_source_ |
The type of input data sources. Valid values: stdout and stderr. |
_image_name_ |
The name of an image. |
_container_name_ |
The name of a container. |
_pod_name_ |
The name of a pod. |
_namespace_ |
The namespace where a pod resides. |
_pod_uid_ |
The unique identifier of a pod. |
_container_id_ |
The IP address of a pod. |
Configuration examples of single-line log collection
- Configure environment variables
Collect the stdout and stderr logs of the containers that meet the following conditions: Environment variables include
NGINX_PORT_80_TCP_PORT=80
and excludePOD_NAMESPACE=kube-system
.The following script shows the log collection configurations:{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeEnv": { "NGINX_PORT_80_TCP_PORT": "80" }, "ExcludeEnv": { "POD_NAMESPACE": "kube-system" } } } ] }
- Configure labels
Collect the stdout and stderr logs of the containers that meet the following conditions: The container labels include
io.kubernetes.container.name=nginx
and excludetype=pre
.The following script shows the label configurations:{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeLabel": { "io.kubernetes.container.name": "nginx" }, "ExcludeLabel": { "type": "pre" } } } ] }
Configuration examples of multi-line log collection
- Sample log entry
2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start 2018-02-03 14:18:41.969 ERROR [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : java.lang.NullPointerException at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ... 2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done
- Log collection configuration
Collect the logs of the containers that meet the following conditions: The container labels include
app=monitor
and the specified first bytes of a line is of a fixed-format date type. To improve match efficiency, only the first 10 bytes of each line are checked.{ "inputs": [ { "detail": { "BeginLineCheckLength": 10, "BeginLineRegex": "\\d+-\\d+-\\d+. *", "IncludeLabel": { "app": "monitor" } }, "type": "service_docker_stdout" } ] }
Data processing examples
- Collect the logs of the containers that meet the following conditions: The container
labels include
app=monitor
and the specified first bytes of a line is of a fixed-format date type. To improve match efficiency, only the first 10 bytes of each line are checked. Regular expressions are used to parse logs into the values of time, level, module, thread, and message. The following script shows the configurations of log collection and data processing:{ "inputs": [ { "detail": { "BeginLineCheckLength": 10, "BeginLineRegex": "\\d+-\\d+-\\d+. *", "IncludeLabel": { "app": "monitor" } }, "type": "service_docker_stdout" } ], "processors": [ { "type": "processor_regex", "detail": { "SourceKey": "content", "Regex": "(\\d+-\\d+-\\d+ \\d+:\\d+:\\d+\\.\\d+)\\s+(\\w+)\\s+\\[([^]]+)]\\s+\\[([^]]+)]\\s+:\\s+([\\s\\S]*)", "Keys": [ "time", "level", "module", "thread", "message" ], "NoKeyError": true, "NoMatchError": true, "KeepSource": false } } ] }
The collected2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done
log entry is processed, as shown in the following script:__tag__:__hostname__:logtail-dfgef _container_name_:monitor _image_name_:registry.cn-hangzhou.aliyuncs.xxxxxxxxxxxxxxx _namespace_:default _pod_name_:monitor-6f54bd5d74-rtzc7 _pod_uid_:7f012b72-04c7-11e8-84aa-00163f00c369 _source_:stdout _time_:2018-02-02T14:18:41.979147844Z time:2018-02-02 02:18:41.968 level:INFO module:spring-cloud-monitor thread:nio-8080-exec-4 class:c.g.s.web.controller.DemoController message:service start done
- Collect the JSON logs of the containers that meet the following conditions: The container
labels include
app=monitor
. The following script shows the configurations of log collection and data processing:{ "inputs": [ { "detail": { "IncludeLabel": { "app": "monitor" } }, "type": "service_docker_stdout" } ], "processors": [ { "type": "processor_json", "detail": { "SourceKey": "content", "NoKeyError":true, "KeepSource": false } } ] }