This topic describes how to create a Logtail configuration in the Log Service console and use the Logtail configuration to collect container stdout and stderr in DaemonSet mode.
Prerequisites
- The Logtail component is installed. For more information, see Install Logtail components.
- A Logstore is created in the project that you use to install the Logtail component. For more information, see Create a Logstore.
Features
Logtail can collect container stdout and stderr, and then upload the stdout and stderr together with container metadata to Log Service. Logtail supports the following features:
- Collects stdout and stderr.
- Uses the container label whitelist to specify containers from which stdout and stderr are collected.
- Uses the container label blacklist to specify containers from which stdout and stderr are not collected.
- Uses the environment variable whitelist to specify containers from which stdout and stderr are collected.
- Uses the environment variable blacklist to specify containers from which stdout and stderr are not collected.
- Collects multi-line logs. For example, Logtail can collect Java stack logs.
- Automatically associates container metadata that needs to be uploaded together with the collected container stdout and stderr. The metadata includes container names, image names, pod names, namespaces, and environment variables.
- If a container runs in a Kubernetes cluster, Logtail also supports the following features:
- Uses Kubernetes namespaces, pod names, and container names to specify containers from which stdout and stderr are collected.
- Uses the Kubernetes label whitelist to specify containers from which stdout and stderr are collected.
- Uses the Kubernetes label blacklist to specify containers from which stdout and stderr are not collected.
- Automatically associates Kubernetes labels that need to be uploaded together with the collected container stdout and stderr.
Implementation
Logtail communicates with the domain socket of Docker. Logtail queries all Docker
containers and identifies the containers from which stdout and stderr are collected
by using the specified labels and environment variables. Logtail runs the docker logs command to collect logs from the specified containers.
Limits
- You can use the Log Service console to collect stdout and stderr in DaemonSet mode only if Logtail runs V0.16.0 or later and runs on Linux. For more information about Logtail versions and version updates, see Install Logtail on a Linux server.
- Logtail collects data from containers that use the Docker engine or containerd engine.
- Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- By default, the last multi-line log that is collected by Logtail is cached for 3 seconds.
This prevents the multi-line log from being split into multiple logs due to output
latency. You can change the cache time by changing the value of the
BeginLineTimeoutMsparameter. We recommend that you do not specify a value less than 1000 with millisecond precision. If you specify a value that is less than 1000, an error may occur. - If Logtail detects the
dieevent on a container that is stopped, Logtail no longer collects stdout and stderr from the container. If collection latency occurs, some stdout and stderr that are collected before the container is stopped may be lost. - The logging driver collects stdout and stderr only in the JSON format from containers that use the Docker engine.
- By default, stdout and stderr that are collected from different containers by using the same Logtail configuration have the same context. If you want to specify a different context for the stdout and stderr of each container, you must create a Logtail configuration for each container.
- By default, the collected data is stored in the
contentfield. Logtail can process the collected data. For more information, see Use Logtail plug-ins to process data.
Create a Logtail configuration
Examples of Logtail configurations for single-line logs
Example 1: Filter containers based on the environment variable whitelist and the environment variable blacklist
Collect stdout and stderr from the containers whose environment variable configurations
include NGINX_SERVICE_PORT=80 but exclude POD_NAMESPACE=kube-system.
- Obtain environment variables.
To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.
- Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeEnv": { "NGINX_SERVICE_PORT": "80" }, "ExcludeEnv": { "POD_NAMESPACE": "kube-system" } } } ] }
Example 2: Filter containers based on the container label whitelist and the container label blacklist
Collect stdout and stderr from the containers whose container label is io.kubernetes.container.name=nginx.
- Obtain container labels.
To view the labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.
- Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeLabel": { "io.kubernetes.container.name": "nginx" } } } ] }
Example 3: Filter containers by using Kubernetes namespaces, pod names, and container names
Collect stdout and stderr from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.
- Obtain different levels of Kubernetes information.
- Obtain information about pods.
- Obtain information about namespaces.
- Obtain information about pods.
- Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "K8sNamespaceRegex":"^(default)$", "K8sPodRegex":"^(nginx-log-demo.*)$", "K8sContainerRegex":"^(nginx-log-demo-0)$" } } ] }
Example 4: Filter containers by using Kubernetes labels
Collect stdout and stderr from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.
- Obtain Kubernetes labels.
- Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeK8sLabel":{ "job-name":"^(nginx-log-demo.*)$" } } } ] }
Examples of Logtail configurations for multi-line logs
- Sample logs
2021-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start 2021-02-03 14:18:41.969 ERROR [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : java.lang.NullPointerException at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ... 2021-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done - Logtail configuration
Collect the Java exception stack logs of the containers whose container label is
app=monitor. The Java exception stack logs start with a date that is in a fixed format. Logtail matches only the first 10 bytes of each line to improve match efficiency. After the logs are collected to Log Service, Log Service uses regular expressions to parse the logs into fields such as time, level, module, thread, and message.- inputs is required and is used to configure the log collection settings for the Logtail
configuration. You must configure inputs based on your data source.
Note You can configure only one type of data source in inputs.
- processors is optional and is used to configure the log processing settings for the Logtail configuration. You can specify one or more processing methods. For more information, see Use Logtail plug-ins to process data.
{ "inputs": [ { "detail": { "BeginLineCheckLength": 10, "BeginLineRegex": "\\d+-\\d+-\\d+.*", "IncludeLabel": { "app": "monitor" } }, "type": "service_docker_stdout" } ], "processors": [ { "type": "processor_regex", "detail": { "SourceKey": "content", "Regex": "(\\d+-\\d+-\\d+ \\d+:\\d+:\\d+\\.\\d+)\\s+(\\w+)\\s+\\[([^]]+)]\\s+\\[([^]]+)]\\s+([\\s\\S]*)", "Keys": [ "time", "level", "module", "thread", "message" ], "NoKeyError": true, "NoMatchError": true, "KeepSource": false } } ] } - inputs is required and is used to configure the log collection settings for the Logtail
configuration. You must configure inputs based on your data source.
- Parsed logs
For example, if the collected log is
2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done, the log is parsed into the following fields:__tag__:__hostname__:logtail-dfgef _container_name_:monitor _image_name_:example.com-hangzhou.aliyuncs.xxxxxxxxxxxxxxx _namespace_:default _pod_name_:monitor-6f54bd5d74-rtzc7 _pod_uid_:7f012b72-04c7-11e8-84aa-00163f00c369 _source_:stdout _time_:2018-02-02T14:18:41.979147844Z time:2018-02-02 02:18:41.968 level:INFO module:spring-cloud-monitor thread:nio-8080-exec-4 class:c.g.s.web.controller.DemoController message:service start done
Log fields
| Log field | Description |
|---|---|
| _time_ | The time at which the data is uploaded. Example: 2021-02-02T02:18:41.979147844Z.
|
| _source_ | The type of the data source. Valid values: stdout and stderr. |
| _image_name_ | The name of the image. |
| _container_name_ | The name of the container. |
| _pod_name_ | The name of the pod. |
| _namespace_ | The namespace of the pod. |
| _pod_uid_ | The unique identifier of the pod. |
| _container_id_ | The IP address of the pod. |