Collect standard output from a Kubernetes container by using DaemonSet (new version) - Simple Log Service

Logtail input configuration now supports the collection of container standard output. This topic introduces the new version of standard output collection configuration.

Features

Logtail can collect container stdout and stderr, uploading them along with container metadata to Simple Log Service. Logtail supports the following features:

Collects stdout and stderr logs.
Uses container label whitelist to specify containers for log collection.
Uses container label blacklist to exclude containers from log collection.
Uses environment variable whitelist to specify containers for log collection.
Uses environment variable blacklist to exclude containers from log collection.
Collects multi-line logs, such as Java stack traces.
Automatically associates metadata required for upload with the collected container logs, including container names, image names, pod names, namespaces, and environment variables.
For containers running in a Kubernetes cluster, Logtail also supports:
- Using Kubernetes namespaces, pod names, and container names to specify containers for log collection.
- Using Kubernetes label whitelist to specify containers for log collection.
- Using Kubernetes label blacklist to exclude containers from log collection.
- Automatically associating Kubernetes labels required for upload with the collected container logs.

Limitations

The new version of the standard output collection plugin currently only supports the Linux operating system and Logtail version 2.1.5 or later. It does not support collecting standard output logs through the host method. For information on checking and upgrading the version, see Install Logtail components (Alibaba Cloud Kubernetes cluster) and Install Logtail components (self-managed Kubernetes cluster).
Logtail collects data from containers using the Docker engine or containerd engine. The access paths are:
- Docker: Logtail requires access to Docker via /run/docker.sock. Ensure this path is present and has the necessary access permissions.
- Containerd: Logtail requires access to Containerd via /run/containerd/containerd.sock. Ensure this path is present and has the necessary access permissions.
The maximum size of each log read is 524288 bytes (512 KB) by default, with a maximum value of 8388608 bytes (8 MB). If your single log exceeds 524288 bytes, you can add the environment variable max_read_buffer_size to the Logtail container to modify it.
Collection stop policy: When a container stops, Logtail ceases to collect its standard output upon detecting the container's die event. In the event of collection latency, some stdout and stderr logs produced prior to the container's cessation may be lost.
Docker engine limitation: The logging driver only collects stdout and stderr logs in JSON format from containers using the Docker engine.
Data processing: By default, the collected data is stored in the content field. Logtail facilitates data processing for standard output from containers. For more information, see how to use Logtail plug-ins for data processing. The latest version of the standard output collection plug-in allows for the integration of native processing plug-ins, which are high-performance and support multithreading, with extended plug-ins.
The CRD method supports AliyunPipelineConfig.
The API method supports CreateLogtailPipelineConfig - Create Logtail pipeline configuration, UpdateLogtailPipelineConfig - Update Logtail pipeline configuration, and other pipeline configurations.

Advantages

The new version of standard output collection offers several advantages over the old version:

Type	Advantages of the new standard output plugin (C++ version)
Significant performance improvement	Reconstructed with C++, performance improved by more than 100% compared to the old version of the standard output plugin.
	Supports native plugin data processing, multithreaded parallel processing, and fully utilizes system resources.
	Flexible combination of native plugins and Go plugins.
Higher reliability	Supports standard output log rotation queue. The log collection mechanism and file collection mechanism are unified, providing high reliability in scenarios where standard output logs rotate quickly.
Lower resource consumption	CPU utilization reduced by 20%.
Lower resource consumption	Memory usage reduced by 20%.
Operational consistency	The parameters of the new standard output collection plugin and the file collection plugin are unified.
Operational consistency	Container metadata field names and tag log storage locations are unified with the file collection scenario. Consumers only need to maintain the same processing logic.

Configuration instructions

Form configuration (console configuration)

Parameter	Description
Standard Output	After you enable Standard Output, Logtail collects container standard output.
Standard Error	After you enable Standard Error, Logtail collects container standard error.
Allow Multiple Collections Of Standard Output	By default, the standard output logs of a container can match only one Logtail new standard output collection configuration. If the standard output needs to be collected by multiple new standard output collection configurations, you need to enable the Allow Multiple Collections Of Standard Output switch.
Enable Container Metadata Preview	After you enable Enable Container Metadata Preview, you can view container metadata, including matched container information and full container information, after creating a Logtail configuration.
Container Filtering	Filter condition description Important Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Retrieve container labels. Environment variables are the same as the environment variables that are configured to start containers. For more information, see Retrieve container environment variables. In Kubernetes scenarios, we recommend that you use Kubernetes-level information (K8s Pod Name Regex Matching, K8s Namespace Regex Matching, K8s Container Name Regex Matching, K8s Pod Label Whitelist, etc.) for container filtering. In Kubernetes, namespaces and container names are mapped to container labels, which are `io.kubernetes.pod.namespace` and `io.kubernetes.container.name`, respectively. We recommend that you use these two container labels for container filtering. For example, if the namespace of a pod is `backend-prod` and the container name is `worker-server`, you can set the container label whitelist to `io.kubernetes.pod.namespace : backend-prod` or `io.kubernetes.container.name : worker-server` to collect logs from the container. If the above two container labels do not meet the filtering requirements, use the environment variable whitelist and blacklist for container filtering. K8s Pod Name Regex Matching Specify the containers from which logs are collected by using pod names. Regular expression matching is supported. For example, if you set it to `^(nginx-log-demo.)$`, it matches all containers under pods whose names start with nginx-log-demo. K8s Namespace Regex Matching* Specify the containers from which logs are collected by using namespace names. Regular expression matching is supported. For example, if you set it to `^(default\|nginx)$`, it matches all containers under the nginx namespace and the default namespace. K8s Container Name Regex Matching Specify the containers from which logs are collected by using container names (Kubernetes container names are defined in spec.containers). Regular expression matching is supported. For example, if you set it to `^(container-test)$`, it matches all containers named container-test. Container Label Whitelist The container label whitelist specifies the containers from which logs are collected. By default, this parameter is empty, which indicates that logs are collected from all containers. If you want to configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, logs are collected from the containers whose container labels contain the keys specified by LabelKey. If a value is specified for the LabelValue parameter, logs are collected from the containers whose container labels include LabelKey=LabelValue. By default, the value of the LabelValue parameter is a string. Logs are collected from the containers whose label values match the value of the LabelValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$`, it matches containers named nginx and cube. Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container is matched. Container Label Blacklist The container label blacklist specifies the containers from which logs are not collected. By default, this parameter is empty, which indicates that logs are collected from all containers. If you want to configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, logs are not collected from the containers whose container labels contain the keys specified by LabelKey. If a value is specified for the LabelValue parameter, logs are not collected from the containers whose container labels include LabelKey=LabelValue. By default, the value of the LabelValue parameter is a string. Logs are not collected from the containers whose label values match the value of the LabelValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$`, it matches containers named nginx and cube. Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container is filtered out. Environment Variable Whitelist The environment variable whitelist specifies the containers from which logs are collected. By default, this parameter is empty, which indicates that logs are collected from all containers. If you want to configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, logs are collected from the containers whose environment variables contain the keys specified by EnvKey. If a value is specified for the EnvValue parameter, logs are collected from the containers whose environment variables include EnvKey=EnvValue. By default, the value of the EnvValue parameter is a string. Logs are collected from the containers whose environment variable values match the value of the EnvValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80\|6379)$`, it matches containers whose service ports are 80 and 6379. Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container is matched. Environment Variable Blacklist The environment variable blacklist specifies the containers from which logs are not collected. By default, this parameter is empty, which indicates that logs are collected from all containers. If you want to configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional. If the EnvValue parameter is empty, logs are not collected from the containers whose environment variables contain the keys specified by EnvKey. If a value is specified for the EnvValue parameter, logs are not collected from the containers whose environment variables include EnvKey=EnvValue. By default, the value of the EnvValue parameter is a string. Logs are not collected from the containers whose environment variable values match the value of the EnvValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80\|6379)$`, it matches containers whose service ports are 80 and 6379. Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container is filtered out. K8s Pod Label Whitelist Uses the Kubernetes label whitelist to specify containers from which logs are collected. If you want to configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, logs are collected from the containers whose Kubernetes labels contain the keys specified by LabelKey. If a value is specified for the LabelValue parameter, logs are collected from the containers whose Kubernetes labels include LabelKey=LabelValue. By default, the value of the LabelValue parameter is a string. Logs are collected from the containers whose Kubernetes label values match the value of the LabelValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set LabelKey to `app` and LabelValue to `^(test1\|test2)$`, it matches containers whose Kubernetes labels include app:test1 and app:test2. Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container is matched. Note Because changing labels on Kubernetes control resources, such as deployments, does not restart specific working resource pods, the pods cannot perceive this change, which may cause the matching rules to fail. When you configure the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods. For more information about Kubernetes labels, see Labels and Selectors. K8s Pod Label Blacklist Uses the Kubernetes label blacklist to exclude containers from which logs are not collected. If you want to configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If the LabelValue parameter is empty, logs are not collected from the containers whose Kubernetes labels contain the keys specified by LabelKey. If a value is specified for the LabelValue parameter, logs are not collected from the containers whose Kubernetes labels include LabelKey=LabelValue. By default, the value of the LabelValue parameter is a string. Logs are not collected from the containers whose Kubernetes label values match the value of the LabelValue parameter. If the value starts with `^` and ends with `$`, regular expression matching is performed. For example, if you set LabelKey to `app` and LabelValue to `^(test1\|test2)$`, it matches containers whose Kubernetes labels include app:test1 and app:test2. Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container is filtered out. Note Because changing labels on Kubernetes control resources, such as deployments, does not restart specific working resource pods, the pods cannot perceive this change, which may cause the matching rules to fail. When you configure the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods. For more information about Kubernetes labels, see Labels and Selectors.
Log Tag Enrichment	You can add environment variables and Kubernetes labels to logs as log tags. Environment Variable Related After you configure environment variable extension fields, Simple Log Service adds environment variable-related fields to logs. For example, if you set Environment Variable Name to `VERSION` and Tag Name to `env_version`, when the container contains the environment variable `VERSION=v1.0.0`, this information is added to the log, which means that the field __tag__:__env_version__: v1.0.0 is added. Pod Label Related After you configure Kubernetes pod extension fields, Simple Log Service adds Kubernetes pod label-related fields to logs. For example, if you set Pod Label Name to `app` and Tag Name to `k8s_pod_app`, when Kubernetes contains the label `app=serviceA`, this information is added to the log, which means that the field __tag__:__k8s_pod_app__: serviceA is added.
First Collection Size	The starting collection position of the standard output log file when the configuration takes effect for the first time. The default value of First Collection Size is 1024 KB. If the file size is less than 1024 KB, Logtail collects data from the beginning of the file. If the file size is greater than 1024 KB, Logtail collects the last 1024 KB of data in the file. You can modify the First Collection Size here. The value range is from 0 to 10485760. Unit: KB.
Advanced Parameters	Some parameters of the Logtail configuration need to be manually entered. For more information, see Create Logtail pipeline configuration.

Configuration example

CRD-AliyunPipelineConfig

Warning

If you use AliyunPipelineConfig, the image version of alibaba-log-controller must be no less than 0.5.1.

Parameter	Type	Required	Default value	Description
Type	string	Yes	/	The type of the plug-in. Fixed to input_container_stdio.
IgnoringStdout	Boolean	No	false	Whether to ignore stdout.
IgnoringStderr	Boolean	No	false	Whether to ignore stderr.
TailSizeKB	uint	No	1024	The starting collection position of the standard output file when the configuration takes effect for the first time. If the file size is less than this value, Logtail collects data from the beginning of the file. The value range is from 0 to 10485760 KB.
Multiline	object	No	Empty	The multi-line aggregation options. For more information, see Table 1: Multi-line aggregation options.
ContainerFilters	object	No	Empty	The container filter options. Multiple options are evaluated by using a logical AND. For more information, see Table 2: Container filter options.
ExternalK8sLabelTag	map	No	Empty	For containers deployed in a Kubernetes environment, additional tags related to pod labels need to be added to logs. The key in the map is the pod label name, and the value is the corresponding tag name. For example, if you add `app: k8s_label_app` to the map, when the pod contains the label `app=serviceA`, this information is added to the log as a tag, which means that the field __tag__:k8s_label_app: serviceA is added. If the pod does not contain the `app` label, an empty field __tag__:k8s_label_app: is added.
ExternalEnvTag	map	No	Empty	For containers deployed in a Kubernetes environment, additional tags related to container environment variables need to be added to logs. The key in the map is the environment variable name, and the value is the corresponding tag name. For example, if you add `VERSION: env_version` to the map, when the container contains the environment variable `VERSION=v1.0.0`, this information is added to the log as a tag, which means that the field __tag__:env_version: v1.0.0 is added. If the container does not contain the `VERSION` environment variable, an empty field __tag__:env_version: is added.
FlushTimeoutSecs	uint	No	5	If no new complete logs appear in the file for a specified period of time, the content in the current read buffer is output as a log.
AllowingIncludedByMultiConfigs	bool	No	false	Whether to allow the current configuration to collect the standard output logs of containers that are already matched by other configurations.

Table 1: Multi-line aggregation options

Parameter	Type	Required	Default value	Description
Mode	string	No	custom	The multi-line aggregation mode. Only custom is supported.
StartPattern	string	Required when Multiline.Mode is set to custom	Empty	The regular expression for the start of a line.
UnmatchedContentTreatment	string	No	single_line	The processing method for unmatched log segments. The following options are available: discard: Discard. single_line: Store each line of the unmatched log segment in a separate event.

Table 2: Container filter options

Parameter	Type	Required	Default value	Description
K8sNamespaceRegex	string	No	Empty	For containers deployed in a Kubernetes environment, specify the namespace condition of the pod to which the container belongs. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported.
K8sPodRegex	string	No	Empty	For containers deployed in a Kubernetes environment, specify the name condition of the pod to which the container belongs. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported.
IncludeK8sLabel	map	No	Empty	For containers deployed in a Kubernetes environment, specify the label condition of the pod to which the container belongs. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the pod label name, and the value is the pod label value. The following describes the conditions: If the value in the map is empty, pods that contain the key as a label key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, pods are matched when the pod label contains the key as a label name and the corresponding label value matches the value. In other cases, pods are matched when the pod label contains the key as a label name and the value as a label value.
ExcludeK8sLabel	map	No	Empty	For containers deployed in a Kubernetes environment, specify the label condition of the pod to which the container belongs that needs to be excluded from collection. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the pod label name, and the value is the pod label value. The following describes the conditions: If the value in the map is empty, pods that contain the key as a label key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, pods are matched when the pod label contains the key as a label name and the corresponding label value matches the value. In other cases, pods are matched when the pod label contains the key as a label name and the value as a label value.
K8sContainerRegex	string	No	Empty	For containers deployed in a Kubernetes environment, specify the name condition of the container to be collected. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported.
IncludeEnv	map	No	Empty	Specify the environment variable conditions of the container to be collected. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the environment variable name, and the value is the environment variable value. The following describes the conditions: If the value in the map is empty, containers that contain the key as an environment variable key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers are matched when the container environment variable contains the key as an environment variable name and the corresponding environment variable value matches the value. In other cases, containers are matched when the container environment variable contains the key as an environment variable name and the value as an environment variable value.
ExcludeEnv	map	No	Empty	Specify the environment variable conditions of the container to be excluded from collection. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the environment variable name, and the value is the environment variable value. The following describes the conditions: If the value in the map is empty, containers that contain the key as an environment variable key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers are matched when the container environment variable contains the key as an environment variable name and the corresponding environment variable value matches the value. In other cases, containers are matched when the container environment variable contains the key as an environment variable name and the value as an environment variable value.
IncludeContainerLabel	map	No	Empty	Specify the label conditions of the container to be collected. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the container label name, and the value is the container label value. The following describes the conditions: If the value in the map is empty, containers that contain the key as a label key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers are matched when the container label contains the key as a label name and the corresponding label value matches the value. In other cases, containers are matched when the container label contains the key as a label name and the value as a label value.
ExcludeContainerLabel	map	No	Empty	Specify the label conditions of the container to be excluded from collection. Multiple conditions are evaluated by using a logical OR. If this parameter is not added, logs are collected from all containers. Regular expression matching is supported. The key in the map is the container label name, and the value is the container label value. The following describes the conditions: If the value in the map is empty, containers that contain the key as a label key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers are matched when the container label contains the key as a label name and the corresponding label value matches the value. In other cases, containers are matched when the container label contains the key as a label name and the value as a label value.

Configuration example

Manage collection configurations with AliyunPipelineConfig

apiVersion: telemetry.alibabacloud.com/v1alpha1
# Create a ClusterAliyunPipelineConfig
kind: ClusterAliyunPipelineConfig
metadata:
  # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. The name is also the name of the iLogtail collection configuration that is created.
  name: example-k8s-stdout
spec:
  # Specify the target project
  project:
    name: k8s-log-clusterid
  # Create a Logstore to store logs
  logstores:
    - name: k8s-stdout
  # Define the iLogtail collection configuration
  config:
    # Specify the sample log. You can leave this parameter empty.
    sample: |
      2024-06-19 16:35:00 INFO test log
      line-1
      line-2
      end
      # Configure the Logtail input plug-ins
    inputs:
      # Use the input_container_stdio plug-in to collect container standard output
      - Type: input_container_stdio
        # Collect stdout logs
        IgnoringStdout: false
        # Do not collect stderr logs
        IgnoringStderr: true
        # Configure container information filter conditions. Multiple options are evaluated by using a logical AND.
        ContainerFilters:
          # Specify the namespace of the pod to which the container belongs. Regular expression matching is supported.
          K8sNamespaceRegex: "^(default)$"
          # Specify the name of the container to be collected. Regular expression matching is supported.
          K8sContainerRegex: "^(.*app.*)$"
        # Configure multi-line chunk configuration
        # Configure the regular expression for the start of a line
        Multiline:
          Mode: custom
          StartPattern: \d+-\d+-\d+.*
          UnmatchedContentTreatment: single_line
    # Configure the Logtail output plug-ins
    flushers:
      # Use the flusher_sls plug-in to send logs to a specific Logstore.
      - Type: flusher_sls
        Logstore: k8s-stdout
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

Preview container metadata

Note

After creating a Logtail configuration, you can view container metadata and reasons for non-matching filter conditions on the Logtail configuration page.

In the Project List, click the project where you created the Logtail configuration.
In the Log Storage > Logstore tab, select the Logstore you used to create the Logtail configuration, then click > on the left side of the Logstore, and select Data Access > Logtail Configuration.
In the Logtail configuration list, click the desired Logtail configuration.
On the Logtail Configuration page, click Edit.
Turn on the Enable Container Metadata Preview switch.
Click Container Metadata Preview.
In the Container Preview dialog box, review the container metadata.
- The Matched Containers tab shows metadata for containers that meet your filter conditions.
- The All Containers tab displays metadata for all containers in the current Kubernetes cluster, including reasons why some containers did not match the filter conditions.

Log fields

The following fields are uploaded by default with each log in a Kubernetes cluster:

Field name	Description
_time_	The time when the log is collected.
_source_	The log source type, stdout or stderr.
__tag__:_image_name_	Image name
__tag__:_container_name_	Container name
__tag__:_pod_name_	Pod name
__tag__:_namespace_	The namespace where the pod resides
__tag__:_pod_uid_	The unique identifier of the pod

Troubleshooting

If you encounter issues when using Logtail to collect logs from containers, including standard and Kubernetes containers, refer to the following topics for troubleshooting:

If standard error logs and access logs have different formats and cannot be parsed normally, consider adding an index to search for errors in the content. Simple Log Service does not support using plugins to extract fields for parsing. For more information, see Create an index.