This topic describes how to create a Logtail configuration in the Simple Log Service console and use the Logtail configuration to collect container stdout and stderr in DaemonSet mode.
Prerequisites
The Logtail component is installed. For more information, see Install Logtail components in a Kubernetes cluster.
A Logstore is created in the project that you use to install the Logtail component. For more information, see Create a Logstore.
The container from which you want to collect logs continuously generates logs.
ImportantLogtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.
Features
Logtail can collect container stdout and stderr, and then upload the stdout and stderr together with container metadata to Simple Log Service. Logtail supports the following features:
- Collects stdout and stderr.
- Uses the container label whitelist to specify containers from which logs are collected.
- Uses the container label blacklist to specify containers from which logs are not collected.
- Uses the environment variable whitelist to specify containers from which logs are collected.
- Uses the environment variable blacklist to specify containers from which logs are not collected.
- Collects multi-line logs. For example, Logtail can collect Java stack logs.
- Automatically associates container metadata that needs to be uploaded together with the collected container logs. The metadata includes container names, image names, pod names, namespaces, and environment variables.
- If a container runs in a Kubernetes cluster, Logtail also supports the following features:
- Uses Kubernetes namespaces, pod names, and container names to specify containers from which logs are collected.
- Uses the Kubernetes label whitelist to specify containers from which logs are collected.
- Uses the Kubernetes label blacklist to specify containers from which logs are not collected.
- Automatically associates Kubernetes labels that need to be uploaded together with the collected container logs.
Implementation
Logtail communicates with the domain socket of Docker. Logtail queries all Docker containers and identifies the containers from which stdout and stderr are collected by using the specified labels and environment variables. Logtail runs the docker logs
command to collect logs from the specified containers.
When Logtail collects stdout and stderr from a container, Logtail periodically stores checkpoints to a checkpoint file. If Logtail is stopped and then started, Logtail collects stdout and stderr from the last checkpoint.
Limits
You can use the Simple Log Service console to collect stdout and stderr in DaemonSet mode only if Logtail runs V0.16.0 or later and runs on Linux. For more information about Logtail versions and version updates, see Install Logtail on a Linux server.
Logtail collects data from containers that use the Docker engine or containerd engine.
- Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
- containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
By default, the last multi-line log that is collected by Logtail is cached for 3 seconds. This prevents the multi-line log from being split into multiple logs due to output latency. You can change the cache time by modifying the
BeginLineTimeoutMs
parameter. We recommend that you do not specify a value less than 1000 with millisecond precision. If you specify a value that is less than 1000, an error may occur.If Logtail detects the
die
event on a container that is stopped, Logtail no longer collects stdout or stderr from the container. If collection latency exists, some stdout and stderr that are collected before the container is stopped may be lost.The logging driver collects stdout and stderr only in the JSON format from containers that use the Docker engine.
By default, stdout and stderr that are collected from different containers by using the same Logtail configuration have the same context. In this case, you cannot view the stdout and stderr of a container in the Simple Log Service console by using the context query feature or Livetail feature. To use the features, add the aggregator_context plug-in. For more information, see Configure aggregators.
By default, the collected data is stored in the
content
field. Logtail can process the collected data. For more information, see Use Logtail plug-ins to process data.
Create a Logtail configuration
- Log on to the Log Service console.
In the Import Data section, click Kubernetes - Standard Output.
Select a project and a Logstore. Then, click Next.
In this example, select the project that you use to install the Logtail component and the Logstore that you create.
Click Use Existing Machine Groups.
After you install the Logtail component, Simple Log Service automatically creates a machine group named
k8s-group-${your_k8s_cluster_id}
. You can select this machine group.Select the
k8s-group-${your_k8s_cluster_id}
machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.ImportantIf the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?
In the Specify Data Source step, specify a data source and click Next.
You can specify a data source by using form configuration or editor configuration in JSON. For more information, see Details of the Logtail configuration.
- Preview data, configure indexes, and then click Next. By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
- Click Log Query. You are redirected to the query and analysis page of your Logstore. You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.
Details of the Logtail configuration
Form configuration
In the Specify Data Source step, configure the parameters, as shown in the following figure.
The following section describes the parameters that you can configure:
Basic Information
Parameter
Description
Config Name
The name of the Logtail configuration.
Standard Output
If you select Standard Output, Logtail collects container stdout.
Standard Error
If you select Standard Error, Logtail collects container stderr.
Collection Mode
The mode of collection. You can collect container stdout and stderr in single-line or multi-line mode.
If you collect container stdout and stderr in multi-line mode, you must configure Regex to Match First Line.
Regex to Match First Line
The regular expression that is used to match the beginning of the first line of a log. If the beginning of a line matches the specified regular expression, the line is considered the first line of a new log. If the beginning of a line does not match the specified regular expression, the line is considered a part of the previous log.
Container Filtering
A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the LabelKey parameter for a namespace is
io.kubernetes.pod.namespace
. The value of the LabelKey parameter for a container name isio.kubernetes.container.name
. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specifyio.kubernetes.pod.namespace : backend-prod
orio.kubernetes.container.name : worker-server
in the container label whitelist.ImportantContainer labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels.
Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values for the LabelKey parameter, only one of the values takes effect.
Parameter
Description
Container Label Whitelist
The container label whitelist. The whitelist specifies the containers from which stdout and stderr are collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are matched.
If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are matched.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.
Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
Container Label Blacklist
The container label blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are filtered out.
If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.
Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.
Environment Variable Whitelist
The environment variable whitelist. The whitelist specifies the containers from which stdout and stderr are collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional.
If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are matched.
If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are matched.
By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.
Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
Environment Variable Blacklist
The environment variable blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional.
If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are filtered out.
If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are filtered out.
By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.
Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.
K8s Label Whitelist
The Kubernetes label whitelist. The whitelist specifies the containers from which stdout and stderr are collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched.
If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are matched.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
), regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.
Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is matched.
NoteLogtail V1.0.34 and later support this parameter.
If you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods. For more information about Kubernetes labels, see Labels and Selectors.
K8s Label Blacklist
The Kubernetes label blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out.
If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
), regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.
Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is filtered out.
NoteLogtail V1.0.34 and later support this parameter.
If you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods. For more information about Kubernetes labels, see Labels and Selectors.
Namespace
The namespace. The namespace specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. For example, if you specify "K8sNamespaceRegex":"^(default|nginx)$", all containers in the nginx and default namespaces are matched.
NoteLogtail V1.0.34 and later support this parameter.
Pod Name
The pod name. The pod name specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. For example, if you specify "K8sPodRegex":"^(nginx-log-demo.*)$",, all containers in the pod whose name starts with nginx-log-demo are matched.
NoteLogtail V1.0.34 and later support this parameter.
Container Name
The container name. The container name specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify "K8scontainerRegex":"^(container-test)$", all containers whose name is container-test are matched.
NoteLogtail V1.0.34 and later support this parameter.
Field Extension
For Logtail V1.0.34 or later, we recommend that you specify environment variables or Kubernetes labels for logs as log labels.
Parameter
Description
Extension with Environment Variable
After you configure this parameter, Simple Log Service adds environment variable-related fields to logs. For example, you set Environment Variable Name to VERSION and set Extended Field Name to env_version. If the environment variable configurations of a container include
VERSION=v1.0.0
, Simple Log Service adds the __tag__:__env_version__: v1.0.0 field to logs.Extension with Kubernetes Label
After you configure this parameter, Simple Log Service adds Kubernetes label-related fields to logs. For example, you set K8s Label to app and set Extended Field Name to
k8s_label_app
. If the label configurations of a Kubernetes cluster includeapp=serviceA
, Simple Log Service adds the __tag__:__k8s_label_app__: serviceA field to logs.Advanced Parameters
Parameter
Description
Maximum Log Length (Unit: Bytes)
The maximum size of a log. Default value: 524288. Valid values: 1024 to 20971520. Unit: bytes.
If the size of a log exceeds the value of this parameter, Logtail stops matching the beginning of the first line of the next log and uploads the log to Simple Log Service.
Maximum Length of Traced Historical Data during First Collection (Unit: Bytes)
The maximum size of historical data that can be traced the first time Logtail collects logs from a log file. The default value is 131072 bytes, which is equivalent to 128 KB. Valid values: 131072 to 1048576. Unit: bytes.
First Line Length Matched by Regex (Unit: Bytes)
The size of the beginning of the first line of a log that matches the specified regular expression. Default value: 10240. Unit: bytes.
You can configure this parameter to check whether the beginning of the first line of a log matches the specified regular expression. We recommend that you configure this parameter to improve match efficiency.
BeginLineTimeoutMs
The timeout period for matching the beginning of the first line of a log based on the specified regular expression. Default value: 3000. Unit: milliseconds.
If no new log is generated within 3,000 milliseconds, Logtail stops matching and uploads the last log to Simple Log Service.
Editor configuration in JSON
In the Plug-in Configuration field, specify the information about your Logtail configuration.
If your logs cannot be parsed based on your Logtail configuration, you can add the processors configuration in the Plug-in Configuration field to add plug-ins for data processing. For example, you can add plug-ins to extract fields, extract log time, mask data, and filter logs. For more information, see Use Logtail plug-ins to process data.
{
"inputs":[
{
"type":"service_docker_stdout",
"detail":{
"Stdout":true,
"Stderr":true,
"IncludeContainerLabel":{
"LabelKey":"LabelValue"
},
"ExcludeContainerLabel":{
"LabelKey":"LabelValue"
},
"IncludeK8sLabel":{
"LabelKey":"LabelValue"
},
"ExcludeK8sLabel":{
"LabelKey":"LabelValue"
},
"IncludeEnv":{
"EnvKey":"EnvValue"
},
"ExcludeEnv":{
"EnvKey":"EnvValue"
},
"ExternalK8sLabelTag":{
"EnvKey":"EnvValue"
},
"ExternalEnvTag":{
"EnvKey":"EnvValue"
},
"K8sNamespaceRegex":"^(default|kube-system)$",
"K8sPodRegex":"^(deploy.*)$",
"K8sContainerRegex":"^(container1|container2)$"
}
}
]
}
The following section describes the parameters that you can configure:
Data source type
The type of the data source is set to service_docker_stdout and cannot be changed.
Parameters related to container filtering
For versions earlier than Logtail V1.0.34, you can filter containers only by using environment variables or container labels.
A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the LabelKey parameter for a namespace is
io.kubernetes.pod.namespace
. The value of the LabelKey parameter for a container name isio.kubernetes.container.name
. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specifyio.kubernetes.pod.namespace : backend-prod
orio.kubernetes.container.name : worker-server
in the container label whitelist.ImportantContainer labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels.
Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values for the LabelKey parameter, only one of the values takes effect.
Parameter
Type
Required
Description
IncludeLabel
Map (The values of the LabelKey and LabelValue parameters are strings.)
No
The container label whitelist. The whitelist specifies the containers from which stdout and stderr are collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are matched.
If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are matched.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.
Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
ExcludeLabel
Map (The values of the LabelKey and LabelValue parameters are strings.)
No
The container label blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose container labels contain the keys specified by LabelKey are filtered out.
If the LabelValue parameter is not empty, containers whose container labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.
Key-value pairs are connected by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.
IncludeEnv
Map (The values of the EnvKey and EnvValue parameters are strings.)
No
The environment variable whitelist. The whitelist specifies the containers from which stdout and stderr are collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional.
If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are matched.
If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are matched.
By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.
Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
ExcludeEnv
Map (The values of the EnvKey and EnvValue parameters are strings.)
No
The environment variable blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. This parameter is empty by default, which indicates that stdout and stderr are collected from all containers. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional.
If the EnvValue parameter is empty, containers whose environment variables contain the keys specified by EnvKey are filtered out.
If the EnvValue parameter is not empty, containers whose environment variables consist of the key-value pairs specified by EnvKey and EnvValue are filtered out.
By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.
Key-value pairs are connected by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.
For Logtail V1.0.34 or later, we recommend that you use different levels of Kubernetes information, such as pod names, namespaces, container names, and labels to filter containers.
NoteIf you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods. For more information about Kubernetes labels, see Labels and Selectors.
Parameter
Type
Required
Description
IncludeK8sLabel
Map (The values of the LabelKey and LabelValue parameters are strings.)
No
The Kubernetes label whitelist. The whitelist specifies the containers from which stdout and stderr are collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched.
If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are matched.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
), regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.
Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is matched.
ExcludeK8sLabel
Map (The values of the LabelKey and LabelValue parameters are strings.)
No
The Kubernetes label blacklist. The blacklist specifies the containers from which stdout and stderr are not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
If the LabelValue parameter is empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out.
If the LabelValue parameter is not empty, containers whose Kubernetes labels consist of the key-value pairs specified by LabelKey and LabelValue are filtered out.
By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (
^
) and ends with a dollar sign ($
), regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.
Key-value pairs are connected by using the OR operator. If a container has a Kubernetes label that consists of one of the specified key-value pairs, the container is filtered out.
K8sNamespaceRegex
string
No
The namespace. The namespace specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. For example, if you specify "K8sNamespaceRegex":"^(default|nginx)$", all containers in the nginx and default namespaces are matched.
K8sPodRegex
string
No
The pod name. The pod name specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. For example, if you specify "K8sPodRegex":"^(nginx-log-demo.*)$",, all containers in the pod whose name starts with nginx-log-demo are matched.
K8sContainerRegex
string
No
The container name. The container name specifies the containers from which stdout and stderr are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify "K8scontainerRegex":"^(container-test)$", all containers whose name is container-test are matched.
Parameters related to log labels
For Logtail V1.0.34 or later, we recommend that you specify environment variables or Kubernetes labels for logs as log labels.
Parameter
Type
Required
Description
ExternalEnvTag
Map (The values of the EnvKey and EnvValue parameters are strings.)
No
After you specify environment variables as log labels, Simple Log Service adds environment variable-related fields to logs. For example, you set EnvKey to VERSION and set EnvValue to env_version. If the environment variable configurations of a container include
VERSION=v1.0.0
, Simple Log Service adds the __tag__:__env_version__: v1.0.0 field to logs.ExternalK8sLabelTag
Map (The values of the LabelKey and LabelValue parameters are strings.)
No
After you specify Kubernetes labels as log labels, Simple Log Service adds Kubernetes label-related fields to logs. For example, you set LabelKey to app and set LabelValue to
k8s_label_app
. If the label configurations of a Kubernetes cluster includeapp=serviceA
, Simple Log Service adds the __tag__:__k8s_label_app__: serviceA field to logs.Other parameters
Parameter
Type
Required
Description
Stdout
boolean
No
Specifies whether to collect container stdout. Valid values:
true (default): collects container stdout.
false: does not collect container stdout.
Stderr
boolean
No
Specifies whether to collect container stderr. Valid values:
true (default): collects container stderr.
false: does not collect container stderr.
BeginLineRegex
string
No
The regular expression that is used to match the beginning of the first line of a log.
This parameter is empty by default, which indicates that each line is regarded as a log.
Logtail uses a regular expression to match the beginning of the first line of a log. If the beginning of a line matches the specified regular expression, the line is considered the first line of a new log. If the beginning of a line does not match the specified regular expression, the line is considered a part of the previous log.
BeginLineTimeoutMs
int
No
The timeout period for matching the beginning of the first line of a log based on the specified regular expression. Default value: 3000. Unit: milliseconds.
If no new log is generated within 3,000 milliseconds, Logtail stops matching and uploads the last log to Simple Log Service.
BeginLineCheckLength
int
No
The size of the beginning of the first line of a log that matches the specified regular expression. Default value: 10240. Unit: bytes.
You can configure this parameter to check whether the beginning of the first line of a log matches the specified regular expression. We recommend that you configure this parameter to improve match efficiency.
MaxLogSize
int
No
The maximum size of a log. Default value: 524288. Valid values: 1024 to 20971520. Unit: bytes.
If the size of a log exceeds the value of this parameter, Logtail stops matching the beginning of the first line of the next log and uploads the log to Simple Log Service.
StartLogMaxOffset
int
No
The maximum size of historical data that can be traced the first time Logtail collects logs from a log file. The default value is 131072 bytes, which is equivalent to 128 KB. Valid values: 131072 to 1048576. Unit: bytes.
Preview container metadata
After you create a Logtail configuration, you can view container metadata and the reason why specific containers failed to match the filter conditions on the Logtail Configuration page.
Only Linux Logtail V1.4.0 and later or Windows Logtail V1.4.0.0 and later support this feature.
- Log on to the Log Service console.
In the Projects section, click the name of the project that you want to manage.
In this example, the project that you use to create the Logtail configuration is selected.
In the left-side navigation pane, choose . On the Logstores tab, click the > icon on the left side of the Logstore that you want to manage and choose .
In this example, the Logstore that you use to create the Logtail configuration is selected.
In the Logtail configuration list, click the name of the Logtail configuration that you want to manage.
On the Logtail Configuration page, click Edit.
In the Container Filtering section of the Configuration Details tab, turn on Container Metadata Preview.
Click Container Metadata Preview.
In the Container Preview dialog box, view container metadata.
The Matched Containers tab displays the metadata of the containers that match the specified filter conditions.
The All Containers tab displays the metadata of all containers in the current Kubernetes cluster and the reason why specific containers failed to match the filter conditions.
Examples of Logtail configurations for single-line logs
Filter containers based on the environment variable whitelist and the environment variable blacklist
Collect stdout and stderr from the containers whose environment variable configurations include NGINX_SERVICE_PORT=80
but exclude POD_NAMESPACE=kube-system
.
Obtain environment variables.
To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.
Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeEnv": { "NGINX_SERVICE_PORT": "80" }, "ExcludeEnv": { "POD_NAMESPACE": "kube-system" } } } ] }
Filter containers based on the container label whitelist and the container label blacklist
Collect stdout and stderr from the containers whose container label is io.kubernetes.container.name=nginx
.
Obtain container labels.
To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.
Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeLabel": { "io.kubernetes.container.name": "nginx" } } } ] }
Filter containers by using Kubernetes namespaces, pod names, and container names
Collect stdout and stderr from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.
Obtain different levels of Kubernetes information.
Obtain information about pods.
Obtain information about namespaces.
Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "K8sNamespaceRegex":"^(default)$", "K8sPodRegex":"^(nginx-log-demo.*)$", "K8sContainerRegex":"^(nginx-log-demo-0)$" } } ] }
Filter containers by using Kubernetes labels
Collect stdout and stderr from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.
Obtain Kubernetes labels.
Create a Logtail configuration.
Example:
{ "inputs": [ { "type": "service_docker_stdout", "detail": { "Stdout": true, "Stderr": true, "IncludeK8sLabel":{ "job-name":"^(nginx-log-demo.*)$" } } } ] }
Examples of Logtail configurations for multi-line logs
Java exception stack logs are multi-line logs. You can create a Logtail configuration to collect the Java exception stack logs based on the following descriptions:
Sample logs
2021-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start 2021-02-03 14:18:41.969 ERROR [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : java.lang.NullPointerException at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ... 2021-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done
Logtail configuration
Collect the Java exception stack logs of the containers whose container label is
app=monitor
. The Java exception stack logs start with a date that is in a fixed format. Logtail matches only the first 10 bytes of each line to improve match efficiency. After the logs are collected and sent to Simple Log Service, Simple Log Service uses regular expressions to parse the logs into fields such as time, level, module, thread, and message.inputs is required and is used to configure the data source settings for the Logtail configuration. You must configure inputs based on your data source.
ImportantYou can specify only one type of data source in inputs.
processors is optional and is used to configure the data processing settings for the Logtail configuration. You can specify one or more processing methods. For more information, see Use Logtail plug-ins to process data.
{ "inputs": [ { "detail": { "BeginLineCheckLength": 10, "BeginLineRegex": "\\d+-\\d+-\\d+.*", "IncludeLabel": { "app": "monitor" } }, "type": "service_docker_stdout" } ], "processors": [ { "type": "processor_regex", "detail": { "SourceKey": "content", "Regex": "(\\d+-\\d+-\\d+ \\d+:\\d+:\\d+\\.\\d+)\\s+(\\w+)\\s+\\[([^]]+)]\\s+\\[([^]]+)]\\s+([\\s\\S]*)", "Keys": [ "time", "level", "module", "thread", "message" ], "NoKeyError": true, "NoMatchError": true, "KeepSource": false } } ] }
Parsed logs
For example, if the collected log is
2018-02-03 14:18:41.968 INFO [spring-cloud-monitor] [nio-8080-exec-4] c.g.s.web.controller.DemoController : service start done
, the log is parsed into the following fields:_time_ is the log collection time. You can configure the Timezone parameter in the Logtail configuration to specify a time zone for _time_.
time is a time value recorded in the log and is extracted from the log.
__tag__:__hostname__:logtail-dfgef _container_name_:monitor _image_name_:example.com-hangzhou.aliyuncs.xxxxxxxxxxxxxxx _namespace_:default _pod_name_:monitor-6f54bd5d74-rtzc7 _pod_uid_:7f012b72-04c7-11e8-84aa-00163f00c369 _source_:stdout _time_:2018-02-02T14:18:41.979147844Z time:2018-02-02 02:18:41.968 level:INFO module:spring-cloud-monitor thread:nio-8080-exec-4 class:c.g.s.web.controller.DemoController message:service start done
Log fields
The following table describes the fields that are uploaded by default for each log in a Kubernetes cluster.
Field | Description |
_time_ | The time at which the log is collected. Example: |
_source_ | The type of the log source. Valid values: stdout and stderr. |
_image_name_ | The name of the image. |
_container_name_ | The name of the container. |
_pod_name_ | The name of the pod. |
_namespace_ | The namespace of the pod. |
_pod_uid_ | The unique identifier of the pod. |
Troubleshooting
If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topics: