Simple Log Service:Use the Simple Log Service console to collect container text logs in DaemonSet mode

Limits

• If Logtail detects the die event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost.

• For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume on the directory of logs. Then, a temporary directory is generated.

  If an Apsara File Storage NAS (NAS) file system is mounted on the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode. For more information, see Use the Simple Log Service console to collect container text logs in Sidecar mode.

• Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.

• If a volume is mounted on the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.

  For example, if a volume is mounted on the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.

• By default, Kubernetes mounts the root directory of the host on the /logtail_host directory of the Logtail container. If you want to collect text logs from the host, you must specify /logtail_host as the prefix of the log file path.

  For example, if you want to collect logs from the /home/logs/app_log/ directory of the host, you must specify /logtail_host/home/logs/app_log/ as the log file path.

• Logtail collects data from containers that use the Docker engine or containerd engine.

  • Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

  • containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

Procedure

1. Log on to the Simple Log Service console.

2. In the Import Data section, find and click Kubernetes - Object.

3. Select a project and a Logstore. Then, click Next.

   In this example, select the project that you use to install the Logtail components and the Logstore that you create.

4. On the Kubernetes Clusters > ACK Daemonset tab, click Use Existing Machine Groups.

   After you install the Logtail components, Simple Log Service automatically creates a machine group named k8s-group-${your_k8s_cluster_id}. You can select this machine group.

5. Select the k8s-group-${your_k8s_cluster_id} machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.

   Important

   If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?

6. Create a Logtail configuration and click Next.

   Global Configurations

   Parameter	Description
   Config Name	Enter a name for the Logtail configuration. The name must be unique in a project. After you create the Logtail configuration, you cannot change the name of the Logtail configuration.
   Log Sample	Enter a sample log that is collected from an actual scenario. You can use the sample log to configure parameters that are related to log processing with ease. Examples: [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16) You can enter multiple sample logs. The total length of the logs cannot exceed 1,500 characters.
   Log Topic Type	Select the topic generation mode. For more information, see Log topics. Valid values: Topic: In this mode, topics are configured at the machine group level. If you want to distinguish the logs that are generated by different servers, select this mode. File Path Extraction: In this mode, you must specify a custom regular expression. The part of a log path that matches the regular expression is used as the topic. If you want to distinguish the logs that are generated by different users or instances, select this mode. Custom: In this mode, you must specify a custom log topic.

   Input Configurations

   Parameter	Description
   Logtail Deployment Mode	Select the deployment mode of Logtail. In this example, select Daemonset.
   File Path Type	Select the file path type of the logs that you want to collect. Valid values: Path in Container and Host Path. In this example, select Path in Container.
   File Path	Specify the directory and name of log files. Configure this parameter based on the location of the logs that you want to collect. If the container runs on a Linux host, the log path must start with a forward slash (/). Example: /apsara/nuwa/**/app.Log. If the container runs on a Windows host, the log path must start with a drive letter. Example: C:\Program Files\Intel\**\*.Log. You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. For more information, see Wildcard matching. When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcard characters. Simple Log Service scans all levels of the specified directory for the log files that match specified conditions. Examples: If you specify /apsara/nuwa/**/*.log, Simple Log Service collects logs from the log files whose names are suffixed by .log in the /apsara/nuwa directory and the recursive subdirectories of the directory. If you specify /var/logs/app_*/**/*.log, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by .log. The file is stored in a subdirectory under the /var/logs directory or in a recursive subdirectory of the subdirectory. The name of the subdirectory matches the app_* pattern. If you specify /var/log/nginx/**/access*, Simple Log Service collects logs from the log files whose names start with access in the /var/log/nginx directory and the recursive subdirectories of the directory.
   Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched for the wildcard characters ** included in the value of File Path. A value of 0 specifies that only the log file directory that you specify is monitored.
   Multi-line Mode	If you want to collect multi-line logs, turn on Multi-line Mode.
   Regex to Match First Line	Configure a regular expression to match the beginning of the first line of a log. Logtail uses the regular expression to match the beginning of the first line of a log and considers the content that does not match the regular expression as part of the log. Simple Log Service can automatically generate a regular expression or use the regular expression that you manually specify. Automatic generation Click Auto Generate. Simple Log Service automatically generates a regular expression based on your sample log to match the beginning of the first line of a log. Manual configuration Click Manual and manually specify a regular expression to match the beginning of the first line of a log. Then, click Validate to check whether the regular expression is valid. For more information, see How do I test a regular expression?
   Enable Container Metadata Preview	If you turn on Enable Container Metadata Preview, you can view the container metadata after you create the Logtail configuration, including the matched container information and full container information.
   Container Filtering	Specify conditions to filter containers. For versions earlier than Logtail V1.0.34, you can filter containers only by using environment variables and container labels. For Logtail V1.0.34 and later, we recommend that you filter containers by using different levels of Kubernetes information, such as pod names, Kubernetes namespaces, container names, and labels. A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the Label Name parameter for a namespace is io.kubernetes.pod.namespace. The value of the Label Name parameter for a container name is io.kubernetes.container.name. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specify io.kubernetes.pod.namespace : backend-prod or io.kubernetes.container.name : worker-server in the container label whitelist. Important Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels. Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
   K8s Pod Name Regular Matching	Enter the pod name. The pod name specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(nginx-log-demo.*)$, all containers in the pod whose name starts with nginx-log-demo are matched.
   K8s Namespace Regular Matching	Enter the namespace. The namespace specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(default|nginx)$, all containers in the nginx and default namespaces are matched.
   K8s Container Name Regular Matching	Enter the container name. The container name specifies the containers from which text logs are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify ^(container-test)$, all containers whose name is container-test are matched.
   Container Label Whitelist	Configure a container label whitelist. The whitelist specifies the containers from which text logs are collected. Important Do not specify duplicate values for the Label Name parameter. If you specify duplicate values, only one value takes effect. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose container labels contain the specified label name are matched. If you specify a value for the Label Name and Tag Value parameters, containers whose container labels contain the specified Label name:Tag value are matched. By default, string matching is performed for the values of the Tag Value parameter. Containers are matched only if the values of the container labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to app and set the Tag Value parameter to ^(test1|test2)$, containers whose container labels contain app:test1 or app:test2 are matched. Key-value pairs are evaluated by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched.
   Container Label Blacklist	Configure a container label blacklist. The blacklist specifies the containers from which text logs are not collected. Important Do not specify duplicate values for the Label Name parameter. If you specify duplicate values, only one value takes effect. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose container labels contain the specified label name are filtered out. If you specify a value for the Label Name and Tag Value parameters, containers whose container labels contain the specified Label name:Tag value are filtered out. By default, string matching is performed for the values of the Tag Value parameter. Containers are filtered out only if the values of the container labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to app and set the Tag Value parameter to ^(test1|test2)$, containers whose container labels contain app:test1 or app:test2 are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out.
   Environment Variable Whitelist	Configure an environment variable whitelist. The whitelist specifies the containers from which text logs are collected. If you specify a value for the Environment Variable Name parameter but do not specify a value for the Environment Variable Value parameter, containers whose environment variables contain the specified environment variable name are matched. If you specify a value for the Environment Variable Name and Environment Variable Value parameters, containers whose environment variables contain the specified Environment variable name:Environment variable value are matched. By default, string matching is performed for the values of the Environment Variable Value parameter. Containers are matched only if the values of the environment variables are the same as the values of the Environment Variable Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the Environment Variable Value parameter, regular expression matching is performed. For example, if you set the Environment Variable Name parameter to NGINX_SERVICE_PORT and set the Environment Variable Value parameter to ^(80|6379)$, containers whose port number is 80 or 6379 are matched. Key-value pairs are evaluated by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched.
   Environment Variable Blacklist	Configure an environment variable blacklist. The blacklist specifies the containers from which text logs are not collected. If you specify a value for the Environment Variable Name parameter but do not specify a value for the Environment Variable Value parameter, containers whose environment variables contain the specified environment variable name are filtered out. If you specify a value for the Environment Variable Name and Environment Variable Value parameters, containers whose environment variables contain the specified Environment variable name:Environment variable value are filtered out. By default, string matching is performed for the values of the Environment Variable Value parameter. Containers are filtered out only if the values of the environment variables are the same as the values of the Environment Variable Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the Environment Variable Value parameter, regular expression matching is performed. For example, if you set the Environment Variable Name parameter to NGINX_SERVICE_PORT and set the Environment Variable Value parameter to ^(80|6379)$, containers whose port number is 80 or 6379 are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out.
   Kubernetes Pod Label Whitelist	Configure a Kubernetes pod label whitelist. The whitelist specifies the containers from which text logs are collected. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose pod labels contain the specified label name are matched. If you specify a value for the Label Name and Tag Value parameters, containers whose pod labels contain the specified Label name:Tag value are matched. By default, string matching is performed for the values of the Tag Value parameter. Containers are matched only if the values of the pod labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($), regular expression matching is performed. For example, if you set the Label Name parameter to environment and set the Tag Value parameter to ^(dev|pre)$, containers whose pod labels contain environment:dev or environment:pre are matched. Key-value pairs are evaluated by using the OR operator. If a container has a pod label that consists of one of the specified key-value pairs, the container is matched.
   Kubernetes Pod Label Blacklist	Configure a Kubernetes pod label blacklist. The blacklist specifies the containers from which text logs are not collected. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose pod labels contain the specified label name are filtered out. If you specify a value for the Label Name and Tag Value parameters, containers whose pod labels contain the specified Label name:Tag value are filtered out. By default, string matching is performed for the values of the Tag Value parameter. Containers are filtered out only if the values of the pod labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to environment and set the Tag Value parameter to ^(dev|pre)$, containers whose pod labels contain environment:dev or environment:pre are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has a pod label that consists of one of the specified key-value pairs, the container is filtered out.
   Log Tag Enrichment	Specify log tags.
   Environment Variables	After you specify environment variables as log tags, Simple Log Service adds environment variable-related fields to logs. For example, if you set the Environment Variable Name parameter to VERSION and set the Tag Name parameter to env_version, Simple Log Service adds the __tag__:__env_version__: v1.0.0 field to logs if the environment variable configurations of a container include VERSION=v1.0.0.
   Pod Labels	After you specify pod labels as log tags, Simple Log Service adds pod label-related fields to logs. For example, if you set the Pod Label Name parameter to environment and set the Tag Name parameter to env, Simple Log Service adds the __tag__:__env__: pre field to logs if the pod label configurations of a container include environment=pre.
   File Encoding	Select the encoding format of log file.
   First Collection Size	Specify the size of data that Logtail can collect from a log file the first time Logtail collects logs from the file. The default value of First Collection Size is 1024. Unit: KB. If the file size is less than 1,024 KB, Logtail collects data from the beginning of the file. If the file size is greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file. You can specify First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.
   Collection Blacklist	If you turn on Collection Blacklist, you must configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcard characters. Important If you use wildcard characters to configure File Path and you want to skip some directories in the specified directory, you must configure Collection Blacklist and enter a complete directory. For example, if you set File Path to /home/admin/app*/log/*.log and you want to skip all subdirectories in the /home/admin/app1* directory, you must select Directory Blacklist and enter /home/admin/app1*/** in the Directory Name field. If you enter /home/admin/app1*, the blacklist does not take effect. When a blacklist is in use, computational overhead is generated. We recommend that you add up to 10 entries to the blacklist. You cannot specify a directory path that ends with a forward slash (/). For example, if you set the path to /home/admin/dir1/, the directory blacklist does not take effect. The following types of blacklists are supported: File Path Blacklist, File Blacklist, and Directory Blacklist. File Path Blacklist If you select File Path Blacklist and enter /home/admin/private*.log in the File Path Name field, all files whose names are prefixed by private and suffixed by .log in the /home/admin/ directory are skipped. If you select File Path Blacklist and enter /home/admin/private*/*_inner.log in the File Path Name field, all files whose names are suffixed by _inner.log in the subdirectories whose names are prefixed by private in the /home/admin/ directory are skipped. For example, the /home/admin/private/app_inner.log file is skipped, but the /home/admin/private/app.log file is not skipped. File Blacklist If you select File Blacklist and enter app_inner.log in the File Name field, all files whose names are app_inner.log are skipped. Directory Blacklist If you select Directory Blacklist and enter /home/admin/dir1 in the Directory Name field, all files in the /home/admin/dir1 directory are skipped. If you select Directory Blacklist and enter /home/admin/dir* in the Directory Name field, the files in all subdirectories whose names are prefixed by dir in the /home/admin/ directory are skipped. If you select Directory Blacklist and enter /home/admin/*/dir in the Directory Name field, all files in the dir subdirectory in each second-level subdirectory of the /home/admin/ directory are skipped. For example, the files in the /home/admin/a/dir directory are skipped, but the files in the /home/admin/a/b/dir directory are not skipped.
   Allow File to Be Collected for Multiple Times	By default, you can use only one Logtail configuration to collect logs from a log file. To use multiple Logtail configurations to collect logs from a log file, turn on Allow File to Be Collected for Multiple Times.

   Processor Configurations

   Each Logtail plug-in defines a processing method. After you add Logtail plug-ins, Logtail calls the processing methods in sequence and parses logs into structured data. You can use the native Logtail plug-ins to perform the following operations. For more information, see Limits.

   • Parsing in regex mode

   • Parsing in JSON mode

   • Parsing in delimiter mode

   • Parsing in NGINX mode

   • Parsing in Apache mode

   • Parsing in IIS mode

   • Time parsing

   • Data filtering

   • Data masking

   If the structure of your business logs is complex or not fixed and native plug-ins cannot meet your log parsing requirements, you can use extended Logtail plug-ins. For more information, see Overview.

   Click Next to complete the Logtail configuration creation. Then, Simple Log Service starts to collect logs.

   Note

   Wait until the Logtail configuration takes effect, which requires up to 3 minutes.

7. Preview data, configure indexes, and then click Next.

   By default, full-text indexing is enabled for Simple Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Simple Log Service automatically creates field indexes. For more information, see Create indexes.

   Important

   If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

8. Click Log Query. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Default fields

Troubleshooting

What do I do if an error occurs when I use Logtail to collect logs from containers?