Use the Simple Log Service console to collect container text logs in DaemonSet mode - Simple Log Service

If you want to use a separate Logtail process to collect logs from all containers on a Kubernetes node, you can install Logtail in a Kubernetes cluster in DaemonSet mode. This topic describes the implementation, limits, prerequisites, and procedure of collecting container text logs in DaemonSet mode.

Implementation

DaemonSet mode

In DaemonSet mode, only one Logtail container runs on a node in a Kubernetes cluster. You can use Logtail to collect logs from all containers on a node.
When a node is added to a Kubernetes cluster, the cluster automatically creates a Logtail container on the new node. When a node is removed from a Kubernetes cluster, the cluster automatically destroys the Logtail container on the current node. DaemonSets and custom identifier-based machine groups eliminate the need to manually manage Logtail processes.

Container discovery

Before a Logtail container can collect logs from other containers, the Logtail container must identify and determine which containers are running. This process is called container discovery.
In the container discovery phase, the Logtail container does not communicate with the kube-apiserver component of the Kubernetes cluster. Instead, the Logtail container communicates with the container runtime daemon of the node on which the Logtail container runs to obtain information about all containers on the node. This prevents the container discovery process from generating pressure on the kube-apiserver component. kube-apiserver is the central management component of a Kubernetes cluster and is responsible for providing API services. For more information, see kube-apiserver.
When you use Logtail to collect logs, you can specify conditions such as namespaces, pod names, pod labels, and container environment variables to determine containers from which Logtail collects or does not collect logs.

File path mapping for containers

Pods in a Kubernetes cluster are isolated. As a result, the Logtail container in a pod cannot directly access the files of containers in a different pod. The file system of a container is created by mounting the file system of the container host to the container. A Logtail container can access any file on the container host only after the file system that includes the root directory of the container host is mounted to the Logtail container. This way, the Logtail container can collect logs from the files in the file system. The relationship between file paths in a container and file paths on the container host is called file path mapping.

For example, a file path in a container is /log/app.log. After file path mapping, the file path on the container host is /var/lib/docker/containers/<container-id>/log/app.log. By default, the file system that includes the root directory of the container host is mounted to the /logtail_host directory of the Logtail container. Therefore, the Logtail container collects logs from /logtail_host/var/lib/docker/containers/<container-id>/log/app.log.

Methods to manage Logtail configurations

Use the Simple Log Service console

The procedure is simple. You do not need to connect to your Kubernetes cluster or use an Alibaba Cloud account for authorization.

(Recommended) Use a CRD

Custom resource definitions (CRDs) allow you to create custom Kubernetes resources. Simple Log Service defines a CRD named AliyunLogConfig, which is equivalent to a Logtail configuration. The CRD allows you to manage your Logtail configuration in cloud-native mode. You can submit a YAML configuration file to the kube-apiserver component to create an AliyunLogConfig CRD. Then, you can create a Logtail configuration. In this case, you must connect to your Kubernetes cluster and complete Kubernetes-related authorization.

Warning

If you use a CRD to create a Logtail configuration and modify the Logtail configuration in the Simple Log Service console, the modification is not synchronized to the CRD. If you want to modify a Logtail configuration that is created by using a CRD, you must modify the CRD. If you modify the configuration in the Simple Log Service console, Logtail configuration inconsistency may occur.

Implementation of CRD-based Logtail configuration management

Simple Log Service creates a Deployment to install the alibaba-log-controller component. The component monitors the changes in the AliyunLogConfig CRD.
If you use kubectl or other Kubernetes management tools to create, update, or delete the AliyunLogConfig CRD, the alibaba-log-controller component monitors the changes and submits related requests to Simple Log Service based on the CRD configuration file and the status of the Logtail configuration in Simple Log Service. For example, the component may request to update the Logtail configuration.

Limits

Container runtime: Logtail can collect data only from the containers that use the Docker engine or containerd engine. For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.
Volume mounting: If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot install Logtail in DaemonSet mode. In this case, we recommend that you install Logtail in Sidecar or Deployment mode. Then, Logtail can collect logs. For more information, see Collect text logs from Kubernetes containers in Sidecar mode and Share a PVC between an application container and a Logtail container to collect logs.
Log file path:
- The symbolic links of file paths in containers are not supported. You must specify an actual path as the collection directory.
- If a volume is mounted to the data directory of an application container, you must specify the data directory or its subdirectory as the collection directory. For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service or its subdirectory as the collection directory.
Log collection stopping:
- Docker: When a container is stopped, Logtail immediately releases the current container file handles. Then, the container can exit as expected. If collection latency exists due to issues such as network latency or resource overconsumption before the container is stopped, some logs that are collected before the container is stopped may be lost.
- containerd: When a container is stopped, Logtail does not release the current container file handles until all logs are sent. In this period, the files keep open. If collection latency exists due to issues such as network latency or resource overconsumption, the container may fail to be destroyed in a timely manner.

Prerequisites

The Logtail components are installed. For more information, see Install Logtail components in an ACK cluster.
A Logstore is created in the project that you use to install the Logtail components. For more information, see Create a Logstore.
Ports 80 and 443 are enabled for the server on which Logtail is installed. If the server is an Elastic Computing Service (ECS) instance, you can reconfigure the related security group rules to enable the ports. For more information about how to configure a security group rule, see Add a security group rule.
Logs are continuously generated in the container from which you want to collect logs. Logtail collects only incremental logs. If a log file on your server is not updated after a Logtail configuration is delivered and applied to the server, Logtail does not collect logs from the file. For more information, see Read log files.
The required UNIX domain socket exists, and Logtail has access permissions on the UNIX domain socket. The UNIX domain socket varies based on the container engine.
- Docker: /run/docker.sock
- containerd: /run/containerd/containerd.sock

Create a Logtail configuration

Warning

Console

Log on to the Simple Log Service console.
On the right side of the page that appears, click Quick Data Import. In the Import Data dialog box, click the Kubernetes - File card.
Select the required project and Logstore. Then, click Next. In this example, select the project that you use to install the Logtail components and the Logstore that you create.
In the Machine Group Configurations step, perform the following operations:
1. Use one of the following settings based on your business requirements:
  - Kubernetes Clusters > ACK Daemonset
  - Kubernetes Clusters > Self-managed Cluster in DaemonSet Mode
    Important
    Subsequent settings vary based on the preceding settings.
2. Confirm that the required machine groups are added to the Applied Server Groups section. Then, click Next. After you install Logtail components in a Container Service for Kubernetes (ACK) cluster, Simple Log Service automatically creates a machine group named k8s-group-${your_k8s_cluster_id}. You can directly use this machine group.
  Important
  - If you want to create a machine group, click Create Machine Group. In the panel that appears, configure the parameters to create a machine group. For more information, see Use the wizard in the ACK console to create an application and configure Simple Log Service.
  - If the heartbeat status of a machine group is FAIL, click Automatic Retry. If the issue persists, see How do I troubleshoot an error that is related to a Logtail machine group in a host environment?

Create a Logtail configuration and click Next. Simple Log Service starts to collect logs after the Logtail configuration is created.

Note

A Logtail configuration requires up to 3 minutes to take effect.

Global Configurations

Parameter	Description
Configuration Name	Enter a name for the Logtail configuration. The name must be unique in a project. After you create the Logtail configuration, you cannot change the name of the Logtail configuration.
Log Topic Type	Select a method to generate log topics. For more information, see Log topics. Machine Group Topic: The topics of the machine groups are used as log topics. If you want to distinguish the logs from different machine groups, select this option. File Path Extraction: You must specify a custom regular expression. A part of the file path that matches the regular expression is used as the log topic. If you want to distinguish the logs from different sources, select this option. Custom: You must specify a custom log topic.
Advanced Parameters	Optional. Configure the advanced parameters that are related to global configurations. For more information, see CreateLogtailPipelineConfig.

Input Configurations

Parameter	Description
Logtail Deployment Mode	Select the deployment mode of Logtail. For this example, select Daemonset.
File Path Type	Select the type of the file path that you want to use to collect logs. Valid values: Path in Container and Host Path. If a hostPath volume is mounted to a container and you want to collect logs from files based on the mapped file path on the container host, set this parameter to Host Path. In other scenarios, set this parameter to Path in Container.
File Path	If the required container runs on a Linux host, specify a path that starts with a forward slash (/). Example: `/apsara/nuwa//app.Log`. If the required container runs on a Windows host, specify a path that starts with a drive letter. Example: `C:\Program Files\Intel\\.Log`. You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. For more information, see Wildcard matching. When you configure this parameter, you can use only asterisks () or question marks (?) as wildcard characters. Simple Log Service scans all levels of the specified directory for the log files that match specified conditions. Examples: If you specify `/apsara/nuwa/*/.log`, Simple Log Service collects logs from the log files whose names are suffixed by .log in the `/apsara/nuwa` directory and the recursive subdirectories of the directory. If you specify `/var/logs/app_//.log`, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by `.log`. The file is stored in a subdirectory under the `/var/logs` directory or in a recursive subdirectory of the subdirectory. The name of the subdirectory matches the `app_` pattern. If you specify `/var/log/nginx//access`, Simple Log Service collects logs from the log files whose names start with `access` in the `/var/log/nginx` directory and the recursive subdirectories of the directory.
Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched for the wildcard characters `` included in the value of File Path. A value of 0 specifies that only the log file directory that you specify is monitored. Warning** We recommend that you configure this parameter based on the minimum requirement. If you specify a large value, Logtail may consume more monitoring resources and cause collection latency.
Enable Container Metadata Preview	If you turn on Enable Container Metadata Preview, you can view the container metadata after you create the Logtail configuration, including the matched container information and full container information.
Container Filtering	Logtail version If the version of Logtail is earlier than 1.0.34, you can use only environment variables and container labels to filter containers. If the version of Logtail is 1.0.34 or later, we recommend that you use different levels of Kubernetes information to filter containers. The information includes pod names, namespaces, container names, and container labels. Filter conditions Important Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels. Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables. A Kubernetes namespace and a container name can be mapped to container labels. The label for a namespace is `io.kubernetes.pod.namespace`. The label for a container name is `io.kubernetes.container.name`. We recommend that you use the two labels to filter containers. For example, the namespace of a pod is `backend-prod`, and the name of a container in the pod is `worker-server`. If you want to collect the logs of the worker-server container, you can specify `io.kubernetes.pod.namespace : backend-prod` or `io.kubernetes.container.name : worker-server` in the container label whitelist. If the two labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. K8s Pod Name Regular Matching Enter the pod name. The pod name specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify `^(nginx-log-demo.)$`, all containers in the pod whose name starts with nginx-log-demo are matched. K8s Namespace Regular Matching* Enter the namespace. The namespace specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify `^(default\|nginx)$`, all containers in the nginx and default namespaces are matched. K8s Container Name Regular Matching Enter the container name. The container name specifies the containers from which text logs are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify `^(container-test)$`, all containers whose name is container-test are matched. Container Label Whitelist Configure a container label whitelist. The whitelist specifies the containers from which text logs are collected. Note Do not specify duplicate values for the Label Name parameter. If you specify duplicate values, only one value takes effect. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose container labels contain the specified label name are matched. If you specify a value for the Label Name and Tag Value parameters, containers whose container labels contain the specified `Label Name:Tag Value` are matched. By default, string matching is performed for the values of the Tag Value parameter. Containers are matched only if the values of the container labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)` for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to `app` and set the Tag Value parameter to `^(test1\|test2)$`, containers whose container labels contain `app:test1` or `app:test2` are matched. Key-value pairs are evaluated by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is matched. Container Label Blacklist Configure a container label blacklist. The blacklist specifies the containers from which text logs are not collected. Note Do not specify duplicate values for the Label Name parameter. If you specify duplicate values, only one value takes effect. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose container labels contain the specified label name are filtered out. If you specify a value for the Label Name and Tag Value parameters, containers whose container labels contain the specified `Label Name:Tag Value` are filtered out. By default, string matching is performed for the values of the Tag Value parameter. Containers are filtered out only if the values of the container labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)` for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to `app` and set the Tag Value parameter to `^(test1\|test2)$`, containers whose container labels contain app:test1 or app:test2 are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has a container label that consists of one of the specified key-value pairs, the container is filtered out. Environment Variable Whitelist Configure an environment variable whitelist. The whitelist specifies the containers from which text logs are collected. If you specify a value for the Environment Variable Name parameter but do not specify a value for the Environment Variable Value parameter, containers whose environment variables contain the specified environment variable name are matched. If you specify a value for the Environment Variable Name and Environment Variable Value parameters, containers whose environment variables contain the specified Environment Variable Name:Environment Variable Value are matched. By default, string matching is performed for the values of the Environment Variable Value parameter. Containers are matched only if the values of the environment variables are the same as the values of the Environment Variable Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)` for the Environment Variable Value parameter, regular expression matching is performed. For example, if you set the Environment Variable Name parameter to `NGINX_SERVICE_PORT` and set the Environment Variable Value parameter to `^(80\|6379)$`, containers whose port number is 80 or 6379 are matched. Key-value pairs are evaluated by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is matched. Environment Variable Blacklist Configure an environment variable blacklist. The blacklist specifies the containers from which text logs are not collected. If you specify a value for the Environment Variable Name parameter but do not specify a value for the Environment Variable Value parameter, containers whose environment variables contain the specified environment variable name are filtered out. If you specify a value for the Environment Variable Name and Environment Variable Value parameters, containers whose environment variables contain the specified Environment Variable Name:Environment Variable Value are filtered out. By default, string matching is performed for the values of the Environment Variable Value parameter. Containers are filtered out only if the values of the environment variables are the same as the values of the Environment Variable Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)` for the Environment Variable Value parameter, regular expression matching is performed. For example, if you set the Environment Variable Name parameter to `NGINX_SERVICE_PORT` and set the Environment Variable Value parameter to `^(80\|6379)$`, containers whose port number is 80 or 6379 are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has an environment variable that consists of one of the specified key-value pairs, the container is filtered out. Kubernetes Pod Label Whitelist Configure a Kubernetes pod label whitelist. The whitelist specifies the containers from which text logs are collected. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose pod labels contain the specified label name are matched. If you specify a value for the Label Name and Tag Value parameters, containers whose pod labels contain the specified `Label Name:Tag Value` are matched. By default, string matching is performed for the values of the Tag Value parameter. Containers are matched only if the values of the pod labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)`, regular expression matching is performed. For example, if you set the Label Name parameter to `environment` and set the Tag Value parameter to `^(dev\|pre)$`, containers whose pod labels contain `environment:dev` or `environment:pre` are matched. Key-value pairs are evaluated by using the OR operator. If a container has a pod label that consists of one of the specified key-value pairs, the container is matched. Kubernetes Pod Label Blacklist Configure a Kubernetes pod label blacklist. The blacklist specifies the containers from which text logs are not collected. If you specify a value for the Label Name parameter but do not specify a value for the Tag Value parameter, containers whose pod labels contain the specified label name are filtered out. If you specify a value for the Label Name and Tag Value parameters, containers whose pod labels contain the specified Label Name:Tag Value are filtered out. By default, string matching is performed for the values of the Tag Value parameter. Containers are filtered out only if the values of the pod labels are the same as the values of the Tag Value parameter. If you specify a value that starts with a caret `(^)` and ends with a dollar sign `($)` for the Tag Value parameter, regular expression matching is performed. For example, if you set the Label Name parameter to `environment` and set the Tag Value parameter to `^(dev\|pre)$`, containers whose pod labels contain `environment:dev` or `environment:pre` are filtered out. Key-value pairs are evaluated by using the OR operator. If a container has a pod label that consists of one of the specified key-value pairs, the container is filtered out.
Log Tag Enrichment	Specify log tags by using environment variables and pod labels.
File Encoding	Select the encoding format of log files.
First Collection Size	Specify the size of data that Logtail can collect from a log file the first time Logtail collects logs from the file. The default value of First Collection Size is 1024. Unit: KB. If the file size is less than 1,024 KB, Logtail collects data from the beginning of the file. If the file size is greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file. You can specify First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.
Collection Blacklist	If you turn on Collection Blacklist, you must configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only asterisks () or question marks (?) as wildcard characters. Important* If you use wildcard characters to configure File Path and you want to skip some directories in the specified directory, you must configure Collection Blacklist and enter a complete directory. For example, if you set File Path to `/home/admin/app/log/.log` and you want to skip all subdirectories in the `/home/admin/app1` directory, you must select Directory Blacklist* and enter `/home/admin/app1/` in the Directory Name field. If you enter `/home/admin/app1`, the blacklist does not take effect. When a blacklist is in use, computational overhead is generated. We recommend that you add up to 10 entries to the blacklist. You cannot specify a directory path that ends with a forward slash (/). For example, if you set the path to `/home/admin/dir1/`, the directory blacklist does not take effect. The following types of blacklists are supported: File Path Blacklist, File Blacklist, and Directory Blacklist. File Path Blacklist If you select File Path Blacklist and enter `/home/admin/private.log` in the File Path Name field, all files whose names are prefixed by private and suffixed by .log in the `/home/admin/` directory are skipped. If you select File Path Blacklist* and enter `/home/admin/private/_inner.log` in the File Path Name field, all files whose names are suffixed by _inner.log in the subdirectories whose names are prefixed by private in the `/home/admin/` directory are skipped. For example, the `/home/admin/private/app_inner.log` file is skipped, but the `/home/admin/private/app.log` file is not skipped. File Blacklist If you select File Blacklist and enter `app_inner.log` in the File Name field, all files whose names are `app_inner.log` are skipped. Directory Blacklist If you select Directory Blacklist and enter `/home/admin/dir1` in the Directory Name field, all files in the `/home/admin/dir1` directory are skipped. If you select Directory Blacklist and enter `/home/admin/dir` in the Directory Name field, the files in all subdirectories whose names are prefixed by dir in the `/home/admin/` directory are skipped. If you select Directory Blacklist* and enter `/home/admin/*/dir` in the Directory Name field, all files in the dir subdirectory in each second-level subdirectory of the `/home/admin/` directory are skipped. For example, the files in the `/home/admin/a/dir` directory are skipped, but the files in the `/home/admin/a/b/dir` directory are not skipped.
Allow File to Be Collected for Multiple Times	By default, you can use only one Logtail configuration to collect logs from a log file. To use multiple Logtail configurations to collect logs from a log file, turn on Allow File to Be Collected for Multiple Times.
Advanced Parameters	You must manually configure specific parameters of a Logtail configuration. For more information, see Create a Logtail pipeline configuration.

Processor Configurations

Parameter	Description
Log Sample	Add a sample log that is collected from an actual scenario. You can use the sample log to configure parameters that are related to log processing with ease. You can add multiple sample logs. Make sure that the total length of the logs does not exceed 1,500 characters. `[2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16)`
Multi-line Mode	Specify the type of multi-line logs. A multi-line log spans multiple consecutive lines. You can configure this parameter to identify each multi-line log in a log file. Custom: A multi-line log is identified based on the value of Regex to Match First Line. Multi-line JSON: Each JSON object is expanded into multiple lines. Example: `{ "name": "John Doe", "age": 30, "address": { "city": "New York", "country": "USA" } }` Configure Processing Method If Splitting Fails. `Exception in thread "main" java.lang.NullPointerException at com.example.MyClass.methodA(MyClass.java:12) at com.example.MyClass.methodB(MyClass.java:34) at com.example.MyClass.main(MyClass.java:½0)` For the preceding sample log, Simple Log Service can discard the log or retain each single line as a log when it fails to split the log. Discard: The log is discarded. Retain Single Line: Each line of log text is retained as a log. A total of four logs are retained.
Processing Method	Select Processors. You can add native plug-ins and extended plug-ins for data processing. For more information about Logtail plug-ins for data processing, see Logtail plug-ins overview. Important You are subject to the limits of Logtail plug-ins for data processing. For more information, see the on-screen instructions in the Simple Log Service console. Logtail earlier than V2.0 You cannot add native plug-ins and extended plug-ins at the same time. You can use native plug-ins only to collect text logs. When you add native plug-ins, take note of the following items: You must add one of the following Logtail plug-ins for data processing as the first plug-in: Data Parsing (Regex Mode), Data Parsing (Delimiter Mode), Data Parsing (JSON Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode). After you add the first plug-in, you can add one Time Parsing plug-in, one Data Filtering plug-in, and multiple Data Masking plug-ins. You can add extended plug-ins only after you add native plug-ins. Logtail V2.0 You can arbitrarily combine native plug-ins for data processing. You can combine native plug-ins and extended plug-ins. Make sure that extended plug-ins are added after native plug-ins.

Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Simple Log Service automatically creates field indexes. For more information, see Create indexes.
Important
If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.
Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

CRD

Create a Logtail configuration

To create a Logtail configuration, you need to only create an AliyunLogConfig CRD. After the Logtail configuration is created, the system automatically applies the Logtail configuration. If you want to delete the Logtail configuration, you need to only delete the CRD.

Log on to your Kubernetes cluster.
Run the following command to create a YAML file.
In the following command, cube.yaml is a sample file name. You can specify a different file name based on your business requirements.
```
vim cube.yaml
```

Enter the following script in the YAML file and configure the parameters based on your business requirements.

Important

The value of the configName parameter must be unique in the Simple Log Service project that you use to install the Logtail components.
If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify one of the CRDs. After a CRD is deleted or modified, the status of other associated CRDs becomes inconsistent with the status of the Logtail configuration in Simple Log Service.

apiVersion: log.alibabacloud.com/v1alpha1      # The default value is used. You do not need to modify this parameter. 
kind: AliyunLogConfig                          # The default value is used. You do not need to modify this parameter. 
metadata:
  name: simple-stdout-example                  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
spec:
  project: k8s-my-project                      # Optional. The name of the project. The default value is the name of the project that you use to install the Logtail components. 
  logstore: k8s-stdout                         # The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore. 
  logstoreMode: standard                       # Optional. The type of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. 
  shardCount: 2                                # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
  lifeCycle: 90                                # The data retention period of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. Valid values: 1 to 3650. Default value: 90. Unit: days. The value 3650 specifies that data is permanently stored in the Logstore. 
  logtailConfig:                               # The Logtail configuration. 
    inputType: plugin                          # The type of the data source. Valid values: file and plugin. The value file specifies text logs. The value plugin specifies stdout and stderr. 
    configName: simple-stdout-example          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    inputDetail:                               # The detailed settings of the Logtail configuration. For more information, see the following configuration examples. 
      ...

Parameters

Parameter	Type	Required	Description
`project`	string	No	The name of the project. The default value is the name of the project that you use to install the Logtail components.
`logstore`	string	Yes	The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore.
`logstoreMode`	string	No	The type of the Logstore. For more information, see Manage a Logstore. Valid values: query: Query Logstore. standard: Standard Logstore. Important The value of this parameter takes effect only if you configure the parameter when you create the Logstore. If you change the value of this parameter for an existing Logstore that is specified by the `logstore` parameter, the new value does not take effect. The value of this parameter takes effect only in alibaba-log-controller V0.3.3 and later.
`shardCount`	int	No	The number of shards. Valid values: 1 to 10. Default value: 2.
`lifeCycle`	int	No	The data retention period of the Logstore. Valid values: 1 to 3650. Default value: 90. Unit: days. The value 3650 specifies that data is permanently stored in the Logstore. Important The value of this parameter takes effect only if you configure the parameter when you create the Logstore. You can configure this parameter only when you create a Logstore. If you change the value of this parameter for an existing Logstore that is specified by the `logstore` parameter, the new value does not take effect.
`machineGroups`	array	No	The machine group. When you install Logtail components, Simple Log Service automatically creates a machine group named `k8s-group-${your_k8s_cluster_id}`.
`logtailConfig`	object	Yes	For more information about parameter settings, see Logtail configurations. For more information about configuration examples, see CRD configuration examples. Logtail version If the version of Logtail is earlier than 1.0.34, you can use only environment variables and container labels to filter containers. If the version of Logtail is 1.0.34 or later, we recommend that you use different levels of Kubernetes information to filter containers. The information includes pod names, namespaces, container names, and container labels. Filter conditions Important Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see How do I obtain the labels and environment variables of a container? Environment variables are the same as the environment variables that are configured to start containers. For more information, see How do I obtain the labels and environment variables of a container? Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values, only one value takes effect. A Kubernetes namespace and a container name can be mapped to container labels. The label for a namespace is `io.kubernetes.pod.namespace`. The label for a container name is `io.kubernetes.container.name`. We recommend that you use the two labels to filter containers. For example, the namespace of a pod is `backend-prod`, and the name of a container in the pod is `worker-server`. If you want to collect the logs of the worker-server container, you can specify `io.kubernetes.pod.namespace : backend-prod` or `io.kubernetes.container.name : worker-server` in the container label whitelist. If the two labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers.

Run the following command to apply the Logtail configuration. After the Logtail configuration is applied, Logtail collects stdout and stderr or text logs from each container, and then sends the collected logs to Simple Log Service.
In the following command, cube.yaml is a sample file name. You can specify a different file name based on your business requirements.
```
kubectl apply -f cube.yaml
```
Important
After logs are collected, you must create indexes. Then, you can query and analyze the logs in the Logstore. For more information, see Create indexes.

View Logtail configurations

Console

Log on to the Simple Log Service console.
In the Projects section, click the target project.
Choose Log Storage > Logstores. Click the > icon of the target Logtail configurations. Choose Data Import > Logtail Configurations.
Click the target Logtail configurations to view the details of the configurations.

CRD

View all Logtail configurations for the current Kubernetes cluster

Run the kubectl get aliyunlogconfigs command. The following figure shows sample output.

View the details and status of a Logtail configuration

Run the kubectl get aliyunlogconfigs config_name -o yaml command. The config_name parameter in the command specifies the name of the Logtail configuration that you want to view. You can specify the name based on your business requirements. The following figure shows sample output.

The status and statusCode fields in the output indicate the status of the Logtail configuration.

If the value of the statusCode field is 200, the Logtail configuration is applied.
If the value of the statusCode field is not 200, the Logtail configuration fails to be applied.

Query and analyze the collected logs

In the Projects section of the Simple Log Service console, click the project that you want to manage to go to the details page of the project.
Find the Logstore that you want to manage, move the pointer over the Logstore, click the icon, and then choose Search & Analysis to view the logs of your Kubernetes cluster.

CRD configuration examples

If you want to collect container text logs, set the inputType parameter to file and specify the required information for the inputDetail parameter. For more information about parameter settings, see Create a Logtail configuration.

Filter containers by using Kubernetes information

Description
Collect logs from the simple.LOG file in the /data/logs/app_1 directory of specific containers. The containers must meet the following requirements:
- The containers belong to the default namespace.
- The containers belong to pods whose name starts with nginx-log-demo.
- The containers are named nginx-log-demo-0.
- The containers have Kubernetes labels whose Key is job-name and Value starts with nginx-log-demo.
Note
If you use Logtail V1.0.34 or later, you can use Kubernetes namespaces, pod names, container names, and Kubernetes labels to filter containers.

Example

Obtain different levels of Kubernetes information.
1. Obtain information about pods.
2. Obtain information about namespaces.
3. Obtain Kubernetes labels.

Create a Logtail configuration

The following sample code shows an example of a Logtail configuration. For more information about parameter settings, see the CRD tab in the Create a Logtail configuration section of this topic.

Note

You must configure Kubernetes-related parameters in the k8s parameter below advanced. The Kubernetes-related parameters include IncludeK8sLabel, ExcludeK8sLabel, K8sNamespaceRegex, K8sPodRegex, K8sContainerRegex, ExternalEnvTag, and ExternalK8sLabelTag.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: simple-file-example
spec:
   # The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore. 
  logstore: k8s-file
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect text logs, set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: simple-file-example
    inputDetail:
      # The settings that allow Logtail to collect text logs in simple mode. 
      logType: common_reg_log
      # The log file path. 
      logPath: /data/logs/app_1
      # The log file name. You can use wildcard characters (* and ?) when you specify the log file name. Example: log_*.log. 
      filePattern: simple.LOG
      # If you want to collect text logs from containers, set the value to true. 
      dockerFile: true
      # The conditions that are used to filter containers. 
      advanced:
        k8s:
          K8sNamespaceRegex: ^(default)$
          K8sPodRegex: '^(nginx-log-demo.*)$'
          K8sContainerRegex: ^(nginx-log-demo-0)$
          IncludeK8sLabel:
            job-name: "^(nginx-log-demo.*)$"

Default fields in container text logs

The following table describes the fields that are included by default in each container text log.

Field name	Description
__tag__:__hostname__	The name of the container host.
__tag__:__path__	The log file path in the container.
__tag__:_container_ip_	The IP address of the container.
__tag__:_image_name_	The name of the image that is used by the container.
__tag__:_pod_name_	The name of the pod.
__tag__:_namespace_	The namespace to which the pod belongs.
__tag__:_pod_uid_	The unique identifier (UID) of the pod.

Troubleshooting

If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:

What do I do if errors occur when I collect logs from containers?

Implementation

DaemonSet mode

Container discovery

File path mapping for containers

Methods to manage Logtail configurations

Use the Simple Log Service console

(Recommended) Use a CRD

Limits

Prerequisites

Create a Logtail configuration

Console

File Path Blacklist

File Blacklist

Directory Blacklist

CRD

Create a Logtail configuration

View Logtail configurations

Console

CRD

View all Logtail configurations for the current Kubernetes cluster

View the details and status of a Logtail configuration

Query and analyze the collected logs

CRD configuration examples

Filter containers by using Kubernetes information

Default fields in container text logs

Troubleshooting