After you install Logtail in the DaemonSet mode in a Kubernetes cluster, you can use a Custom Resource Definition (CRD) to configure log collection of Kubernetes clusters. This topic describes how to use CRDs to collect Kubernetes container logs in the DaemonSet mode.
Prerequisites
Procedure

- Use the
kubectl
tool or other tools to create an aliyunlogconfigs CRD. - The alibaba-log-controller package detects the creation of the CRD.
- The alibaba-log-controller Deployment controller sends a request to Log Service to create a Logstore, configure a server group, and configure Logtail based on the CRD.
- Logtail periodically requests the master node where log collection is configured to obtain new or updated configuration files and perform hot reloading.
- Logtail collects stdout and stderr logs or text logs from each container based on the updated configurations.
- Logtail sends the collected logs to Log Service.
Configure log collection
apiVersion: log.alibabacloud.com/v1alpha1 ## The default setting, which you do not need to modify.
kind: AliyunLogConfig ## The default setting, which you do not need to modify.
metadata:
name: simple-stdout-example ## The resource name, which must be unique in the cluster.
spec:
project: k8s-my-project ## The project name. You can specify a project that is not occupied. By default, the project created when you installed the Helm package is selected. The parameter is optional.
logstore: k8s-stdout ## The Logstore name. A Logstore is automatically created if the specified Logstore does not exist.
shardCount: 2 ## The number of Logstore shards. Valid values: 1 to 10. Default value: 2. The parameter is optional.
lifeCycle: 90 ## The retention period for which log data is stored in the Logstore. Unit: days. Valid values: 1 to 7300. Default value: 90. The value 7300 indicates that log data is permanently stored in the Logstore. The parameter is optional.
logtailConfig: ## The Logtail settings.
inputType: plugin ## The type of input data sources. Valid values: file and plugin.
configName: simple-stdout-example ## The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter.
inputDetail: ## The detailed settings of Logtail. For more information, see the examples.
...
For more information about the logtailConfig
parameter, see Logtail configuration. For configuration examples of the logtailConfig
parameter, see Example configurations for collecting stdout and stderr logs and Example configurations for collecting text logs.
After you complete the configurations, Logtail automatically collects and uploads logs to Log Service.
View log collection configurations
- Use the
kubectl get aliyunlogconfigs
command to view all Logtail configurations.[root@iZbp1dsbiaZ ~]# kubectl get aliyunlogconfigs NAME AGE regex-file-example 10s regex-stdout-example 4h simple-file-example 5s
- Use the
kubectl get aliyunlogconfigs ${config_name} -o yaml
command to view the detailed settings and status of a Logtail configuration file. The{config_name}
parameter in the command is the name of the Logtail configuration file that you want to query.The
status
field in the output indicates whether the Logtail configuration file is applied. If thestatusCode
field value is 200, the Logtail configurations are applied. If thestatusCode
field value is not 200, the Logtail configuration file is not applied.[root@iZbp1dsbiaZ ~]# kubectl get aliyunlogconfigs simple-file-example -o yaml apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"log.alibabacloud.com/v1alpha1","kind":"AliyunLogConfig","metadata":{"annotations":{},"name":"simple-file-example","namespace":"default"},"spec":{"logstore":"k8s-file","logtailConfig":{"configName":"simple-file-example","inputDetail":{"dockerFile":true,"dockerIncludeEnv":{"ALIYUN_LOGTAIL_USER_DEFINED_ID":""},"filePattern":"simple.LOG","logPath":"/usr/local/ilogtail","logType":"common_reg_log"},"inputType":"file"}}} clusterName: "" creationTimestamp: 2018-05-17T08:44:46Z generation: 0 name: simple-file-example namespace: default resourceVersion: "21790443" selfLink: /apis/log.alibabacloud.com/v1alpha1/namespaces/default/aliyunlogconfigs/simple-file-example uid: 8d3a09c4-59ae-11e8-851d-00163f008685 spec: lifeCycle: null logstore: k8s-file logtailConfig: configName: simple-file-example inputDetail: dockerFile: true dockerIncludeEnv: ALIYUN_LOGTAIL_USER_DEFINED_ID: "" filePattern: simple.LOG logPath: /usr/local/ilogtail logType: common_reg_log inputType: file machineGroups: null project: "" shardCount: null status: status: OK statusCode: 200
Example configurations for collecting stdout and stderr logs
To collect Kubernetes stdout and stderr logs, set inputType
to plugin
and specify the configuration details in the plugin
field under inputDetail
. For more information, see Use the console to collect Kubernetes stdout and stderr logs in the DaemonSet mode.
- Collect logs in the simple mode
The following example shows how to collect stdout and stderr logs of all containers except the containers whose environment variables include
COLLECT_STDOUT_FLAG=false
.apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: simple-stdout-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-stdout # The detailed Logtail settings. logtailConfig: # The type of input data sources. inputType: plugin # The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter. configName: simple-stdout-example inputDetail: plugin: inputs: - # The type of input data sources. type: service_docker_stdout detail: # Collect stdout and stderr logs. Stdout: true Stderr: true # Collect stdout and stderr logs of all containers except the containers whose environment variables include COLLECT_STDOUT_FLAG=false. ExcludeEnv: COLLECT_STDOUT_FLAG: "false"
- Collect logs in the custom mode
The following example shows how to collect access logs of Grafana and parse the access logs into structured data. The environment variable of the Grafana container is
GF_INSTALL_PLUGINS=grafana-piechart-....
. You can setIncludeEnv
toGF_INSTALL_PLUGINS: ''
to collect only stdout and stderr logs from this container.The format of Grafana access log entries is as follows:t=2018-03-09T07:14:03+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=172.16.64.154 time_ms=0 size=29 referer=
The following sample code shows how to use regular expressions to parse access logs:apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: regex-stdout-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-stdout-regex # The detailed Logtail settings. logtailConfig: # The type of input data sources. inputType: plugin # The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter. configName: regex-stdout-example inputDetail: plugin: inputs: - # The type of input data sources. type: service_docker_stdout detail: # Collect stdout logs, but do not collect stderr logs. Stdout: true Stderr: false # Collect stdout logs of the containers whose environment variable keys include GF_INSTALL_PLUGINS. IncludeEnv: GF_INSTALL_PLUGINS: '' processors: - # Use a regular expression. type: processor_regex detail: # The key of the data collected from the docker is content by default. SourceKey: content # Use a regular expression to extract fields. Regex: 't=(\d+-\d+-\w+:\d+:\d+\+\d+) lvl=(\w+) msg="([^"]+)" logger=(\w+) userId=(\w+) orgId=(\w+) uname=(\S*) method=(\w+) path=(\S+) status=(\d+) remote_addr=(\S+) time_ms=(\d+) size=(\d+) referer=(\S*). *' # The extracted keys. Keys: ['time', 'level', 'message', 'logger', 'userId', 'orgId', 'uname', 'method', 'path', 'status', 'remote_addr', 'time_ms', 'size', 'referer'] # Keep original fields. KeepSource: true NoKeyError: true NoMatchError: true
The following figure shows the data that is collected to Log Service after the configurations are applied:
Example configurations for collecting text logs
To collect Kubernetes text logs, set inputType
to file
and specify the configuration details in the plugin field under inputDetail
. For more information, see Use the console to collect Kubernetes text logs in the DaemonSet mode.
- Collect logs in the simple mode
The following example shows how to collect text logs from containers whose environment variable keys include
ALIYUN_LOGTAIL_USER_DEFINED_ID
. The log path is/data/logs/app_1
and the log file name issimple.LOG
.apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: simple-file-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-file # The detailed Logtail settings. logtailConfig: # The input type. "inputType": "file", # The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter. configName: simple-file-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the log path. logPath: /data/logs/app_1 # Specify the names of log files to be collected. You can include wildcards in the parameter value, for example, log_*.log. filePattern: simple.LOG # Set dockerFile to true. dockerFile: true # Collect logs from containers whose environment variable keys include ALIYUN_LOGTAIL_USER_DEFINED_ID. dockerIncludeEnv: ALIYUN_LOGTAIL_USER_DEFINED_ID: ""
- Use regular expressions to parse collected logs
The following example shows a log entry generated by a Java application. The log entry contains error stack traces, which may divide the log entry into multiple lines. You must specify a regular expression to match the first line of a log entry.
[2018-05-11T20:10:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions java.sql.SQLException: Incorrect string value: '\xF0\x9F\x8E\x8F",...' for column 'data' at row 1 at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) at org.springframework.jdbc.support.AbstractFallbackSQLException
The following script shows an example Logtail configuration file that includes regular expressions:apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: regex-file-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-file logtailConfig: # The type of input data sources. "inputType": "file", # The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter. configName: regex-file-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the log path. logPath: /app/logs # Specify the names of log files to be collected. You can include wildcards in the parameter value, for example, log_*.log. filePattern: error.LOG # The regular expression that is used to match the first line of a log entry. logBeginRegex: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s. *' # The regular expression that is used to parse log entries. regex: '\[([^]]+)]\s\[(\w+)]\s\[(\w+)]\s\[([^:]+):(\d+)]\s(. *)' # Specify the keys extracted from log entries. key : ["time", "level", "method", "file", "line", "message"] # Specify the format of the time field extracted from log entries. The parameter is optional. timeFormat: '%Y-%m-%dT%H:%M:%S' # Set dockerFile to true. dockerFile: true # Collect logs from containers whose environment variable keys include ALIYUN_LOGTAIL_USER_DEFINED_ID. dockerIncludeEnv: ALIYUN_LOGTAIL_USER_DEFINED_ID: ""
The following figure shows the data that is collected to Log Service after the configurations are applied: - Use delimiters to parse collected logs
To use delimiters to parse collected logs, use the following example code:
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: delimiter-file-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-file logtailConfig: # The type of input data sources. "inputType": "file", configName: delimiter-file-example # The name of the Logtail configuration file. This name must be the same as the resource name specified by the metadata.name parameter. inputDetail: # Set logType to delimiter_log. logType: delimiter_log # Specify the log path. logPath: /usr/local/ilogtail # Specify the names of log files to be collected. You can include wildcards in the parameter value, for example, log_*.log. filePattern: delimiter_log.LOG # Use multi-character delimiters. separator: '|&|' # Specify the keys extracted from log entries. key : ['time', 'level', 'method', 'file', 'line', 'message'] # The key used to parse time. The parameter is optional. timeKey: 'time' # The method used to parse time. The parameter is optional. timeFormat: '%Y-%m-%dT%H:%M:%S' # Set dockerFile to true. dockerFile: true # Collect logs from containers whose environment variable keys include ALIYUN_LOGTAIL_USER_DEFINED_ID. dockerIncludeEnv: ALIYUN_LOGTAIL_USER_DEFINED_ID: ''
- Use the JSON method to parse collected logs in the JSON format
If each line of log data in a file is a JSON object, you can parse the file as follows:
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The resource name, which must be unique in the cluster. name: json-file-example spec: # The name of the Logstore to which logs are uploaded. logstore: k8s-file logtailConfig: # The type of input data sources. "inputType": "file", # The name of the Logtail configuration file, which must be the same as the resource name specified by the metadata.name parameter. configName: json-file-example inputDetail: # Set logType to json_log. logType: json_log # Specify the log path. logPath: /usr/local/ilogtail # Specify the names of log files to be collected. You can include wildcards in the parameter value, for example, log_*.log. filePattern: json_log.LOG # Specify the key of the time information extracted from log entries. The parameter is optional. timeKey: 'time' # Specify the format of the time field extracted from log entries. The parameter is optional. timeFormat: '%Y-%m-%dT%H:%M:%S' # Set dockerFile to true. dockerFile: true # Collect logs from containers whose environment variable keys include ALIYUN_LOGTAIL_USER_DEFINED_ID. dockerIncludeEnv: ALIYUN_LOGTAIL_USER_DEFINED_ID: ""