After you install Logtail in DaemonSet mode in a Kubernetes container, you can use a custom resource definition (CRD) to create a Logtail configuration to collect logs from the Kubernetes container. This topic describes how to use CRDs to collect Kubernetes container logs in DaemonSet mode.

Prerequisites

The Helm package alibaba-log-controller is installed. For more information, see Install Logtail.

Implementation

Kubernetes-CRDs implementation
The following steps describe the process to collect logs:
  1. Use the kubectl tool or other tools to create an aliyunlogconfigs CRD.
  2. The alibaba-log-controller package detects the update of the CRD.
  3. The alibaba-log-controller Deployment controller sends a request to Log Service to create a Logstore, create a Logtail configuration, and apply a machine group based on the CRD.
  4. Logtail periodically requests the master node on which the Logtail configuration is created to obtain new or updated Logtail configurations and perform hot reloading.
  5. Logtail collects stdout and stderr logs or text logs from each container based on the updated Logtail configurations.
  6. Logtail sends the collected logs to Log Service.

Create a Logtail configuration

You can configure an AliyunLogConfig CRD to create a Logtail configuration. The following script shows how to configure a CRD. To delete a Logtail configuration, you must delete the corresponding CRD.
apiVersion: log.alibabacloud.com/v1alpha1      ## The default value. You do not need to modify this parameter. 
kind: AliyunLogConfig                          ## The default value. You do not need to modify this parameter. 
metadata:
  name: simple-stdout-example                  ## The resource name. The name must be unique in the cluster. 
spec:
  project: k8s-my-project                      ## Optional. The name of the project. You can specify a project that is not occupied. By default, the project that is configured when you install the Helm package is selected. 
  logstore: k8s-stdout                         ## The name of the Logstore. A Logstore is automatically created if the specified Logstore does not exist. 
  shardCount: 2                                ## Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
  lifeCycle: 90                                ## Optional. The retention period for which log data is stored in the Logstore. Unit: days. Valid values: 1 to 7300. Default value: 90. The value 7300 indicates that log data is permanently stored in the Logstore. 
  logtailConfig:                               ## The details of the Logtail configuration.
    inputType: plugin                          ## The type of the data source. Valid values: file and plugin. 
    configName: simple-stdout-example          ## The name of the Logtail configuration. This name must be the same as the resource name specified by the metadata.name field. 
    inputDetail:                               ## The detailed settings of the Logtail configuration. For more information, see the following examples. 
      ...
Notice The value of the configName field in the CRD must be unique in the specified Log Service Logstore. If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify an associated CRD. In this case, the status of the other CRDs that are associated with the Logtail configuration is inconsistent with the status of the server.

For more information about the logtailConfig field, see Logtail configuration files. For information about the configuration examples of the logtailConfig field, see Configuration examples for collecting stdout and stderr logs and Configuration examples for collecting text logs.

After you create a Logtail configuration, Logtail automatically collects and uploads logs to Log Service based on the Logtail configuration.

View Logtail configurations

You can view Logtail configurations by using Kubernetes CRDs or the Log Service console. For more information about how to view Logtail configurations by using the Log Service console, see Manage Logtail configurations for log collection.
Note If you use the Log Service console to modify the settings of a Logtail configuration that is created by using a CRD, the modified settings are overwritten after you use the CRD to update the Logtail configuration.
  • Run the kubectl get aliyunlogconfigs command to view all Logtail configurations.
    [root@iZbp1dsbiaZ ~]# kubectl get aliyunlogconfigs
    NAME                   AGE
    regex-file-example     10s
    regex-stdout-example   4h
    simple-file-example    5s
  • Run the kubectl get aliyunlogconfigs ${config_name} -o yaml command to view the detailed settings and status of a Logtail configuration. The {config_name} field in the command is the name of the Logtail configuration that you want to view.

    The status field in the output indicates whether the Logtail configuration is applied. If the value of the statusCode field is 200, the Logtail configuration is applied. If the value of the statusCode field is not 200, the Logtail configuration is not applied.

    [root@iZbp1dsbiaZ ~]# kubectl get aliyunlogconfigs simple-file-example -o yaml
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"log.alibabacloud.com/v1alpha1","kind":"AliyunLogConfig","metadata":{"annotations":{},"name":"simple-file-example","namespace":"default"},"spec":{"logstore":"k8s-file","logtailConfig":{"configName":"simple-file-example","inputDetail":{"dockerFile":true,"dockerIncludeEnv":{"ALIYUN_LOGTAIL_USER_DEFINED_ID":""},"filePattern":"simple.LOG","logPath":"/usr/local/ilogtail","logType":"common_reg_log"},"inputType":"file"}}}
    clusterName: ""
    creationTimestamp: 2018-05-17T08:44:46Z
    generation: 0
    name: simple-file-example
    namespace: default
    resourceVersion: "21790443"
    selfLink: /apis/log.alibabacloud.com/v1alpha1/namespaces/default/aliyunlogconfigs/simple-file-example
    uid: 8d3a09c4-59ae-11e8-851d-00163f008685
    spec:
    lifeCycle: null
    logstore: k8s-file
    logtailConfig:
      configName: simple-file-example
      inputDetail:
        dockerFile: true
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""
        filePattern: simple.LOG
        logPath: /usr/local/ilogtail
        logType: common_reg_log
      inputType: file
    machineGroups: null
    project: ""
    shardCount: null
    status:
    status: OK
    statusCode: 200

Configuration examples for collecting stdout and stderr logs

To collect Kubernetes stdout and stderr logs, set the inputType field to plugin and add the detailed settings to the plugin field of the inputDetail field. For more information, see Use the console to collect Kubernetes stdout and stderr logs in DaemonSet mode.

  • Collect logs in simple mode

    The following example shows how to collect stdout and stderr logs of all containers except the containers whose environment variables include COLLECT_STDOUT_FLAG=false.

    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: simple-stdout-example
    spec:
      # logstore name to upload log
      logstore: k8s-stdout
      # logtail config detail
      logtailConfig:
        # docker stdout's input type is 'plugin'
        inputType: plugin
        # logtail config name, should be same with [metadata.name]
        configName: simple-stdout-example
        inputDetail:
          plugin:
            inputs:
              -
                # input type
                type: service_docker_stdout
                detail:
                  # collect stdout and stderr
                  Stdout: true
                  Stderr: true
                  # collect all container's stdout except containers with "COLLECT_STDOUT_FLAG:false" in docker env config
                  ExcludeEnv:
                    COLLECT_STDOUT_FLAG: "false"
  • Collect logs in custom mode

    The following example shows how to collect the access logs of Grafana and parse the access logs into structured data. The environment variable of the Grafana container is GF_INSTALL_PLUGINS=grafana-piechart-..... You can set IncludeEnv to GF_INSTALL_PLUGINS: '' to collect stdout and stderr logs only from this container.

    Collect logs in custom mode
    The following example shows the format of Grafana access logs:
    t=2018-03-09T07:14:03+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=172.16.64.154 time_ms=0 size=29 referer=
    The following sample code shows how to use regular expressions to parse access logs:
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: regex-stdout-example
    spec:
      # logstore name to upload log
      logstore: k8s-stdout-regex
      # logtail config detail
      logtailConfig:
        # docker stdout's input type is 'plugin'
        inputType: plugin
        # logtail config name, should be same with [metadata.name]
        configName: regex-stdout-example
        inputDetail:
          plugin:
            inputs:
              -
                # input type
                type: service_docker_stdout
                detail:
                  # Collect stdout logs, but do not collect stderr logs.
                  Stdout: true
                  Stderr: false
                  # Collect stdout logs of the containers whose environment variable keys include GF_INSTALL_PLUGINS. 
                  IncludeEnv:
                    GF_INSTALL_PLUGINS: ''
            processors:
              -
                # Use a regular expression.
                type: processor_regex
                detail:
                  # By default, the key of the data that is collected from Dockers is content. 
                  SourceKey: content
                  # Use a regular expression to extract fields. 
                  Regex: 't=(\d+-\d+-\w+:\d+:\d+\+\d+) lvl=(\w+) msg="([^"]+)" logger=(\w+) userId=(\w+) orgId=(\w+) uname=(\S*) method=(\w+) path=(\S+) status=(\d+) remote_addr=(\S+) time_ms=(\d+) size=(\d+) referer=(\S*).*'
                  # The extracted keys. 
                  Keys: ['time', 'level', 'message', 'logger', 'userId', 'orgId', 'uname', 'method', 'path', 'status', 'remote_addr', 'time_ms', 'size', 'referer']
                  # Reserve original fields. 
                  KeepSource: true
                  NoKeyError: true
                  NoMatchError: true
    The following figure shows a sample log that is collected and sent to Log Service after the Logtail configuration is applied. Collected log data

Configuration examples for collecting text logs

To collect Kubernetes text logs, set the inputType to file and add the detailed settings to the inputDetail field. For more information, see Use the console to collect Kubernetes text logs in DaemonSet mode.

  • Collect logs in simple mode
    The following example shows how to collect text logs from containers whose environment variable keys include ALIYUN_LOGTAIL_USER_DEFINED_ID. In this example, the log path is /data/logs/app_1 and the log file name is simple.LOG.
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: simple-file-example
    spec:
      # logstore name to upload log
      logstore: k8s-file
      # logtail config detail
      logtailConfig:
        # log file's input type is 'file'
        inputType: file
        # logtail config name, should be same with [metadata.name]
        configName: simple-file-example
        inputDetail:
          # Set logType to common_reg_log. 
          logType: common_reg_log
          # Specify the log path.
          logPath: /data/logs/app_1
          # Specify the names of log files from which you want to collect data. You can include wildcard characters in the field value, for example, log_*.log. 
          filePattern: simple.LOG
          # Set dockerFile to true. 
          dockerFile: true
          # only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
          dockerIncludeEnv:
            ALIYUN_LOGTAIL_USER_DEFINED_ID: ""
  • Collect logs in full regex mode
    The following example shows a log that is generated by a Java application. The log contains error stack traces that may divide the log into multiple lines. You must specify a regular expression to match the start part in the first line of a log.
    [2018-05-11T20:10:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions
    java.sql.SQLException: Incorrect string value: '\xF0\x9F\x8E\x8F",...' for column 'data' at row 1
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
    at org.springframework.jdbc.support.AbstractFallbackSQLException
    The following sample code shows how to use regular expressions to parse logs:
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: regex-file-example
    spec:
      # logstore name to upload log
      logstore: k8s-file
      logtailConfig:
        # log file's input type is 'file'
        inputType: file
        # logtail config name, should be same with [metadata.name]
        configName: regex-file-example
        inputDetail:
          # Set logType to common_reg_log. 
          logType: common_reg_log
          # Specify the log path.
          logPath: /app/logs
          # Specify the names of log files from which you want to collect data. You can include wildcard characters in the field value, for example, log_*.log. 
          filePattern: error.LOG
          # The regular expression that is used to match the start part in the first line of a log.
          logBeginRegex: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*'
          # Use a regular expression to parse logs.
          regex: '\[([^]]+)]\s\[(\w+)]\s\[(\w+)]\s\[([^:]+):(\d+)]\s(.*)'
          # The extracted keys.
          key : ["time", "level", "method", "file", "line", "message"]
          # Specify the format of the time field that is extracted from logs. The field is optional. If you specify the timeFormat field, data is parsed based on the time zone (UTC+0) of the region to which the Logtail container belongs. If the region is in the UTC+8 time zone, the time when a log is generated is 8 hours later than the time when the log is parsed. For example, if you upload a log generated at 11:00 to Log Service, the log is parsed to 19:00 on the same day. 
          timeFormat: '%Y-%m-%dT%H:%M:%S'
          # Set dockerFile to true. 
          dockerFile: true
          # only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
          dockerIncludeEnv:
            ALIYUN_LOGTAIL_USER_DEFINED_ID: ""
    The following figure shows a sample log that is collected and sent to Log Service after the Logtail configuration is applied. Collect logs in full regex mode
  • Collect logs in delimiter mode
    The following sample code shows how to use delimiters to parse logs:
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: delimiter-file-example
    spec:
      # logstore name to upload log
      logstore: k8s-file
      logtailConfig:
        # log file's input type is 'file'
        inputType: file
        configName: delimiter-file-example
        # logtail config name, should be same with [metadata.name]
        inputDetail:
          # Set logType to delimiter_log. 
          logType: delimiter_log
          # Specify the log path.
          logPath: /usr/local/ilogtail
          # Specify the names of log files from which you want to collect data. You can include wildcard characters in the field value, for example, log_*.log. 
          filePattern: delimiter_log.LOG
          # Use multi-character delimiters.
          separator: '|&|'
          # The extracted keys.
          key : ['time', 'level', 'method', 'file', 'line', 'message']
          # The key that is used to parse time. If you do not need to specify a key, enter ''. 
          timeKey: 'time'
          # The method that is used to parse time. If you do not need to specify a method, enter ''. 
          timeFormat: '%Y-%m-%dT%H:%M:%S'
          # Set dockerFile to true. 
          dockerFile: true
          # only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
          dockerIncludeEnv:
            ALIYUN_LOGTAIL_USER_DEFINED_ID: ''
  • Collect logs in JSON mode
    If each line of log data in a file is a JSON object, you can parse the data by using the following sample code:
    apiVersion: log.alibabacloud.com/v1alpha1
    kind: AliyunLogConfig
    metadata:
      # your config name, must be unique in you k8s cluster
      name: json-file-example
    spec:
      # logstore name to upload log
      logstore: k8s-file
      logtailConfig:
        # log file's input type is 'file'
        inputType: file
        # logtail config name, should be same with [metadata.name]
        configName: json-file-example
        inputDetail:
          # Set logType to json_log. 
          logType: json_log
          # Specify the log path.
          logPath: /usr/local/ilogtail
          # Specify the names of log files from which you want to collect data. You can include wildcard characters in the field value, for example, log_*.log. 
          filePattern: json_log.LOG
          # The key that is used to parse time. If you do not need to specify a key, enter ''. 
          timeKey: 'time'
          # The method that is used to parse time. If you do not need to specify a method, enter ''. 
          timeFormat: '%Y-%m-%dT%H:%M:%S'
          # Set dockerFile to true. 
          dockerFile: true
          # only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
          dockerIncludeEnv:
            ALIYUN_LOGTAIL_USER_DEFINED_ID: ""