Log Service:Use CRDs to collect container logs in DaemonSet mode

Collect stdout and stderr from all containers except the containers whose environment variable configurations include COLLECT_STDOUT_FLAG=false. To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: simple-stdout-example
spec:
  # If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-stdout
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect stdout and stderr, you must set the value to plugin. 
    inputType: plugin
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: simple-stdout-example
    inputDetail:
      plugin:
        inputs:
          -
            # input type
            type: service_docker_stdout
            detail:
              # The settings that allow Logtail to collect both stdout and stderr. 
              Stdout: true
              Stderr: true
              # The environment variable blacklist. In this example, stdout and stderr are collected from all containers except the containers whose environment variable configurations include COLLECT_STDOUT_FLAG=false. 
              ExcludeEnv:
                COLLECT_STDOUT_FLAG: "false"

Collect the access logs of Grafana from containers in simple mode and parse the access logs into structured data by using regular expressions. The environment variable configurations of a container from which the access logs of Grafana can be collected include GF_INSTALL_PLUGINS=grafana-piechart-..... To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.

t=2018-03-09T07:14:03+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=172.16.64.154 time_ms=0 size=29 referer=

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: regex-stdout-example
spec:
  # If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-stdout-regex
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect stdout and stderr, you must set the value to plugin. 
    inputType: plugin
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: regex-stdout-example
    inputDetail:
      plugin:
        inputs:
          -
            # input type
            type: service_docker_stdout
            detail:
              # The settings that allow Logtail to collect only stdout. 
              Stdout: true
              Stderr: false
              # The environment variable whitelist. In this example, stdout is collected only from containers whose environment variable configurations include a key of GF_INSTALL_PLUGINS. 
              IncludeEnv:
                GF_INSTALL_PLUGINS: ''
        processors:
          -
            # Use the processor_regex plug-in to parse the collected stdout by using a regular expression. 
            type: processor_regex
            detail:
              # The name of the source field. By default, the collected stdout is stored in the content field. 
              SourceKey: content
              # The regular expression that is used to extract log content. 
              Regex: 't=(\d+-\d+-\w+:\d+:\d+\+\d+) lvl=(\w+) msg="([^"]+)" logger=(\w+) userId=(\w+) orgId=(\w+) uname=(\S*) method=(\w+) path=(\S+) status=(\d+) remote_addr=(\S+) time_ms=(\d+) size=(\d+) referer=(\S*).*'
              # The keys that you want to extract from logs. 
              Keys: ['time', 'level', 'message', 'logger', 'userId', 'orgId', 'uname', 'method', 'path', 'status', 'remote_addr', 'time_ms', 'size', 'referer']
              # The settings that allow Logtail to retain the source field. 
              KeepSource: true
              # The settings that allow Logtail to report an error when the specified source field does not exist. 
              NoKeyError: true
              # The settings that allow Logtail to report an error when the specified regular expression does not match the value of the specified source field. 
              NoMatchError: true

1. Obtain Kubernetes labels.
2. Create a Logtail configuration.
   The following code shows an example of a Logtail configuration. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster. 
     name: simple-stdout-example
   spec:
     # If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
     logstore: k8s-stdout
     # The Logtail configuration. 
     logtailConfig:
       # The type of the data source. If you want to collect stdout and stderr, you must set the value to plugin. 
       inputType: plugin
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
       configName: k8s-stdout-example
       inputDetail:
         plugin:
           inputs:
             -
               # input type
               type: service_docker_stdout
               detail:
                 # The settings that allow Logtail to collect both stdout and stderr. 
                 Stdout: true
                 Stderr: true
                 # The Kubernetes label whitelist. In this example, stdout and stderr are collected only from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo. 
                 IncludeK8sLabel:
                   job-name: "^(nginx-log-demo.*)$"

1. Obtain different levels of Kubernetes information.
   1. Obtain information about pods.
   2. Obtain information about namespaces.
2. Create a Logtail configuration.
   The following code shows an example of a Logtail configuration. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster. 
     name: simple-stdout-example
   spec:
     # If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
     logstore: k8s-stdout
     # The Logtail configuration. 
     logtailConfig:
       # The type of the data source. If you want to collect stdout and stderr, you must set the value to plugin. 
       inputType: plugin
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
       configName: k8s-stdout-example
       inputDetail:
         plugin:
           inputs:
             -
               # input type
               type: service_docker_stdout
               detail:
                 # The settings that allow Logtail to collect both stdout and stderr. 
                 Stdout: true
                 Stderr: true
                 K8sNamespaceRegex: ^(default)$
                 K8sPodRegex: '^(nginx-log-demo.*)$'
                 K8sContainerRegex: ^(nginx-log-demo-0)$
                           

Collect text logs from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. The log file path is /data/logs/app_1/simple.LOG.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: simple-file-example
spec:
   # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: simple-file-example
    inputDetail:
      # The settings that allow Logtail to collect text logs in simple mode. 
      logType: common_reg_log
      # The log file path. 
      logPath: /data/logs/app_1
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: simple.LOG
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

A Java program generates a multi-line log that contains error stack information. You can collect the log in full regex mode and specify a regular expression that is used to match the beginning of the first line of the log in the Logtail configuration.

[2018-05-11T20:10:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions
java.sql.SQLException: Incorrect string value: '\xF0\x9F\x8E\x8F",...' for column 'data' at row 1
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
at org.springframework.jdbc.support.AbstractFallbackSQLException

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: regex-file-example
spec:
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: regex-file-example
    inputDetail:
      # The settings that allow Logtail to collect text logs in full regex mode. 
      logType: common_reg_log
      # The log file path. 
      logPath: /app/logs
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: error.LOG
      # The regular expression that is used to match the beginning of the first line of a log. 
      logBeginRegex: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*'
      # The regular expression that is used to extract log content. 
      regex: '\[([^]]+)]\s\[(\w+)]\s\[(\w+)]\s\[([^:]+):(\d+)]\s(.*)'
      # The keys that you want to extract from logs. 
      key: ["time", "level", "method", "file", "line", "message"]
      # The format of the time values that are extracted from logs. When logs are collected in full regex mode, the time values are extracted from the time field of the logs by default. If you do not want to extract time values, you can leave this parameter empty. If you configure the timeFormat parameter, you must also configure the adjustTimezone and logTimezone parameters. 
      timeFormat: '%Y-%m-%dT%H:%M:%S'
      # By default, Logtail uses UTC. You must configure the following parameter setting before you can forcefully change the time zone: 
      adjustTimezone: true
      # The time zone offset. The time zone of logs is UTC+8. You can change the value of this parameter to change the time zone. 
      logTimezone: "GMT+08:00"
      # The settings that allow Logtail to upload raw logs if the logs fail to be parsed. 
      discardUnmatch: false
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

If the container text logs that you want to collect contain delimiters, you can collect the container text logs in delimiter mode. Logs that are in the delimiter-separated values (DSV) format use line feeds as boundaries. Each log is placed in a separate line. Each log is parsed into multiple fields by using delimiters.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: delimiter-file-example
spec:
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    configName: delimiter-file-example
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    inputDetail:
      # The settings that allow Logtail to collect text logs in delimiter mode. 
      logType: delimiter_log
      # The log file path. 
      logPath: /usr/local/ilogtail
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: delimiter_log.LOG
      # The delimiters. 
      separator: '|&|'
      # The keys that you want to extract from logs. 
      key: ['time', 'level', 'method', 'file', 'line', 'message']
      # The name of the field from which time values are extracted. 
      timeKey: 'time'
      # The format of the time values that are extracted from logs. When logs are collected in delimiter mode, the time values are extracted from the time field of the logs by default. If you do not want to extract time values, you can leave this parameter empty. If you configure the timeFormat parameter, you must also configure the adjustTimezone and logTimezone parameters. 
      timeFormat: '%Y-%m-%dT%H:%M:%S'
      # By default, Logtail uses UTC. You must configure the following parameter setting before you can forcefully change the time zone: 
      adjustTimezone: true
      # The time zone offset. The time zone of logs is UTC+8. You can change the value of this parameter to change the time zone. 
      logTimezone: "GMT+08:00"
      # The settings that allow Logtail to upload raw logs if the logs fail to be parsed. 
      discardUnmatch: false
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ''

If the container text logs that you want to collect are JSON logs of the Object type, you can collect the container text logs in JSON mode.

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2020:13:30:28"}

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: json-file-example
spec:
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: json-file-example
    inputDetail:
      # The settings that allow Logtail to collect text logs in JSON mode. 
      logType: json_log
      # The log file path. 
      logPath: /usr/local/ilogtail
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: json_log.LOG
      # The name of the field from which time values are extracted. If no requirements are specified, use the timeKey: '' setting. 
      timeKey: 'time'
      # The format of the time values that are extracted from logs. If no requirements are specified, use the timeFormat: '' setting. 
      timeFormat: '%Y-%m-%dT%H:%M:%S'
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

1. Obtain different levels of Kubernetes information.
   1. Obtain information about pods.
   2. Obtain information about namespaces.
   3. Obtain Kubernetes labels.
2. Create a Logtail configuration

   The following code shows an example of a Logtail configuration. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

   Note You must configure Kubernetes-related parameters in the k8s parameter below advanced. The Kubernetes-related parameters include IncludeK8sLabel, ExcludeK8sLabel, K8sNamespaceRegex, K8sPodRegex, K8sContainerRegex, ExternalEnvTag, and ExternalK8sLabelTag. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster. 
     name: simple-file-example
   spec:
      # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
     logstore: k8s-file
     # The Logtail configuration. 
     logtailConfig:
       # The type of the data source. If you want to collect text logs, you must set the value to file. 
       inputType: file
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
       configName: simple-file-example
       inputDetail:
         # The settings that allow Logtail to collect text logs in simple mode. 
         logType: common_reg_log
         # The log file path. 
         logPath: /data/logs/app_1
         # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
         filePattern: simple.LOG
         # If you want to collect container text logs, you must set dockerFile to true. 
         dockerFile: true
         # The conditions that are used to filter containers. 
         advanced:
           k8s:
             K8sNamespaceRegex: ^(default)$
             K8sPodRegex: '^(nginx-log-demo.*)$'
             K8sContainerRegex: ^(nginx-log-demo-0)$
             IncludeK8sLabel:
               job-name: "^(nginx-log-demo.*)$"