Configure Kubernetes log collection on CRD

apiVersion: log.alibabacloud.com/v1alpha1 ## Default value, no need for change
kind: AliyunLogConfig ## Default value, no need for change
metadata:
  name: simple-stdout-example ## Resource name, which must be unique in the cluster
spec:
  logstore: k8s-stdout ## Logstore name, automatically created if no name exists
  shardCount: 2 ## [Optional] Number of Logstore shards. The default value is 2. The value range is 1 to 10.
  lifeCycle: 90 ## [Optional] Storage period of the Logstore. The default value is 90. The value range is 1 to 7300. The value 7300 indicates permanent storage.
  logtailConfig: ## Detailed configuration
    inputType: plugin ## Input type of collection. Generally, the value is file or plugin.
    configName: simple-stdout-example ## Collection configuration name. The value must the same as the resource name (metadata.name).
    inputDetail: ## Detailed configuration information, see the example
      ...

• Simple file
  Collect log files from containers whose environment variable configuration contains key ALIYUN_LOGTAIL_USER_DEFINED_ID. The log file path is /data/logs/app_1 and the file name is simple.LOG.

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # your config name, must be unique in your k8s cluster
    name: simple-file-example
  spec:
    # logstore name to upload log
    logstore: k8s-file
    # logtail config detail
    logtailConfig:
      # log file's input type is 'file'
      inputType: file
      # logtail config name, must same with [metadata.name]
      configName: simple-file-example
      inputDetail:
        # Set logType to "common_reg_log" for simple mode logs
        logType: common_reg_log
        # Log file folder
        logPath: /data/logs/app_1
        # File name, which supports wildcards, for example, log_*.log
        filePattern: simple.LOG
        # Collect files from the container. dockerFile flag is set to true
        dockerFile: true
        # Only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

• Complete regular expression files
  The following is an example of a Java program log:

  [2018-05-11T20:10:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions
  java.sql.SQLException: Incorrect string value: '\xF0\x9F\x8E\x8F",...' for column 'data' at row 1
  at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
  at org.springframework.jdbc.support.AbstractFallbackSQLException

  A log entry may be divided into multiple lines because the log contains error stacking information. Therefore, you must set a regular expression for the beginning of a line. To extract each field, use a regular expression. The detailed configuration is as follows:

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # your config name, must be unique in your k8s cluster
    name: regex-file-example
  spec:
    # logstore name to upload log
    logstore: k8s-file
    logtailConfig:
      # log file's input type is 'file'
      inputType: file
      # logtail config name, should be same with [metadata.name]
      configName: regex-file-example
      inputDetail:
        # Set logType to "common_reg_log" for logs of the regular expression type.
        logType: common_reg_log
        # Log file folder
        logPath: /app/logs
        # File name, which supports wildcards, for example, log_*.log
        filePattern: error.LOG
        # Regular expression for first line 
        logBeginRegex: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s. *'
        # Parse the regular expression
        regex: '\[([^]]+)]\s\[(\w+)]\s\[(\w+)]\s\[([^:]+):(\d+)]\s(. *)'
        # List of extracted keys
        key : ["time", "level", "method", "file", "line", "message"]
        # Logs in regular expression. `time` in the logs are extracted for time parsing by default. If time is not required, ignore the field.
        timeFormat: '%Y-%m-%dT%H:%M:%S'
        # Collect files from the container. dockerFile flag is set to true
        dockerFile: true
        # Only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

  After the configuration is applied, the data collected by Log Service is as follows:

  
  
  

• Delimiter pattern file
  Logtail supports log parsing in delimiter mode, an example is as follows:

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # your config name, must be unique in your k8s cluster
    name: delimiter-file-example
  spec:
    # logstore name to upload log
    logstore: k8s-file
    logtailConfig:
      # log file's input type is 'file'
      inputType: file
      configName: delimiter-file-example
      # logtail config name, should be same with [metadata.name]
      inputDetail:
        # Set logType to delimiter_log for logs of the delimiter type
        logType: delimiter_log
        # Log file folder
        logPath: /usr/local/ilogtail
        # File name, which supports wildcards, for example, log_*.log
        filePattern: delimiter_log.LOG
        # Use a multi-character delimiter
        separator: '|&|'
        # List of extracted keys
        key : ['time', 'level', 'method', 'file', 'line', 'message']
        # Keys for parsing time. Ignore the field if time parsing is not required
        timeKey: 'time'
        # Time parsing method. Ignore the field if time parsing is not required
        timeFormat: '%Y-%m-%dT%H:%M:%S'
        # Collect files from the container. dockerFile flag is set to true
        dockerFile: true
        # Only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ''

• JSON mode file
  If each data line in a file is a JSON object, you can use the JSON method for parsing, an example is as follows:

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # your config name, must be unique in you k8s cluster
    name: json-file-example
  spec:
    # logstore name to upload log
    logstore: k8s-file
    logtailConfig:
      # log file's input type is 'file'
      inputType: file
      # logtail config name, should be same with [metadata.name]
      configName: json-file-example
      inputDetail:
        # Set logType to json_log for logs of the delimiter type
        logType: json_log
        # Log file folder
        logPath: /usr/local/ilogtail
        # File name, which supports wildcards, for example, log_*.log
        filePattern: json_log.LOG
        # Keys for parsing time. Ignore the field if time parsing is not required
        timeKey: 'time'
        # Time parsing method. Ignore the field if time parsing is not required
        timeFormat: '%Y-%m-%dT%H:%M:%S'
        # Collect files from the container. dockerFile flag is set to true
        dockerFile: true
        # Only collect container with "ALIYUN_LOGTAIL_USER_DEFINED_ID" in docker env config
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""