Use CRDs to collect container logs in DaemonSet mode

Implementation

Limits

• Limits on text log collection
  • If Logtail detects the die event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency occurs, some text logs that are generated before the container is stopped may be lost.
  • Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
  • If a volume is mounted to the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.

    For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.

  • By default, Kubernetes mounts the root directory of the host to the /logtail_host directory of the Logtail container. If you want to collect text logs from the host, you must specify /logtail_host as the prefix of the log file path.

    For example, if you want to collect logs from the /home/logs/app_log/ directory of the host, you must specify /logtail_host/home/logs/app_log/ as the log file path.

  • For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.
• Limits on stdout and stderr log collection

  The logging driver collects stdout and stderr logs only in the JSON format from containers that use the Docker engine.

• General limits
  Logtail collects data from containers that use the Docker engine or containerd engine.

  • Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
  • containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

Create a Logtail configuration

To create a Logtail configuration, you need to only create an AliyunLogConfig CRD. After the Logtail configuration is created, Logtail automatically collects logs to Log Service based on the Logtail configuration. If you want to delete the Logtail configuration, you need to only delete the CRD.

1. Log on to your Kubernetes cluster.
2. Run the following command to create a YAML file.

   In this example, the file name is cube.yaml. You can specify a file name based on your business requirements.

   vim cube.yaml

3. Enter the following script in the YAML file and configure the parameters based on your business requirements.
   Notice

   • The value of the configName parameter must be unique in the Log Service project that you use.
   • If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify one of the CRDs. After the deletion or modification, the status of the other CRDs that are associated with the Logtail configuration becomes inconsistent with the status of the Logtail configuration in Log Service.

   apiVersion: log.alibabacloud.com/v1alpha1      # The default value is used. You do not need to modify this parameter. 
   kind: AliyunLogConfig                          # The default value is used. You do not need to modify this parameter. 
   metadata:
     name: simple-stdout-example                  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
   spec:
     project: k8s-my-project                      # Optional. The name of the project. The default value is the name of the project that you use to install the Logtail component. 
     logstore: k8s-stdout                         # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
     shardCount: 2                                # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
     lifeCycle: 90                                # Optional. The data retention period for the Logstore. The value of this parameter takes effect only when you create a Logstore. Valid values: 1 to 3650. Unit: days. Default value: 90. A value of 3650 specifies that log data is permanently stored in the Logstore. 
     logtailConfig:                               # The Logtail configuration. 
       inputType: plugin                          # The type of the data source. Valid values: file and plugin. A value of file specifies text logs. A value of plugin specifies stdout and stderr logs. 
       configName: simple-stdout-example          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
       inputDetail:                               # The detailed settings of the Logtail configuration. For more information, see the following configuration examples. 
         ...

   Parameter	Data type	Required	Description
   project	string	No	The name of the project. The default value is the name of the project that you use to install the Logtail component.
   logstore	string	Yes	The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore.
   shardCount	int	No	The number of shards. Valid values: 1 to 10. Default value: 2.
   lifeCycle	int	No	The data retention period for the Logstore. Valid values: 1 to 3650. Unit: days. Default value: 90. A value of 3650 specifies that log data is permanently stored in the Logstore. Notice The value of this parameter takes effect only when you create a Logstore. If you change the value of the lifeCycle parameter for an existing Logstore that is specified by the logstore parameter, the new value does not take effect.
   machineGroups	array	No	The machine group to which the Logtail configuration is applied. The default value is the machine group named k8s-group-${your_k8s_cluster_id}. This machine group is automatically created by Log Service when you install the Logtail component.
   logtailConfig	object	Yes	The detailed settings of the Logtail configuration. In most cases, you need to only configure the inputType, configName, and inputDetail parameters. For more information about the parameters, see Logtail configurations. For more information about configuration examples, see Examples of Logtail configurations that are used to collect stdout and stderr logs and Examples of Logtail configurations that are used to collect text logs.

4. Run the following command to apply the Logtail configuration.

   In this example, the file name is cube.yaml. You can specify a file name based on your business requirements.

   kubectl apply -f cube.yaml

   After the Logtail configuration is applied, Logtail collects stdout and stderr logs or text logs from each container, and then sends the collected logs to Log Service.

View Logtail configurations

View all Logtail configurations in the current Kubernetes cluster

You can run the kubectl get aliyunlogconfigs command to view all Logtail configurations. The following figure shows the result.

View the details and status of a Logtail configuration

You can run the kubectl get aliyunlogconfigs config_name -o yaml command to view the details and status of a Logtail configuration. The config_name field in the command specifies the name of the Logtail configuration that you want to view. You can specify a name based on your business requirements. The following figure shows the result.

Examples of Logtail configurations that are used to collect stdout and stderr logs

If you want to collect container stdout and stderr logs, you must set the inputType parameter to plugin and add detailed settings to the plugin field of the inputDetail parameter. For more information about the parameters and the descriptions of the parameters, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

Example 1: Collect container stdout and stderr logs in simple mode

Collect stdout and stderr logs from all containers except the containers whose environment variable configurations include COLLECT_STDOUT_FLAG=false. To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables. CRD configuration example:

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: simple-stdout-example
spec:
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-stdout
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect stdout and stderr logs, you must set the value to plugin. 
    inputType: plugin
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: simple-stdout-example
    inputDetail:
      plugin:
        inputs:
          -
            # input type
            type: service_docker_stdout
            detail:
              # The settings that allow Logtail to collect stdout and stderr logs. 
              Stdout: true
              Stderr: true
              # The environment variable blacklist. In this example, stdout and stderr logs are collected from all containers except the containers whose environment variable configurations include COLLECT_STDOUT_FLAG=false. 
              ExcludeEnv:
                COLLECT_STDOUT_FLAG: "false"

Example 2: Collect container stdout logs in simple mode and process the logs by using regular expressions

To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.

Examples of Logtail configurations that are used to collect text logs

If you want to collect container text logs, you must set the inputType parameter to file and add detailed settings to the inputDetail parameter. For more information about the parameters and the descriptions of the parameters, see Use the Log Service console to collect container text logs in DaemonSet mode.

Example 1: Collect container text logs in simple mode

Collect container text logs whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. The log file path is /data/logs/app_1/simple.LOG.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: simple-file-example
spec:
   # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    configName: simple-file-example
    inputDetail:
      # The settings that allow Logtail to collect text logs in simple mode. 
      logType: common_reg_log
      # The log file path. 
      logPath: /data/logs/app_1
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: simple.LOG
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

Example 2: Collect container text logs in full regex mode

A Java program generates a multi-line log that contains error stack information. You can collect the log in full regex mode and specify a regular expression that is used to match the beginning of the first line of the log in the Logtail configuration.

• Sample log

  [2018-05-11T20:10:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions
  java.sql.SQLException: Incorrect string value: '\xF0\x9F\x8E\x8F",...' for column 'data' at row 1
  at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
  at org.springframework.jdbc.support.AbstractFallbackSQLException

• CRD configuration

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # The name of the resource. The name must be unique in the current Kubernetes cluster. 
    name: regex-file-example
  spec:
    # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
    logstore: k8s-file
    logtailConfig:
      # The type of the data source. If you want to collect text logs, you must set the value to file. 
      inputType: file
      # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
      configName: regex-file-example
      inputDetail:
        # The settings that allow Logtail to collect text logs in full regex mode. 
        logType: common_reg_log
        # The log file path. 
        logPath: /app/logs
        # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
        filePattern: error.LOG
        # The regular expression that is used to match the beginning of the first line of the log. 
        logBeginRegex: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*'
        # The regular expression that is used to extract log content. 
        regex: '\[([^]]+)]\s\[(\w+)]\s\[(\w+)]\s\[([^:]+):(\d+)]\s(.*)'
        # The keys that you want to extract from logs. 
        key : ["time", "level", "method", "file", "line", "message"]
        # The format of the time values that are extracted from logs. When logs are collected in full regex mode, the time values are extracted from the time field of the logs by default. If you do not want to extract time values, you can leave this parameter empty. If you configure the timeFormat parameter, you must also configure the adjustTimezone and logTimezone parameters. 
        timeFormat: '%Y-%m-%dT%H:%M:%S'
        # By default, Logtail uses UTC. You must configure the following parameter before you can forcefully change the time zone: 
        adjustTimezone: true
        # The time zone offset. The time zone of logs is UTC+8. You can change the value of this parameter to change the time zone. 
        logTimezone: "GMT+08:00"
        # The settings that allow Logtail to upload raw logs if the logs fail to be parsed. 
        discardUnmatch: false
        # If you want to collect container text logs, you must set dockerFile to true. 
        dockerFile: true
        # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""

• Collected log

Example 3: Collect container text logs in delimiter mode

If the container text logs that you want to collect contain delimiters, you can collect the container text logs in delimiter mode. Logs that are in the delimiter-separated values (DSV) format use line feeds as boundaries. Each log is placed in a separate line. Each log is parsed into multiple fields by using delimiters.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
  name: delimiter-file-example
spec:
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: k8s-file
  logtailConfig:
    # The type of the data source. If you want to collect text logs, you must set the value to file. 
    inputType: file
    configName: delimiter-file-example
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
    inputDetail:
      # The settings that allow Logtail to collect text logs in delimiter mode. 
      logType: delimiter_log
      # The log file path. 
      logPath: /usr/local/ilogtail
      # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
      filePattern: delimiter_log.LOG
      # The delimiter. 
      separator: '|&|'
      # The keys that you want to extract from logs. 
      key : ['time', 'level', 'method', 'file', 'line', 'message']
      # The name of the field from which time values are extracted. 
      timeKey: 'time'
      # The format of the time values that are extracted from logs. When logs are collected in delimiter mode, the time values are extracted from the time field of the logs by default. If you do not want to extract time values, you can leave this parameter empty. If you configure the timeFormat parameter, you must also configure the adjustTimezone and logTimezone parameters. 
      timeFormat: '%Y-%m-%dT%H:%M:%S'
      # By default, Logtail uses UTC. You must configure the following parameter before you can forcefully change the time zone: 
      adjustTimezone: true
      # The time zone offset. The time zone of logs is UTC+8. You can change the value of this parameter to change the time zone. 
      logTimezone: "GMT+08:00"
      # The settings that allow Logtail to upload raw logs if the logs fail to be parsed. 
      discardUnmatch: false
      # If you want to collect container text logs, you must set dockerFile to true. 
      dockerFile: true
      # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
      dockerIncludeEnv:
        ALIYUN_LOGTAIL_USER_DEFINED_ID: ''

Example 4: Collect container text logs in JSON mode

If the container text logs that you want to collect are JSON logs of the Object type, you can collect the container text logs in JSON mode.

• Raw log

  {"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2020:13:30:28"}

• CRD configuration

  apiVersion: log.alibabacloud.com/v1alpha1
  kind: AliyunLogConfig
  metadata:
    # The name of the resource. The name must be unique in the current Kubernetes cluster. 
    name: json-file-example
  spec:
    # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
    logstore: k8s-file
    logtailConfig:
      # The type of the data source. If you want to collect text logs, you must set the value to file. 
      inputType: file
      # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. 
      configName: json-file-example
      inputDetail:
        # The settings that allow Logtail to collect text logs in JSON mode. 
        logType: json_log
        # The log file path. 
        logPath: /usr/local/ilogtail
        # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
        filePattern: json_log.LOG
        # The name of the field from which time values are extracted. If no requirements are specified, set the value to timeKey: ''. 
        timeKey: 'time'
        # The format of the time values that are extracted from logs. If no requirements are specified, set the value to timeFormat: ''. 
        timeFormat: '%Y-%m-%dT%H:%M:%S'
        # If you want to collect container text logs, you must set dockerFile to true. 
        dockerFile: true
        # The environment variable whitelist. In this example, text logs are collected only from containers whose environment variable configurations include a key of ALIYUN_LOGTAIL_USER_DEFINED_ID. 
        dockerIncludeEnv:
          ALIYUN_LOGTAIL_USER_DEFINED_ID: ""