A Logtail configuration file is a set of policies that Logtail uses to collect logs. The configuration file specifies the directory of the files from which logs are collected and the collection method. This topic describes how to configure Logtail when you use the API to collect data.

Logtail configuration parameters

The following table lists the Logtail configuration parameters for log collection.
Parameter Type Required Description
configName String Yes The name of the Logtail configuration file. The name must be unique in a project.
  • The name can contain lowercase letters, digits, hyphens (-), and underscores (_).
  • The name must start and end with a lowercase letter or digit.
  • The name must be 3 to 128 bytes in length.
logSample String No The sample log entry.
inputType String Yes The method that is used to collect logs. Valid values: plugin and file.
inputDetail Object Yes The configuration details for log collection.
outputType String Yes The output type of collected logs. Valid value: LogService.
outputDetail Object Yes The output configuration details for collected logs.
createTime Int32 No The time when the Logtail configuration file was created. This parameter is returned by the server and cannot be set.
lastModifyTime Int32 No The time when the Logtail configuration file was last modified. This parameter is returned by the server and cannot be set.
The following tables describe the attributes in the inputDetail and outputDetail objects.
  • inputDetail
    • Basic configurations
      Attribute Type Required Description
      filterKey Array No The keys that are used to filter logs. A log entry is collected only when the values of the keys match the corresponding regular expressions that are specified by the filterRegex parameter.
      filterRegex Array No The regular expressions that are used to match the values of the keys that are specified by the filterKey parameter. The length of filterRegex must be the same as that of filterKey.
      shardHashKey Array No The mode of data writes. By default, data is written to Log Service in the load balancing mode. If you specify this parameter, log data is written in the shard hash key mode. Valid values: __topic__, __hostname__, and __source__.
      enableRawLog Boolean No Specifies whether to upload raw logs.
      sensitive_keys Array No The desensitization configuration, in the format of an array. For more information, see the following table.
      mergeType String No The aggregation method. Default value: topic. This value indicates that data is aggregated by topic. Valid values: topic and logstore.
      delayAlarmBytes Integer No The threshold of log collection delay. Default value: 209715200. This value indicates that an alert is triggered if the delay exceeds 200 MB.
      adjustTimezone Boolean No Specifies whether to modify the time zone of logs. This parameter is required only when time parsing is configured.
      logTimezone String No The offset of the time zone. If the time zone of logs is UTC+8, set this parameter to GMT+08:00.
      priority Integer No The priority for sending logs. Default value: 0. If you need to specify a high priority for a Logtail configuration file, set this parameter to 1.
      The following table describes the attributes of the sensitive_keys array.
      • Attributes
        Attribute Type Required Description
        key String Yes The name of the key.
        type String Yes The desensitization method. Valid values: const and md5.
        • If this attribute is set to const, the sensitive content is replaced with the value of the const attribute.
        • If this attribute is set to md5, the sensitive content is replaced with its corresponding MD5 value.
        regex_begin String Yes The prefix of the sensitive content.
        regex_content String Yes The regular expression that is used to match the sensitive content.
        all Boolean Yes Specifies whether to replace all sensitive content in the field that is specified by the key attribute. We recommend that you set this attribute to true.
        const String No The content that is used to replace the sensitive content. This attribute is required if you set the attribute type to const.
      • Configuration example
        For example, a log entry contains the content field, whose value is [{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]. To desensitize password, set sensitive_keys in the following format:
        "key" : "content"
        "type" : "const"
        "regex_begin" : "'password':'"
        "regex_content" : "[^']*"
        "all" : true
        "const" : "********"
                                                        
        The following content is the desensitized log entry:
        [{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]
    • Logtail configurations for collecting text logs
      • Basic configurations
        Attribute Type Required Description
        logType String Yes The log collection mode. You must specify this parameter if you set the inputType parameter to file. Valid values:
        • json_log: collects logs in the JSON mode.
        • common_reg_log: collects logs in the full regex mode.
        • delimiter_log: collects logs in the delimiter mode.
        logPath String Yes The parent directory of the logs to be collected, for example, /var/logs/.
        filePattern String Yes The pattern of a log file name, for example, access*.log.
        topicFormat String Yes The topic generation method. Valid values:
        • none: The topic of the collected logs is a null string.
        • default: The topic of the collected logs is the directory of the logs.
        • group_topic: The topic of the collected logs is the groupTopic attribute of the machine group to which this configuration file is applied.
        • A part of the log directory: The topic of the collected logs is a part of the log directory, for example, /var/log/(. *).log.
        timeFormat String No The format of the log time, for example, %Y/%m/%d %H:%M:%S.
        preserve Boolean No Specifies whether the monitored directory times out. Default value: true. This value indicates that the monitored directory never times out. The value false indicates that the timeout duration for the monitored directory is 30 minutes.
        preserveDepth Integer No Specifies the depth of the monitored directories if the preserve parameter is set to false. Maximum value: 3.
        fileEncoding String No The file encoding method. Valid values: utf8 and gbk.
        discardUnmatch Boolean No Specifies whether to discard the logs that fail to match the specified filter conditions.
        maxDepth Integer No The depth of the monitored directories. Valid values: 0 to 1000. The value 0 indicates that only the directory that is specified in the log directory is monitored.
        delaySkipBytes Integer No The threshold of lagged data. Unit: bytes. If the threshold is exceeded, the lagged data is discarded. Default value: 0. This value indicates that lagged data is not discarded.
        dockerFile Boolean No Specifies whether logs are collected from containers. Default value: false.
        dockerIncludeLabel Object No The whitelist of container labels. Logtail collects logs from Docker containers whose labels are in the whitelist. If you do not specify this parameter, Logtail collects logs from all containers.
        dockerExcludeLabel Object No The blacklist of container labels. Logtail does not collect logs from Docker containers whose labels are in the blacklist. If you do not specify this parameter, Logtail collects logs from all containers.
        dockerIncludeEnv Object No The whitelist of container environment variables. Logtail collects logs from Docker containers whose environment variables are in the whitelist. If you do not specify this parameter, Logtail collects logs from all containers.
        dockerExcludeEnv Object No The blacklist of container environment variables. Logtail does not collect logs from Docker containers whose environment variables are in the blacklist. If you do not specify this parameter, Logtail collects logs from all containers.
      • Configurations that are specific to log collection in the full regex and simple mode
        Attribute Type Required Description
        key Array Yes The list of keys in the log extraction results.
        logBeginRegex String No The regular expression that is used to match the first line of a log entry.
        regex String No The regular expression that is used to extract the value of a field.
        Note Log collection in the simple mode is a special case of log collection in the full regex mode. The following example is sample configurations of log collection in the simple mode.
        "key" : ["content"]
        "logBeginRegex" : ".*"
        "regex" : "(. *)"
      • Configurations that are specific to log collection in the JSON mode
        Attribute Type Required Description
        timeKey String No The key that specifies the time field.
      • Configurations that are specific to log collection in the delimiter mode
        Attribute Type Required Description
        separator String No The delimiter that is used to delimit the fields of each log entry.
        quote String Yes The quote that is used to enclose log fields that contain the delimiter.
        key Array Yes The list of keys in the log extraction results.
        timeKey String Yes The key that specifies the time field, which must be in the key list.
        autoExtend Boolean No Specifies whether to automatically increase the number of keys if the number of parsed log fields is greater than the number of configured keys.
    • Configure plug-ins
      The following table describes the configuration that is specific to log collection by using a Logtail plug-in.
      Attribute Type Required Description
      plugin Object Yes The plug-in configurations, in the format of a JSON object. For more information, see custom plug-in.
  • outputDetail
    The following table describes how to configure the project and Logstore for output logs.
    Attribute Type Required Description
    projectName String Yes The name of the project. The name must be the same as the requested project.
    logstoreName String Yes The name of the Logstore.

Logtail configuration examples

  • Configuration example for log collection in the JSON mode
    {
        "configName": "logConfigName", 
        "outputType": "LogService", 
        "inputType": "file", 
        "inputDetail": {
            "logPath": "/logPath", 
            "filePattern": "access.log", 
            "logType": "json_log", 
            "topicFormat": "default", 
            "discardUnmatch": false, 
            "enableRawLog": true, 
            "fileEncoding": "gbk", 
            "maxDepth": 10
        }, 
        "outputDetail": {
            "projectName": "test-project", 
            "logstoreName": "test-logstore"
        }
    }
  • Configuration example for log collection in the delimiter mode
    {
        "configName": "logConfigName", 
        "logSample": "testlog", 
        "inputType": "file", 
        "outputType": "LogService", 
        "inputDetail": {
            "logPath": "/logPath", 
            "filePattern": "*", 
            "logType": "delimiter_log", 
            "topicFormat": "default", 
            "discardUnmatch": true, 
            "enableRawLog": true, 
            "fileEncoding": "utf8", 
            "maxDepth": 999, 
            "separator": ",", 
            "quote": "\"", 
            "key": [
                "test", 
                "test2"
            ], 
            "autoExtend": true
        }, 
        "outputDetail": {
            "projectName": "test-project", 
            "logstoreName": "test-logstore"
        }
    }
  • Configuration example for log collection in the full regex mode
    {
        "configName": "logConfigName", 
        "outputType": "LogService", 
        "inputType": "file", 
        "inputDetail": {
            "logPath": "/logPath", 
            "filePattern": "*", 
            "logType": "common_reg_log", 
            "topicFormat": "default", 
            "discardUnmatch": false, 
            "enableRawLog": true, 
            "fileEncoding": "utf8", 
            "maxDepth": 10, 
            "key": [
                "content"
            ], 
            "logBeginRegex": ".*", 
            "regex": "(.*)"
        }, 
        "outputDetail": {
            "projectName": "test-project", 
            "logstoreName": "test-logstore"
        }
    }
  • Configuration example for docker stdout log collection by using a plug-in
    {
        "configName": "logConfigName", 
        "outputType": "LogService", 
        "inputType": "plugin",
        "inputDetail": {
            "plugin": {
                "inputs": [
                    {
                        "detail": {
                            "ExcludeEnv": null, 
                            "ExcludeLabel": null, 
                            "IncludeEnv": null, 
                            "IncludeLabel": null, 
                            "Stderr": true, 
                            "Stdout": true
                        }, 
                        "type": "service_docker_stdout"
                    }
                ]
            }
        }, 
        "outputDetail": {
            "projectName": "test-project", 
            "logstoreName": "test-logstore"
        }
    }