Customize collection policies by creating Logtail configurations - Simple Log Service

A Logtail configuration is a set of policies that Logtail uses to collect logs. You can customize a collection policy by setting parameters, such as the data source and collection mode, when you create the Logtail configuration. This topic describes the parameters related to Logtail configurations in API mode.

The configuration method for the legacy CRD-AliyunLogConfig is no longer maintained. Use the new AliyunPipelineConfig instead. For more information, see CreateLogtailPipelineConfig - Create a Logtail pipeline configuration.

Basic parameters of a Logtail configuration

Parameter	Type	Required	Example	Description
configName	string	Yes	config-sample	The name of the Logtail configuration. The name must be unique within the project. You cannot change the name after the Logtail configuration is created. The name must meet the following requirements: It can contain only lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit. It must be 2 to 128 characters in length.
inputType	string	Yes	file	The type of data source. Valid values: plugin: Collects logs, such as MySQL binary logging (Binlog) data, using Logtail plug-ins. file: Collects logs from text files using a fixed mode, such as full regex mode or delimiter mode.
inputDetail	JSON object	Yes	None	The detailed configuration of the data source. For more information, see inputDetail parameters.
outputType	string	Yes	LogService	The output destination of collected logs. Only LogService is supported. This means collected data can be uploaded only to Simple Log Service.
outputDetail	JSON object	Yes	None	The detailed configuration for log output. For more information, see outputDetail parameters.
logSample	string	No	None	The sample log. Note The sample log must be less than 1,500 bytes in length.

inputDetail parameters

Basic parameters

Parameter	Type	Required	Example	Description
filterKey	array	No	["ip"]	The fields used to filter logs. A log is collected only if the values of these fields match the regular expressions specified in the filterRegex parameter. Note This parameter is for text log collection only.
filterRegex	array	No	["^10.*"]	The regular expressions corresponding to the filterKey parameter. The number of elements in filterRegex must be the same as the number of elements in filterKey. Note This parameter is for text log collection only.
shardHashKey	array	No	["__source__"]	The data write mode. By default, data is written in load balancing mode. Load balancing (LoadBalance) mode: Automatically writes data to any writable shard in the Logstore with load balancing. This method provides high write availability and is suitable for data consumption scenarios that do not require sequential order. Shard (KeyHash) mode: Adds a Key field to the URL parameters to determine which shard to write data to. This parameter is optional. If not set, the system automatically switches to load balancing mode. For example, you can hash a producer, such as an instance, to a specific shard based on its name. This ensures that data written to and consumed from that shard is strictly ordered. During shard merging and splitting, data with the same key is guaranteed to be in only one shard at any given time. For more information, see Shards. If you configure this parameter, data is written in shard mode. The __source__ field is supported.
enableRawLog	boolean	No	false	Specifies whether to upload raw logs. Valid values: true: Upload raw logs. false (default): Do not upload raw logs.
sensitive_keys	array	No	None	The data masking feature. For more information, see sensitive_keys parameters.
mergeType	string	No	topic	The aggregation method. Valid values: topic (default): Aggregates by topic. logstore: Aggregates by Logstore.
delayAlarmBytes	int	No	209715200	The alert threshold for collection delay. Default value: 209715200, which is 200 MB.
adjustTimezone	boolean	No	false	Specifies whether to adjust the log time zone. This parameter is used only when time parsing is configured, for example, when the timeFormat parameter is set.
logTimezone	string	No	GMT+08:00	The time zone offset. The format is GMT+HH:MM for eastern time zones and GMT-HH:MM for western time zones. For example, if the log time is in UTC+8, set this value to GMT+08:00.
advanced	JSON object	No	None	The advanced features. For more information, see advanced parameters.

sensitive_keys parameters

Parameters

Parameter	Type	Required	Example	Description
key	string	Yes	content	The name of the log field.
type	string	Yes	const	The data masking method. Valid values: const: Replaces sensitive content with the value of the const field. md5: Replaces sensitive content with its corresponding MD5 hash.
regex_begin	string	Yes	'password':'	The regular expression for the prefix of the sensitive content. It is used to find the sensitive content. Use RE2 syntax. For more information, see RE2 syntax.
regex_content	string	Yes	[^']*	The regular expression for the sensitive content. Use RE2 syntax. For more information, see RE2 syntax.
all	boolean	Yes	true	Specifies whether to replace all sensitive content in the field. Valid values: true (recommended): Replaces all sensitive content. false: Replaces only the first part of the content that matches the regular expression.
const	string	No	"********"	This parameter is required if you set type to const.

Configuration example
For example, if the value of the content field in a log is [{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}], you can replace the value of the password field with ******** by setting sensitive_keys as follows.
```
sensitive_keys = [{"all": true,
"const": "********",
"regex_content": "[^']*",
"regex_begin": "'password':'",
"type": "const",
"key": "content"}]                    
```

Sample log

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

advanced parameters

Parameter	Type	Required	Example	Description
enable_root_path_collection	boolean	No	false	Specifies whether to allow data collection from Windows root directories, such as D:\log. Valid values: true: Allowed. false (default): Not allowed. Important* This is a global parameter. If you enable this feature in one Logtail configuration, all Logtail configurations on the server are allowed to collect from root directories until Logtail is restarted. This feature is supported only on Windows Logtail 1.0.0.22 and later.
exactly_once_concurrency	int	No	1	Specifies whether to enable the ExactlyOnce write feature. This feature specifies the number of concurrent sends allowed for a single file. The value must be in the range of 0 to 512. For more information, see Appendix: ExactlyOnce write feature. Valid values: 0: Disables the ExactlyOnce write feature. Other values: Enables the ExactlyOnce write feature and specifies the number of concurrent sends allowed for a single file. Important A large value for this parameter increases memory and disk overhead. Evaluate this parameter based on your local write traffic. Logtail performs local randomization. Even if this parameter value is less than the number of shards on the server, write balancing is ensured and does not need to be perfectly aligned. After you configure this parameter, it takes effect only on new files. This feature is supported only in Logtail 1.0.21 and later.
enable_log_position_meta	boolean	No	true	Specifies whether to add metadata from the original log file to the log. This adds the __tag__:__inode__ and __file_offset__ fields. Valid values: true: Adds the metadata. false: Does not add the metadata. Note This feature is supported only in Logtail 1.0.21 and later.
specified_year	uint	No	0	If the time in the raw log is missing the year, you can set this parameter to complete the log time with the current year or a specified year. Valid values: 0: Uses the current year. A specific year (for example, 2020): Uses the specified year. Note This feature is supported only in Logtail 1.0.21 and later.
force_multiconfig	boolean	No	false	Specifies whether to allow this Logtail configuration to collect files that are already matched by other Logtail configurations. The default value is false, which means it is not allowed. This is applicable to multi-write scenarios, for example, when a file is collected by two collection configurations into different Logstores.
raw_log_tag	string	No	__raw__	The field used to store the raw log when uploading raw logs. The default value is __raw__.
blacklist	object	No	None	The collection blacklist configuration. For more information, see blacklist parameters.
tail_size_kb	int	No	1024	The size of the initial collection for a new file. This size determines the starting position of the collection. The default initial collection size is 1024 KB. During the initial collection, if the file is smaller than 1024 KB, collection starts from the beginning of the file. During the initial collection, if the file is larger than 1024 KB, collection starts from 1024 KB before the end of the file. You can modify the initial collection size here. The value must be in the range of 0 to 10485760. The unit is KB.
batch_send_interval	int	No	3	The aggregation sending period. The unit is seconds. The default value is 3.
max_rotate_queue_size	int	No	20	The length of the single-file rotation queue. The default value is 20.
enable_precise_timestamp	boolean	No	false	Specifies whether to extract high-precision time. If this parameter is not added, the default value is false, which means high-precision time is not extracted. By default, after you set this to true, Logtail parses the value of the specified time field into a millisecond-level timestamp and stores it in the field corresponding to the precise_timestamp_key parameter. Note Make sure that the switch for using system time in the Logtail collection configuration is turned off. This feature is supported only in Logtail 1.0.32 and later.
precise_timestamp_key	string	No	"precise_timestamp"	The field that stores the high-precision timestamp. If this parameter is not added, the default field is precise_timestamp.
precise_timestamp_unit	string	No	"ms"	The unit of the high-precision timestamp. If this parameter is not added, the default is ms. Valid values include ms (millisecond), us (microsecond), and ns (nanosecond).

The following table describes the parameters of blacklist.

Parameter	Type	Required	Example	Description
dir_blacklist	array	No	["/home/admin/dir1", "/home/admin/dir2*"]	The blacklist of directories (absolute paths). You can use an asterisk (*) as a wildcard character to match multiple directories. For example, if you configure the path as /home/admin/dir1, all content in the /home/admin/dir1 directory is ignored during collection.
filename_blacklist	array	No	["app*.log", "password"]	The blacklist of file names. Specified file names are not collected from any directory. You can use an asterisk (*) as a wildcard character to match multiple file names.
filepath_blacklist	array	No	["/home/admin/private*.log"]	The blacklist of file paths (absolute paths). You can use an asterisk () as a wildcard character to match multiple files. If you configure the path as /home/admin/private.log, all files in the /home/admin/ directory that start with "private" and end with ".log" are ignored during collection.

Configurations specific to Logtail for text logs

Basic parameters

Parameter	Type	Required	Example	Description
logType	string	Yes	common_reg_log	The log collection mode. The following modes are supported: json_log: JSON mode. common_reg_log: Full regex mode. plugin: Plug-in mode. delimiter_log: Delimiter mode.
logPath	string	Yes	/var/log/http/	The path of the log file.
filePattern	string	Yes	access*.log	The name of the log file.
topicFormat	string	Yes	none	The topic generation method. Valid values: none: Does not generate log topics. default: Uses the log file path as the log topic. group_topic: Uses the topic of the machine group to which this Logtail configuration is applied as the log topic. File path regular expression: Uses a part of the log file path as the log topic. For example, /var/log/(.*).log. For more information, see Log topics.
timeFormat	string	No	%Y/%m/%d %H:%M:%S	The log time format. For more information, see Time formats.
preserve	boolean	No	true	If a log file is not updated within a specified time, it is considered to have timed out. Valid values: true (default): Never times out. false: If a log file is not updated within 30 minutes, it is considered timed out and is no longer monitored.
preserveDepth	integer	No	1	When you set preserve to false, you must specify the maximum directory depth for the timeout. The value must be in the range of 1 to 3.
fileEncoding	string	No	utf8	The log file encoding format. Valid values are utf8 and gbk.
discardUnmatch	boolean	No	true	Specifies whether to discard logs that fail to be matched. Valid values: true: Discards logs that fail to be matched. false: Does not discard logs that fail to be matched.
maxDepth	int	No	100	The maximum depth of the monitored log directory. The value must be in the range of 0 to 1,000. A value of 0 means only the current directory is monitored.
delaySkipBytes	int	No	0	The threshold for discarding delayed data. Valid values: 0 (default): Does not discard data. Other values: If the collection delay exceeds this value (for example, 1024 KB), the delayed data is discarded.
dockerFile	boolean	No	false	Specifies whether the object file for collection is a file within a container. The default value is false.
dockerIncludeLabel	JSON object	No	None	The container label whitelist, used to specify the containers to be collected. By default, it is empty, which means logs or standard output from all containers are collected. If you want to set a container label whitelist, LabelKey is required and LabelValue is optional. If LabelValue is empty, all containers with a label that includes the LabelKey are matched. If LabelValue is not empty, only containers with a label of LabelKey=LabelValue are matched. By default, LabelValue is matched as a string. A match occurs only if the LabelValue is identical to the container label's value. If the value starts with `^` and ends with `$`, it is treated as a regular expression. For example, set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$` to match containers named nginx or cube. Note Do not set the same LabelKey multiple times. If there are duplicates, only one will take effect. Multiple whitelist entries have an OR relationship. A container is matched if its label satisfies any of the whitelist entries.
dockerExcludeLabel	JSON object	No	None	The container label blacklist, used to exclude containers from collection. By default, it is empty, which means no containers are excluded. If you want to set a container label blacklist, LabelKey is required and LabelValue is optional. If LabelValue is empty, all containers with a label that includes the LabelKey are excluded. If LabelValue is not empty, only containers with a label of LabelKey=LabelValue are excluded. By default, LabelValue is matched as a string. A match occurs only if the LabelValue is identical to the container label's value. If the value starts with `^` and ends with `$`, it is treated as a regular expression. For example, set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$` to match containers named nginx or cube. Note Do not set the same LabelKey multiple times. If there are duplicates, only one will take effect. Multiple blacklist entries have an OR relationship. A container is excluded if its label satisfies any of the blacklist key-value pairs.
dockerIncludeEnv	JSON object	No	None	The environment variable whitelist, used to specify the containers to be collected. By default, it is empty, which means logs or standard output from all containers are collected. If you want to set an environment variable whitelist, EnvKey is required and EnvValue is optional. If EnvValue is empty, all containers with an environment variable that includes the EnvKey are matched. If EnvValue is not empty, only containers with an environment variable of EnvKey=EnvValue are matched. By default, EnvValue is matched as a string. A match occurs only if the EnvValue is identical to the environment variable's value. If the value starts with `^` and ends with `$`, it is treated as a regular expression. For example, set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80\|6379)$` to match containers with a service port of 80 or 6379. Note Multiple whitelist entries have an OR relationship. A container is matched if its environment variables satisfy any of the whitelist entries.
dockerExcludeEnv	JSON object	No	None	The environment variable blacklist, used to exclude containers from collection. By default, it is empty, which means no containers are excluded. If you want to set an environment variable blacklist, EnvKey is required and EnvValue is optional. If EnvValue is empty, all containers with an environment variable that includes the EnvKey are excluded. If EnvValue is not empty, only containers with an environment variable of EnvKey=EnvValue are excluded. By default, EnvValue is matched as a string. A match occurs only if the EnvValue is identical to the environment variable's value. If the value starts with `^` and ends with `$`, it is treated as a regular expression. For example, set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80\|6379)$` to match containers with a service port of 80 or 6379. Note Multiple blacklist entries have an OR relationship. A container is excluded if its environment variables satisfy any of the key-value pairs.

Configurations specific to full regex mode and simple mode

Parameters

Parameter	Type	Required	Example	Description
key	array	Yes	["content"]	A list of fields used to configure fields for the raw log content.
logBeginRegex	string	No	.*	The regular expression for the beginning of a log entry.
regex	string	No	(.*)	The regular expression used to extract fields.

Configuration example

{
    "configName": "logConfigName", 
    "outputType": "LogService", 
    "inputType": "file", 
    "inputDetail": {
        "logPath": "/logPath", 
        "filePattern": "*", 
        "logType": "common_reg_log", 
        "topicFormat": "default", 
        "discardUnmatch": false, 
        "enableRawLog": true, 
        "fileEncoding": "utf8", 
        "maxDepth": 10, 
        "key": [
            "content"
        ], 
        "logBeginRegex": ".*", 
        "regex": "(.*)"
    }, 
    "outputDetail": {
        "projectName": "test-project", 
        "logstoreName": "test-logstore"
    }
}

Configurations specific to JSON mode

Parameter	Type	Required	Example	Description
timeKey	string	No	time	The key name of the time field.

Configurations specific to delimiter mode

Parameters

Parameter	Type	Required	Example	Description
separator	string	No	,	Select the correct separator based on your log format. For more information, see Collect logs in delimiter mode.
quote	string	Yes	\	If a log field contains a separator, you must specify a quote to enclose it. The content enclosed by the quote is parsed by Simple Log Service as a single field. Select the correct quote based on your log format. For more information, see Collect logs in delimiter mode.
key	array	Yes	[ "ip", "time"]	A list of fields used to configure fields for the raw log content.
timeKey	string	Yes	time	Specifies a field from the key list as the time field.
autoExtend	boolean	No	true	Specifies whether to upload parsed fields if the number of fields split from the log is less than the number of configured keys. For example, a log is 11\|22\|33\|44\|55, the separator is a vertical bar (\|), and the log content is parsed into 11, 22, 33, 44, and 55. The keys are set to A, B, C, D, and E respectively. true: When collecting the log 11\|22\|33\|55, 55 is uploaded to Simple Log Service as the value for key D. false: When collecting the log 11\|22\|33\|55, this log entry is discarded because the number of fields does not match the number of keys.

Configuration example

{
    "configName": "logConfigName", 
    "logSample": "testlog", 
    "inputType": "file", 
    "outputType": "LogService", 
    "inputDetail": {
        "logPath": "/logPath", 
        "filePattern": "*", 
        "logType": "delimiter_log", 
        "topicFormat": "default", 
        "discardUnmatch": true, 
        "enableRawLog": true, 
        "fileEncoding": "utf8", 
        "maxDepth": 999, 
        "separator": ",", 
        "quote": "\"", 
        "key": [
            "ip", 
            "time"
        ], 
        "autoExtend": true
    }, 
    "outputDetail": {
        "projectName": "test-project", 
        "logstoreName": "test-logstore"
    }
}

Configurations specific to Logtail plug-ins

Parameters
The following table describes the configurations that are specific to log collection using plug-ins.
Parameter
Type
Required
Example
Description
plugin
JSON object
Yes
None
Configure this parameter when you use a Logtail plug-in to collect logs. For more information, see Use Logtail plug-ins to collect data.

Configuration example

{
    "configName": "logConfigName", 
    "outputType": "LogService", 
    "inputType": "plugin",
    "inputDetail": {
        "plugin": {
            "inputs": [
                {
                    "detail": {
                        "ExcludeEnv": null, 
                        "ExcludeLabel": null, 
                        "IncludeEnv": null, 
                        "IncludeLabel": null, 
                        "Stderr": true, 
                        "Stdout": true
                    }, 
                    "type": "service_docker_stdout"
                }
            ]
        }
    }, 
    "outputDetail": {
        "projectName": "test-project", 
        "logstoreName": "test-logstore"
    }
}

outputDetail parameters

These parameters are used to configure the destination project and Logstore for log output.

Parameter	Type	Required	Example	Description
projectName	string	Yes	my-project	The project name. It must be the same as the project name in the request.
logstoreName	string	Yes	my-logstore	The Logstore name.

Appendix: ExactlyOnce write feature

When you enable the ExactlyOnce write feature, Logtail records fine-grained, file-level checkpoints to the local disk. If an exception such as a process error or server restart occurs, Logtail uses these checkpoints to determine the reprocessing scope for each file when collection resumes. It also uses the incremental sequence numbers from Simple Log Service to prevent duplicate data from being sent. However, this feature consumes some disk write resources. The following limits apply:

Checkpoints are stored on the local disk. If checkpoints cannot be recorded because the disk is full or corrupted, recovery may fail.
Checkpoints record only file metadata, not file data. If the file itself is deleted or modified, recovery may not be possible.
The ExactlyOnce write feature depends on the current write sequence numbers recorded by Simple Log Service. Each shard supports a maximum of 10,000 records. If this limit is exceeded, older records are replaced. To ensure reliability, the result of `Number of active files × Number of Logtail instances` for the same Logstore must not exceed 9,500. We recommend that you leave a buffer.
- Active files: The number of files that are being read and sent. Rotated files with the same logical name are sent serially and count as a single active file.
- Logtail instances: The number of Logtail processes. By default, each server has one instance. Therefore, this number is typically the same as the number of servers.

For performance reasons, Logtail does not call `sync` to write checkpoints to disk by default. If the server restarts before buffered data is written to disk, checkpoints may be lost. To enable synchronous writes, add "enable_checkpoint_sync_write": true, to the Logtail startup configuration file (/usr/local/ilogtail/ilogtail_config.json). For more information, see Set Logtail startup parameters.