Create a Logtail pipeline configuration using CreateLogtailPipelineConfig - Simple Log Service

Creates a Logtail pipeline configuration.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

log:CreateLogtailPipelineConfig

create

*All Resource

*

log:TLSVersion

None

Request syntax

POST /pipelineconfigs HTTP/1.1

Request parameters

Parameter	Type	Required	Description	Example
project	string	Yes	The name of the project.	test-project
body	object	No	The content of the Logtail pipeline configuration.
configName	string	Yes	The name of the configuration. Note The configuration name must be unique within the project and cannot be modified after the configuration is created. The name must follow these rules: It can contain only lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit. It must be 2 to 128 characters in length.	test-config
logSample	string	No	A sample log. Multiple log entries are supported.	2022-06-14 11:13:29.796 \| DEBUG \| __main__::1 - hello world
global	object	No	The global configuration.
inputs	array<object>	Yes	The list of input plug-ins. Important Currently, you can configure only one input plug-in.
	object	No	The input plug-in. Note For information about the parameters of the file input plug-in, see File plug-in. For information about the parameters of other input plug-ins, see Processing plug-ins.	{ "Type": "input_file", "FilePaths": ["/var/log/*.log"] }
processors	array<object>	No	The list of processing plug-ins. Note Processing plug-ins are classified into native processing plug-ins and extension processing plug-ins. For more information, see Processing plug-ins. Important Note Native plug-ins can be used only to collect text logs. You cannot add native plug-ins and extension plug-ins at the same time. When you use native plug-ins, the following requirements must be met: The first processing plug-in must be a regular expression-based parsing plug-in, a separator-based parsing plug-in, a JSON-based parsing plug-in, an NGINX-based parsing plug-in, an Apache-based parsing plug-in, or an IIS-based parsing plug-in. After the first processing plug-in, you can add only one time parsing plug-in, one filter plug-in, and multiple data masking plug-ins.
	object	No	The processing plug-in. Note For more information about native and extension processing plug-ins, see Processing plug-ins.	{ "Type": "processor_parse_json_native", "SourceKey": "content" }
aggregators	array<object>	No	The list of aggregation plug-ins. Important This parameter is valid only when you use extension processing plug-ins. You can use a maximum of one aggregation plug-in.
	object	No	The aggregation plug-in.
flushers	array<object>	Yes	The list of output plug-ins. Important Currently, you can add only one flusher_sls plug-in.
	object	No	The output plug-in.	{ "Type": "flusher_sls", "Logstore": "test" }
task	object	No	The task configuration.

Global configuration

Parameter	Type	Required	Default value	Example	Description
TopicType	string	No	None	filepath	The topic type. Valid values: filepath: extracts information from the log file path as the topic. This value is valid only when the input plug-in is input_file. machine_group_topic: uses the topic of the machine group to which the configuration is applied. custom: a custom topic. For more information, see Log topic.
TopicFormat	string	No. This parameter is required if you set TopicType to filepath or custom.	/	/var/log/(.*).log	The topic format.
EnableTimestampNanosecond	bool	No	false	false	Specifies whether to enable nanosecond precision for timestamps.
PipelineMetaTagKey	object	No	None	{"HOST_NAME":"__hostname__"}	Important This parameter is supported only by LoongCollector 3.0.10 and later. Controls the tags related to LoongCollector information. The key is the tag parameter name, and the value is the field name of the tag in the log. If you set the value to __default__, the default value is used. If you set the value to an empty string, the tag is deleted. The following tags can be configured: HOST_NAME: the hostname. This tag is added by default. The default value is "__hostname__". AGENT_TAG: the custom identifier. This tag is added by default. The default value is "__user_defined_id__". HOST_ID: the host ID. This tag is not added by default. The default value is "__host_id__". CLOUD_PROVIDER: This tag is not added by default. The default value is "__cloud_provider__".

Input plug-ins

File input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	input_file	The plug-in type. Set the value to input_file.
FilePaths	[string]	Yes	/	["/var/log/*.log"]	The paths of the log files that you want to collect. Currently, only one path is supported. You can use the wildcard characters () and () in the path. The double-asterisk (*) wildcard character can appear only once and can be used only before the file name.
MaxDirSearchDepth	uint	No	0	0	The maximum depth of the directories that are matched by the double-asterisk () wildcard character in a file path. This parameter is valid only when a file path contains the double-asterisk () wildcard character. Valid values: 0 to 1000.
ExcludeFilePaths	[string]	No	None	["/home/admin/*.log"]	The blacklist of file paths. The paths must be absolute paths. The wildcard character (*) is supported.
ExcludeFiles	[string]	No	None	["app*.log", "password"]	The blacklist of file names. The wildcard character (*) is supported.
ExcludeDirs	[string]	No	None	["/home/admin/dir1", "/home/admin/dir2*"]	The blacklist of directories. The paths must be absolute paths. The wildcard character (*) is supported.
FileEncoding	string	No	utf8	utf8	The encoding format of the file. Valid values: utf8 and gbk.
TailSizeKB	uint	No	1024	1024	The size of the data to be collected from the end of a file when the configuration is applied for the first time. If the file size is smaller than the specified value, data is collected from the beginning of the file. Unit: KB. Valid values: 0 to 10485760.
Multiline	object	No	None	/	The multiline log aggregation options.
Multiline.Mode	string	No	custom	custom	The multiline log aggregation mode. Valid values: custom and JSON.
Multiline.StartPattern	string	This parameter is required if you set Multiline.Mode to custom.	None	\d+-\d+-\d+.*	The regular expression to match the first line of a multiline log.
EnableContainerDiscovery	bool	No	false	true	Specifies whether to enable container discovery. This parameter is valid only when Logtail runs in DaemonSet mode and the collection path is a path within a container.
ContainerFilters	object	No	None	/	The container filtering options. Multiple options are combined using the AND operator. This parameter is valid only if you set EnableContainerDiscovery to true.
ContainerFilters.K8sNamespaceRegex	string	No	None	default	For containers deployed in a Kubernetes environment, this parameter specifies the namespace of the pods to which the containers to be collected belong. If you do not specify this parameter, all containers are collected. Regular expressions are supported.
ContainerFilters.K8sPodRegex	string	No	None	test-pod	For containers deployed in a Kubernetes environment, this parameter specifies the names of the pods to which the containers to be collected belong. If you do not specify this parameter, all containers are collected. Regular expressions are supported.
ContainerFilters.IncludeK8sLabel	map	No	None	/	For containers deployed in a Kubernetes environment, this parameter specifies the labels of the pods to which the containers to be collected belong. Multiple conditions are combined using the OR operator. If you do not specify this parameter, all containers are collected. Regular expressions are supported. The key in the map is the pod label name, and the value is the pod label value. The following rules apply: If the value in the map is empty, pods that contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, pods that have a label name matching the key and a label value matching the regular expression are matched. In other cases, pods that have a label name matching the key and a label value matching the value are matched.
ContainerFilters.ExcludeK8sLabel	map	No	None	/	For containers deployed in a Kubernetes environment, this parameter specifies the labels of the pods to which the containers to be excluded belong. Multiple conditions are combined using the OR operator. If you do not specify this parameter, all containers are collected. Regular expressions are supported. The key in the map is the pod label name, and the value is the pod label value. The following rules apply: If the value in the map is empty, pods that contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, pods that have a label name matching the key and a label value matching the regular expression are matched. In other cases, pods that have a label name matching the key and a label value matching the value are matched.
ContainerFilters.K8sContainerRegex	string	No	None	test-container	For containers deployed in a Kubernetes environment, this parameter specifies the names of the containers to be collected. If you do not specify this parameter, all containers are collected. Regular expressions are supported.
ContainerFilters.IncludeEnv	map	No	None	/	The environment variables of the containers to be collected. Multiple conditions are combined using the OR operator. If you do not specify this parameter, all containers are collected. Regular expressions are supported. The key in the map is the environment variable name, and the value is the environment variable value. The following rules apply: If the value in the map is empty, containers whose environment variables contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers that have an environment variable name matching the key and an environment variable value matching the regular expression are matched. In other cases, containers that have an environment variable name matching the key and an environment variable value matching the value are matched.
ContainerFilters.ExcludeEnv	map	No	None	/	The environment variables of the containers to be excluded. Multiple conditions are combined using the OR operator. If you do not specify this parameter, all containers are collected. Regular expressions are supported. The key in the map is the environment variable name, and the value is the environment variable value. The following rules apply: If the value in the map is empty, containers whose environment variables contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers that have an environment variable name matching the key and an environment variable value matching the regular expression are matched. In other cases, containers that have an environment variable name matching the key and an environment variable value matching the value are matched.
ContainerFilters.IncludeContainerLabel	map	No	None	/	The labels of the containers to be collected. Multiple conditions are combined using the OR operator. If you do not specify this parameter, the default value is empty, which indicates that all containers are collected. Regular expressions are supported. The key in the map is the container label name, and the value is the container label value. The following rules apply: If the value in the map is empty, containers that contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers that have a label name matching the key and a label value matching the regular expression are matched. In other cases, containers that have a label name matching the key and a label value matching the value are matched.
ContainerFilters.ExcludeContainerLabel	map	No	None	/	The labels of the containers to be excluded. Multiple conditions are combined using the OR operator. If you do not specify this parameter, the default value is empty, which indicates that all containers are collected. Regular expressions are supported. The key in the map is the container label name, and the value is the container label value. The following rules apply: If the value in the map is empty, containers that contain the key are matched. If the value in the map is not empty: If the value starts with `^` and ends with `$`, containers that have a label name matching the key and a label value matching the regular expression are matched. In other cases, containers that have a label name matching the key and a label value matching the value are matched.
ExternalK8sLabelTag	map	No	None	/	For containers deployed in a Kubernetes environment, this parameter specifies the pod label-related tags that you want to add to logs. The key in the map is the pod label name, and the value is the corresponding tag name. For example, if you add `app: k8s_label_app` to the map and a pod contains the `app=serviceA` label, the information is added to the log as a tag. The added field is `__tag__:k8s_label_app: serviceA`. If the pod does not contain the `app` label, the empty field `__tag__:k8s_label_app:` is added.
ExternalEnvTag	map	No	None	/	For containers deployed in a Kubernetes environment, this parameter specifies the container environment variable-related tags that you want to add to logs. The key in the map is the environment variable name, and the value is the corresponding tag name. For example, if you add `VERSION: env_version` to the map and a container has the `VERSION=v1.0.0` environment variable, the information is added to the log as a tag. The added field is `__tag__:env_version: v1.0.0`. If the container does not have the `VERSION` environment variable, the empty field `__tag__:env_version:` is added.
CollectingContainersMeta	bool	No	false	true	Specifies whether to enable container metadata preview.
AppendingLogPositionMeta	bool	No	false	false	Specifies whether to add the metadata of the file to which the log belongs. The metadata includes the __tag__:__inode__ field and the __file_offset__ field.
AllowingIncludedByMultiConfigs	bool	No	false	false	Specifies whether to allow the current configuration to collect files that are matched by other configurations.
Tags	object	No	None	{"FileInodeTagKey":"__inode__"}	Important This parameter is supported only by LoongCollector 3.0.10 and later. Controls the tags related to file collection. The key is the tag parameter name, and the value is the field name of the tag in the log. If you set the value to __default__, the default value is used. If you set the value to an empty string, the tag is deleted. The following tags can be configured: FileInodeTagKey: the inode of the file. This tag is not added by default. The default value is "__inode__". FilePathTagKey: the file path. This tag is added by default. The default value is "__path__". The following parameters are valid only if you set the EnableContainerDiscovery parameter to true. K8sNamespaceTagKey: the namespace of the container in which the file is located. This tag is added by default. The default value is "_namespace_". K8sPodNameTagKey: the name of the pod in which the file is located. This tag is added by default. The default value is "_pod_name_". K8sPodUidTagKey: the UID of the pod in which the file is located. This tag is added by default. The default value is "_pod_uid_". ContainerNameTagKey: the name of the container in which the file is located. This tag is added by default. The default value is "_container_name_". ContainerIpTagKey: the IP address of the container in which the file is located. This tag is added by default. The default value is "_container_ip_". ContainerImageNameTagKey: the image of the container in which the file is located. This tag is added by default. The default value is "_image_name_".
FileOffsetKey	string	No	None	__file_offset__	Important This parameter is supported only by LoongCollector 3.0.10 and later. The tag for the position of the log in the file. This tag is not added by default. The default value is __file_offset__. If you set the value to __default__, the default value is used. If you set the value to an empty string, the tag is deleted. If the EnableLogPositionMeta parameter and the Tags.FileInodeTagKey or FileOffsetKey parameter are specified at the same time, the EnableLogPositionMeta parameter is ignored.

Container stdout (legacy)

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_docker_stdout	The plug-in type. Set the value to service_docker_stdout.
Stdout	Boolean	No	true	true	Specifies whether to collect stdout.
Stderr	Boolean	No	true	true	Specifies whether to collect stderr.
StartLogMaxOffset	Integer	No	128 × 1024	131072	The length of historical data to be collected for the first time. Unit: bytes. We recommend that you set this parameter to a value between 131072 and 1048576.
IncludeLabel	Map. LabelKey and LabelValue are of the String type.	No	None		The whitelist of container labels. This parameter specifies the containers from which you want to collect logs. By default, this parameter is left empty, which indicates that the standard output of all containers is collected. If you want to configure a whitelist of container labels, you must specify LabelKey. LabelValue is optional. If LabelValue is empty, containers whose labels contain LabelKey are matched. If LabelValue is not empty, only containers whose labels contain LabelKey=LabelValue are matched. By default, LabelValue is used for string matching. A match is found only if LabelValue is identical to the value of a container label. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$`, containers named nginx and cube are matched. Multiple whitelists are combined using the OR operator. A container is matched if its label meets the condition in any of the whitelists.
ExcludeLabel	Map. LabelKey and LabelValue are of the String type.	No	None		The blacklist of container labels. This parameter specifies the containers that you want to exclude. By default, this parameter is left empty, which indicates that no containers are excluded. If you want to configure a blacklist of container labels, you must specify LabelKey. LabelValue is optional. If LabelValue is empty, containers whose labels contain LabelKey are excluded. If LabelValue is not empty, only containers whose labels contain LabelKey=LabelValue are excluded. By default, LabelValue is used for string matching. A match is found only if LabelValue is identical to the value of a container label. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set LabelKey to `io.kubernetes.container.name` and LabelValue to `^(nginx\|cube)$`, containers named nginx and cube are matched. Multiple blacklists are combined using the OR operator. A container is excluded if its label meets the condition in any of the blacklists.
IncludeEnv	Map. EnvKey and EnvValue are of the String type.	No	None		The whitelist of environment variables. This parameter specifies the containers from which you want to collect logs. By default, this parameter is left empty, which indicates that the standard output of all containers is collected. If you want to configure a whitelist of environment variables, you must specify EnvKey. EnvValue is optional. If EnvValue is empty, containers whose environment variables contain EnvKey are matched. If EnvValue is not empty, only containers whose environment variables contain EnvKey=EnvValue are matched. By default, EnvValue is used for string matching. A match is found only if EnvValue is identical to the value of an environment variable. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80
ExcludeEnv	Map. EnvKey and EnvValue are of the String type.	No	None		The blacklist of environment variables. This parameter specifies the containers that you want to exclude. By default, this parameter is left empty, which indicates that no containers are excluded. If you want to configure a blacklist of environment variables, you must specify EnvKey. EnvValue is optional. If EnvValue is empty, logs from containers whose environment variables contain EnvKey are excluded. If EnvValue is not empty, only containers whose environment variables contain EnvKey=EnvValue are excluded. By default, EnvValue is used for string matching. A match is found only if EnvValue is identical to the value of an environment variable. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set EnvKey to `NGINX_SERVICE_PORT` and EnvValue to `^(80\|6379)$`, containers whose service ports are 80 and 6379 are matched. Multiple blacklists are combined using the OR operator. A container is excluded if its environment variable meets the condition in any of the key-value pairs.
IncludeK8sLabel	Map. LabelKey and LabelValue are of the String type.	No	None		The whitelist of Kubernetes labels (defined in template.metadata). This parameter specifies the containers from which you want to collect logs. If you want to configure a whitelist of Kubernetes labels, you must specify LabelKey. LabelValue is optional. If LabelValue is empty, containers whose Kubernetes labels contain LabelKey are matched. If LabelValue is not empty, only containers whose Kubernetes labels contain LabelKey=LabelValue are matched. By default, LabelValue is used for string matching. A match is found only if LabelValue is identical to the value of a Kubernetes label. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set LabelKey to `app` and LabelValue to `^(test1\|test2)$`, containers whose Kubernetes labels contain app:test1 or app:test2 are matched. Multiple whitelists are combined using the OR operator. A container is matched if its Kubernetes label meets the condition in any of the whitelists.
ExcludeK8sLabel	Map. LabelKey and LabelValue are of the String type.	No	None		The blacklist of Kubernetes labels (defined in template.metadata). This parameter specifies the containers that you want to exclude. If you want to configure a blacklist of Kubernetes labels, you must specify LabelKey. LabelValue is optional. If LabelValue is empty, containers whose Kubernetes labels contain LabelKey are excluded. If LabelValue is not empty, only containers whose Kubernetes labels contain LabelKey=LabelValue are excluded. By default, LabelValue is used for string matching. A match is found only if LabelValue is identical to the value of a Kubernetes label. If the value starts with `^` and ends with `$`, it is used for regular expression matching. For example, if you set LabelKey to `app` and LabelValue to `^(test1\|test2)$`, containers whose Kubernetes labels contain app:test1 or app:test2 are matched. Multiple blacklists are combined using the OR operator. A container is excluded if its Kubernetes label meets the condition in any of the blacklists.
K8sNamespaceRegex	String	No	None	^(default\|nginx)$	The namespace based on which you want to collect logs from containers. Regular expressions are supported. For example, if you set this parameter to `^(default\|nginx)$`, all containers in the nginx and default namespaces are matched.
K8sPodRegex	String	No	None	^(nginx-log-demo.*)$	The pod name based on which you want to collect logs from containers. Regular expressions are supported. For example, if you set this parameter to `^(nginx-log-demo.*)$`, all containers in pods whose names start with nginx-log-demo are matched.
K8sContainerRegex	String	No	None	^(container-test)$	The container name based on which you want to collect logs. The Kubernetes container name is defined in spec.containers. Regular expressions are supported. For example, if you set this parameter to `^(container-test)$`, all containers named container-test are matched.

Data processing parameters

Parameter	Type	Required	Default value	Example	Description
BeginLineRegex	String	No	None		The regular expression to match the first line of a log. If you leave this parameter empty, the single-line mode is used. If the beginning of a line matches the regular expression, the line is considered the first line of a new log. Otherwise, the line is appended to the previous log.
BeginLineCheckLength	Integer	No	None		The maximum length for first-line matching. Unit: bytes. The default value is 10 × 1024 bytes. If the regular expression for first-line matching can be matched within the first N bytes, we recommend that you set this parameter to improve matching efficiency.
BeginLineTimeoutMs	Integer	No	None		The timeout period for first-line matching. Unit: milliseconds. The default value is 3,000 milliseconds. If no new log is generated within 3,000 milliseconds, the matching ends, and the last log is uploaded to Simple Log Service.
MaxLogSize	Integer	No	None		The maximum length of a log. The default value is 0. Unit: bytes. The default value is 512 × 1024 bytes. If the length of a log exceeds the specified value, the system stops searching for the first line and directly uploads the log.
ExternalK8sLabelTag	Map. LabelKey and LabelValue are of the String type.	No	None		The Kubernetes label (defined in template.metadata) tags. After you configure this parameter, iLogtail adds Kubernetes label-related fields to logs. For example, if you set LabelKey to app and LabelValue to `k8s_label_app`, and a pod contains the label `app=serviceA`, iLogtail adds this information to the log. The added field is k8s_label_app: serviceA. If the pod does not contain the app label, the empty field k8s_label_app: is added.
ExternalEnvTag	Map. EnvKey and EnvValue are of the String type.	No	None		The container environment variable tags. After you configure this parameter, iLogtail adds container environment variable-related fields to logs. For example, if you set EnvKey to `VERSION` and EnvValue to `env_version`, and a container has the environment variable `VERSION=v1.0.0`, this information is added to the log as a tag. The added field is env_version: v1.0.0. If the container does not have the VERSION environment variable, the empty field env_version: is added.

Data processing environment variables

Environment variable Type Required Default value Example Description

ALIYUN_LOG_ENV_TAGS

String

None

The global environment variable tags. After you configure this parameter, iLogtail adds fields related to the environment variables of the container where iLogtail resides to logs. Separate multiple environment variable names with vertical bars (|).

For example, if you set this parameter to node_name|node_ip and the iLogtail container exposes the related environment variables, this information is added to the log as tags. The added fields are node_ip:172.16.0.1 and node_name:worknode.

MySQL input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_mysql	The plug-in type. Set the value to service_mysql.
Address	string	No	127.0.0.1:3306	rm-*.mysql.rds.aliyuncs.com	The address of the MySQL database.
User	string	No	root	root	The username that is used to log on to the MySQL database.
Password	string	No	None		The password of the user that is used to log on to the MySQL database. For security purposes, you can set the username and password to xxx. After the collection configuration is synchronized to your on-premises machine, find the configuration in the `/usr/local/ilogtail/user_log_config.json` file and modify it. For more information, see Modify local configurations. Important If you modify this parameter in the console, the local configuration is overwritten after synchronization.
DataBase	string	No	/	project_database	The name of the database.
DialTimeOutMs	int	No	5000	5000	The timeout period for connecting to the MySQL database. Unit: ms.
ReadTimeOutMs	int	No	5000	5000	The timeout period for reading the MySQL query results. Unit: ms.
StateMent	string	No	/		The SELECT statement. If you set CheckPoint to true, the WHERE clause in the SELECT statement must contain the checkpoint column (CheckPointColumn). You can use a question mark (?) as a placeholder to work with the checkpoint column. For example, you can set CheckPointColumn to id, CheckPointStart to 0, and StateMent to `SELECT * from ... where id > ?`. After each collection, the system saves the ID of the last data entry as a checkpoint. In the next collection, the question mark (?) in the query statement is replaced with the ID corresponding to the checkpoint.
Limit	bool	No	false	true	Specifies whether to use LIMIT for paging. true: uses LIMIT for paging. false (default): does not use LIMIT for paging. We recommend that you use LIMIT for paging. If you set Limit to true, the system automatically appends a LIMIT clause to the SELECT statement during SQL queries.
PageSize	int	No	/	10	The number of entries to return on each page. This parameter is required if you set Limit to true.
MaxSyncSize	int	No	0	0	The maximum number of records to synchronize at a time. The default value is 0, which indicates that no limit is imposed.
CheckPoint	bool	No	false	true	Specifies whether to use a checkpoint. true: uses a checkpoint. false (default): does not use a checkpoint. A checkpoint can be used as the starting point for the next data collection to implement incremental data collection.
CheckPointColumn	string	No	None	1	The name of the checkpoint column. This parameter is required if you set CheckPoint to true. Warning The values in this column must be incremental. Otherwise, data may fail to be collected. The maximum value in each query result is used as the input for the next query.
CheckPointColumnType	string	No	None	int	The data type of the checkpoint column. Valid values: int and time. The int type is stored as int64. The time type supports the date, datetime, and time types of MySQL. This parameter is required if you set CheckPoint to true.
CheckPointStart	string	No	None		The initial value of the checkpoint column. This parameter is required if you set CheckPoint to true.
CheckPointSavePerPage	bool	No	true	true	Specifies whether to save a checkpoint for each page. true (default): saves a checkpoint for each page. false: saves a checkpoint after each synchronization is complete.
IntervalMs	int	No	60000	60000	The synchronization interval. The default value is 60,000. Unit: ms.

HTTP input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	metric_http	The plug-in type. Set the value to metric_http.
Address	string	Yes	/		The list of URLs. Important The URLs must start with `http` or `https`.
IntervalMs	int	Yes	/	10	The interval between requests. Unit: ms.
Method	string	No	GET	GET	The request method. The value must be in uppercase.
Body	string	No	None		The content of the HTTP body.
Headers	map	No	None	{"key":"value"}	The content of the HTTP header. For example, `{"key":"value"}`. Replace the content with the actual value.
PerAddressSleepMs	int	No	100	100	The interval between requests for each URL in the Addresses list. Unit: ms.
ResponseTimeoutMs	int	No	5000	5000	The request timeout period. Unit: ms.
IncludeBody	bool	No	false	true	Specifies whether to collect the request body. The default value is false. If you set this parameter to true, the request body content is stored in a key named content.
FollowRedirects	bool	No	false	false	Specifies whether to automatically handle redirections.
InsecureSkipVerify	bool	No	false	false	Specifies whether to skip HTTPS security checks.
ResponseStringMatch	string	No	/		Performs a regular expression check on the returned body content. The check result is stored in a key named _response_match_. If a match is found, the value is yes. If no match is found, the value is no.

Syslog input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_syslog	The plug-in type. Set the value to service_syslog.
Address	string	No	tcp://127.0.0.1:9999		The protocol, address, and port that Logtail listens on. Logtail listens on the specified protocol, address, and port and obtains log data. The format is `[tcp/udp]://[` ip `]:[` port `]`. If you do not configure this parameter, the default value `tcp://127.0.0.1:9999` is used. This indicates that only logs forwarded from the local machine can be received. Note The protocol, address, and port number specified in the Logtail configuration must be the same as the forwarding rule specified in the rsyslog configuration file. If the server where Logtail is installed has multiple IP addresses that can receive logs, you can set the address to `0.0.0.0`. This indicates that Logtail listens on all IP addresses of the server.
ParseProtocol	string	No	None	rfc3164	The protocol used to parse logs. The default value is empty, which indicates that logs are not parsed. Valid values: Empty: does not parse logs. rfc3164: uses the RFC3164 protocol to parse logs. rfc5424: uses the RFC5424 protocol to parse logs. auto: automatically selects a suitable parsing protocol based on the log content.
IgnoreParseFailure	bool	No	true	true	The operation to perform after a parsing failure. If you do not configure this parameter, the default value `true` is used, which indicates that parsing is abandoned and the returned content field is directly populated. If you set this parameter to `false`, the log is discarded if parsing fails.

Systemd Journal input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_journal	The plug-in type. Set the value to service_journal.
JournalPaths	[string]	Yes	None	/var/log/journal	The Journal log path. We recommend that you set this parameter to the directory where Journal logs are stored.
SeekPosition	string	No	tail	tail	The initial collection method. Valid values: head and tail. head: collects all data. tail: collects only new data after the Logtail collection configuration is applied.
Kernel	bool	No	true	true	Specifies whether to collect kernel logs.
Units	[string]	No	None	""	The list of units to collect. By default, this parameter is left empty, which indicates that all units are collected.
ParseSyslogFacility	bool	No	false	false	Specifies whether to parse the facility field of syslog logs. If you do not configure this parameter, the field is not parsed.
ParsePriority	bool	No	false	false	Specifies whether to parse the Priority field. If you do not configure this parameter, the field is not parsed. If you set this parameter to true, the Priority field is mapped as follows: `plaintext "0": "emergency" "1": "alert" "2": "critical" "3": "error" "4": "warning" "5": "notice" "6": "informational" "7": "debug"`
UseJournalEventTime	bool	No	false	false	Specifies whether to use the field in the Journal log as the log time. If you do not configure this parameter, the collection time is used as the log time. The time difference for real-time log collection is generally within 3 seconds.

SQL Server input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_mssql	The plug-in type. Set the value to service_mssql.
Address	string	No	127.0.0.1:1433	rm-*.sqlserver.rds.aliyuncs.com	The address of the SQL Server database.
User	string	No	root	root	The username that is used to log on to the SQL Server database.
Password	string	No	None		The password of the user that is used to log on to the SQL Server database. For security purposes, you can set the username and password to xxx. After the collection configuration is synchronized to your on-premises machine, find the configuration in the `/usr/local/ilogtail/user_log_config.json` file and modify it. For more information, see Modify local configurations. Important If you modify this parameter in the console, the local configuration is overwritten after synchronization.
DataBase	string	No	/	project_database	The name of the database.
DialTimeOutMs	int	No	5000	5000	The timeout period for connecting to the SQL Server database. Unit: ms.
ReadTimeOutMs	int	No	5000	5000	The timeout period for reading the SQL Server query results. Unit: ms.
StateMent	string	No	/		The SELECT statement. If you set CheckPoint to true, the WHERE clause in the SELECT statement must contain the checkpoint column (CheckPointColumn). You can use a question mark (?) as a placeholder to work with the checkpoint column. For example, you can set CheckPointColumn to id, CheckPointStart to 0, and StateMent to `SELECT * from ... where id > ?`. After each collection, the system saves the ID of the last data entry as a checkpoint. In the next collection, the question mark (?) in the query statement is replaced with the ID corresponding to the checkpoint.
Limit	bool	No	false	true	Specifies whether to use LIMIT for paging. true: uses LIMIT for paging. false (default): does not use LIMIT for paging. We recommend that you use LIMIT for paging. If you set Limit to true, the system automatically appends a LIMIT clause to the SELECT statement during SQL queries.
PageSize	int	No	/	10	The number of entries to return on each page. This parameter is required if you set Limit to true.
MaxSyncSize	int	No	0	0	The maximum number of records to synchronize at a time. The default value is 0, which indicates that no limit is imposed.
CheckPoint	bool	No	false	true	Specifies whether to use a checkpoint. true: uses a checkpoint. false (default): does not use a checkpoint. A checkpoint can be used as the starting point for the next data collection to implement incremental data collection.
CheckPointColumn	string	No	None	1	The name of the checkpoint column. This parameter is required if you set CheckPoint to true. Warning The values in this column must be incremental. Otherwise, data may fail to be collected. The maximum value in each query result is used as the input for the next query.
CheckPointColumnType	string	No	None	int	The data type of the checkpoint column. Valid values: int and time. The int type is stored as int64. The time type supports the date, datetime, and time types of SQL Server. This parameter is required if you set CheckPoint to true.
CheckPointStart	string	No	None		The initial value of the checkpoint column. This parameter is required if you set CheckPoint to true.
CheckPointSavePerPage	bool	No	true	true	Specifies whether to save a checkpoint for each page. true (default): saves a checkpoint for each page. false: saves a checkpoint after each synchronization is complete.
IntervalMs	int	No	60000	60000	The synchronization interval. The default value is 60,000. Unit: ms.

PostgreSQL input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	service_pgsql	The plug-in type. Set the value to service_pgsql.
Address	string	No	127.0.0.1:5432	rm-*.pg.rds.aliyuncs.com	The address of the PostgreSQL database.
User	string	No	root	root	The username that is used to log on to the PostgreSQL database.
Password	string	No	None		The password of the user that is used to log on to the PostgreSQL database. For security purposes, you can set the username and password to xxx. After the collection configuration is synchronized to your on-premises machine, find the configuration in the `/usr/local/ilogtail/user_log_config.json` file and modify it. For more information, see Modify local configurations. Important If you modify this parameter in the console, the local configuration is overwritten after synchronization.
DataBase	string	No	/	project_database	The name of the PostgreSQL database.
DialTimeOutMs	int	No	5000	5000	The timeout period for connecting to the PostgreSQL database. Unit: ms.
ReadTimeOutMs	int	No	5000	5000	The timeout period for reading the PostgreSQL query results. Unit: ms.
StateMent	string	No	/		The SELECT statement. If you set CheckPoint to true, the WHERE clause in the SELECT statement must contain the checkpoint column (CheckPointColumn) and the value of the column must be set to $1. For example, you can set CheckPointColumn to id and StateMent to `SELECT * from ... where id > $1`
Limit	bool	No	false	true	Specifies whether to use LIMIT for paging. true: uses LIMIT for paging. false (default): does not use LIMIT for paging. We recommend that you use LIMIT for paging. If you set Limit to true, the system automatically appends a LIMIT clause to the SELECT statement during SQL queries.
PageSize	int	No	/	10	The number of entries to return on each page. This parameter is required if you set Limit to true.
MaxSyncSize	int	No	0	0	The maximum number of records to synchronize at a time. The default value is 0, which indicates that no limit is imposed.
CheckPoint	bool	No	false	true	Specifies whether to use a checkpoint. true: uses a checkpoint. false (default): does not use a checkpoint. A checkpoint can be used as the starting point for the next data collection to implement incremental data collection.
CheckPointColumn	string	No	None	1	The name of the checkpoint column. This parameter is required if you set CheckPoint to true. Warning The values in this column must be incremental. Otherwise, data may fail to be collected. The maximum value in each query result is used as the input for the next query.
CheckPointColumnType	string	No	None	int	The data type of the checkpoint column. Valid values: int and time. The int type is stored as int64. The time type supports the time types of PostgreSQL. This parameter is required if you set CheckPoint to true.
CheckPointStart	string	No	None		The initial value of the checkpoint column. This parameter is required if you set CheckPoint to true.
CheckPointSavePerPage	bool	No	true	true	Specifies whether to save a checkpoint for each page. true (default): saves a checkpoint for each page. false: saves a checkpoint after each synchronization is complete.
IntervalMs	int	No	60000	60000	The synchronization interval. The default value is 60,000. Unit: ms.

SNMP input plug-in

Parameter	Type	Required	Default value	Example	Description
Targets	[string]	Yes	/	127.0.0.1	The IP addresses of the target machine group.
Port	string	No	161	161	The port used by the SNMP protocol.
Community	string	No	public	public	The community name. SNMPv1 and SNMPv2 use community names for authentication.
UserName	string	No	None	root	The username. SNMPv3 supports authentication using a username.
AuthenticationProtocol	string	No	NoAuth	NoAuth	The authentication protocol. SNMPv3 supports authentication using an authentication protocol.
AuthenticationPassphrase	string	No	None		The authentication password. The default value is empty. If you set `AuthenticationProtocol` to `MD5` or `SHA`, you must set `AuthenticationPassphrase`.
PrivacyProtocol	string	No	NoPriv	NoPriv	The privacy protocol. SNMPv3 supports authentication using a privacy protocol.
PrivacyPassphrase	string	No	None		The privacy protocol password. By default, it is the same as the authentication password. If you set `PrivacyProtocol` to `DES` or `AES`, you must set `PrivacyPassphrase`.
Timeout	int	No	5	5	The timeout period for a query operation. Unit: seconds.
Version	int	No	2	2	The SNMP protocol version. Valid values: `1`, `2`, and `3`.
Transport	string	No	udp	udp	The SNMP communication method. Valid values: `udp` and `tcp`.
MaxRepetitions	int	No	0	0	The number of retries after a query times out.
Oids	[string]	No	None	1	The object identifiers to query in the target machine.
Fields	[string]	No	None	int	The fields to query in the target machine. This plug-in first translates the fields by looking up the local management information base, translates them into object identifiers, and queries them together.
Tables	[string]	No	None		The tables to query in the target machine. This plug-in first queries all fields in the table, then looks up the local management information base, translates them into object identifiers, and queries them together.

Script input plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	input_command	The plug-in type. Set the value to input_command.
ScriptType	string	Yes	None	shell	The type of the script content. Supported types: bash, shell, python2, and python3.
User	string	Yes	/	public	The username used to run the command. Only non-root users are supported. Note * Make sure the specified username exists on the machine. We recommend that you configure the least privilege by granting rwx permissions only to the directories or files that need to be monitored.
ScriptContent	string	Yes	None		The script content. PlainText and Base64-encrypted content are supported. The length must be within 512 × 1024 bytes.
ContentEncoding	string	No	PlainText	PlainText	The text format of the script content. Valid values: PlainText (default): plain text, not encoded. Base64: Base64 encoded.
LineSplitSep	string	No	None		The separator for the script output content. If left empty, no splitting is performed, and the entire output is returned as a single data entry.
CmdPath	string	No	None	/usr/bin/bash	The path to execute the script command. If left empty, the default path is used. The default paths are as follows: bash: /usr/bin/bash shell: /usr/bin/sh python2: /usr/bin/python2 python3: /usr/bin/python3
TimeoutMilliSeconds	int	No	3000	3000	The timeout period for executing the script. Unit: milliseconds.
IgnoreError	bool	No	false	false	Specifies whether to ignore error logs when the plug-in execution fails. The default value is false, which indicates that error logs are not ignored.
Environments	[string]	No			The environment variables. The default value is the value of os.Environ(). If you set Environments, the specified environment variables are appended to the value of os.Environ().
IntervalMs	int	No	5000	5000	The collection trigger frequency or script execution frequency. Unit: milliseconds.

Native processing plug-ins

Native regular expression parsing plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_parse_regex_native	The plug-in type. Set the value to processor_parse_regex_native.
SourceKey	string	Yes	/	content	The source field.
Regex	string	Yes	/	(\d+-\d+-\d+)\s+(.*)	The regular expression.
Keys	[string]	Yes	/	["time", "msg"]	The list of extracted fields.
KeepingSourceWhenParseFail	bool	No	false	false	Specifies whether to keep the source field if parsing fails.
KeepingSourceWhenParseSucceed	bool	No	false	false	Specifies whether to keep the source field if parsing succeeds.
RenamedSourceKey	string	No	None	key	The name of the field that is used to store the source field when the source field is kept. If you do not specify this parameter, the source field is not renamed.

Native JSON parsing plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_parse_json_native	The plug-in type. Set the value to processor_parse_json_native.
SourceKey	string	Yes	/	content	The source field.
KeepingSourceWhenParseFail	bool	No	false	false	Specifies whether to keep the source field if parsing fails.
KeepingSourceWhenParseSucceed	bool	No	false	false	Specifies whether to keep the source field if parsing succeeds.
RenamedSourceKey	string	No	None	key	The name of the field that is used to store the source field when the source field is kept. If you do not specify this parameter, the source field is not renamed.

Native separator parsing plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_parse_delimiter_native	The plug-in type. Set the value to processor_parse_delimiter_native.
SourceKey	string	Yes	/	content	The source field.
Separator	string	Yes	/	,	The separator.
Quote	string	No	"	"	The quote.
Keys	[string]	Yes	/	["time", "msg"]	The list of extracted fields.
AllowingShortenedFields	bool	No	true	true	Specifies whether to allow the number of extracted fields to be smaller than the number of keys. If not allowed, this scenario is considered a parsing failure.
OverflowedFieldsTreatment	string	No	extend	extend	The behavior when the number of extracted fields is greater than the number of keys. Valid values: extend: keeps the extra fields, and each extra field is added to the log as a separate field. The field names of the extra fields are __column$i__, where $i represents the extra field sequence number, starting from 0. keep: keeps the extra fields, but adds the extra content as a single field to the log. The field name is __column0__. discard: discards the extra fields.
KeepingSourceWhenParseFail	bool	No	false	false	Specifies whether to keep the source field if parsing fails.
KeepingSourceWhenParseSucceed	bool	No	false	false	Specifies whether to keep the source field if parsing succeeds.
RenamedSourceKey	string	No	None	key	The name of the field that is used to store the source field when the source field is kept. If you do not specify this parameter, the source field is not renamed.

Native Apsara parsing plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_parse_apsara_native	The plug-in type. Set the value to processor_parse_apsara_native.
SourceKey	string	Yes	/	content	The source field.
Timezone	string	No	None	GMT+08:00	The time zone of the log time. The format is GMT+HH:MM (east of GMT) or GMT-HH:MM (west of GMT).
KeepingSourceWhenParseFail	bool	No	false	false	Specifies whether to keep the source field if parsing fails.
KeepingSourceWhenParseSucceed	bool	No	false	false	Specifies whether to keep the source field if parsing succeeds.
RenamedSourceKey	string	No	None	key	The name of the field that is used to store the source field when the source field is kept. If you do not specify this parameter, the source field is not renamed.

Native time parsing plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_parse_timestamp_native	The plug-in type. Set the value to processor_parse_timestamp_native.
SourceKey	string	Yes	/	content	The source field.
SourceFormat	string	Yes	/	%Y/%m/%d %H:%M:%S	The log time format. For more information, see Time formats.
SourceTimezone	string	No	None	GMT+08:00	The time zone of the log time. The format is GMT+HH:MM (east of GMT) or GMT-HH:MM (west of GMT).

Native filter plug-in

Parameter	Type	Required	Example	Default value	Description
Type	string	Yes	processor_filter_regex_native	/	The plug-in type. Set the value to processor_filter_regex_native.
Include	map	Yes	/	/	The whitelist of log fields. The key is the field name, and the value is a regular expression. This parameter specifies the conditions that the content of the field specified by the key must meet for the current event to be collected. Multiple conditions are combined using the AND operator. The log is collected only when all conditions are met.

Native data masking plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_desensitize_native	The plug-in type. Set the value to processor_desensitize_native.
SourceKey	string	Yes	/	content	The source field.
Method	string	Yes	/	const	The data masking method. Valid values: const: replaces sensitive content with a constant. md5: replaces sensitive content with its MD5 value.
ReplacingString	string	No. This parameter is required if you set Method to const.	/	******	The constant string used to replace sensitive content.
ContentPatternBeforeReplacedString	string	Yes	/	'password:'	The regular expression for the prefix of the sensitive content.
ReplacedContentPattern	string	Yes	/	[^']*	The regular expression for the sensitive content.
ReplacingAll	bool	No	true	true	Specifies whether to replace all matched sensitive content.

Extension processing plug-ins

Extract fields

Regular expression mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_regex	The plug-in type. Set the value to processor_regex.
SourceKey	string	Yes	/	content	The source field.
Regex	string	Yes	/	(\d+-\d+-\d+)\s+(.*)	The regular expression. You need to use parentheses `()` to specify the fields to be extracted.
Keys	[string]	Yes	/	["ip", "time", "method"]	The names of the extracted fields, such as ["ip", "time", "method"].
NoKeyError	boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
NoMatchError	boolean	No	false	false	Specifies whether to report an error if the specified regular expression does not match the value of the source field. true: reports an error. false (default): does not report an error.
KeepSource	boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: keeps the source field. false (default): does not keep the source field.
FullMatch	boolean	No	true	true	Specifies whether to extract only full matches. true (default): The field values are extracted only if all fields specified in the Keys parameter can be matched with the value of the source field using the regular expression in the Regex parameter. false: Partial matches are also extracted.
KeepSourceIfParseError	boolean	true	true	false	Specifies whether to keep the source field in the parsed log if parsing fails. true: keeps the source field. false (default): does not keep the source field.

Anchor mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_anchor	The plug-in type. Set the value to processor_anchor.
SourceKey	Anchor array	Yes	/	content	The source field.
Anchors	string	Yes	/		The list of anchor items.
Start	String	Yes	None	time	The start keyword. If left empty, it matches the beginning of the string.
Stop	String	Yes	None	\t	The end keyword. If left empty, it matches the end of the string.
FieldName	String	Yes	None	time	The name of the extracted field.
FieldType	String	Yes	None	string	The type of the field. Valid values: string and json.
ExpondJson	boolean	No	false	false	Specifies whether to expand JSON fields. true: expands JSON fields. false (default): does not expand JSON fields.
ExpondConnecter	String	No	_	_	The connector for JSON expansion. The default value is an underscore (_).
MaxExpondDepth	Int	No	0	0	The maximum depth for JSON expansion. The default value is 0, which indicates no limit.
NoAnchorError	Boolean	No	false	false	Specifies whether to report an error if the anchor item is not found. true: reports an error. false (default): does not report an error.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
KeepSource	Boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: reports an error. false (default): does not report an error.

CSV mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_csv	The plug-in type. Set the value to processor_csv.
SourceKey	String	Yes	/	csv	The source field.
SplitKeys	String array	Yes	/	["date", "ip", "content"]	The names of the extracted fields, such as ["date", "ip", "content"]. Important If the number of fields to be split is smaller than the number of fields in the SplitKeys parameter, the extra fields in the SplitKeys parameter are ignored.
PreserveOthers	Boolean	No	false	false	Specifies whether to keep the extra fields if the number of fields to be split is greater than the number of fields in the SplitKeys parameter. true: keeps the extra fields. false (default): does not keep the extra fields.
ExpandOthers	Boolean	No	false	false	Specifies whether to parse the extra fields. true: parses the extra fields. You can parse the extra fields through the ExpandOthers parameter and then specify the prefix for the extra field names through the ExpandKeyPrefix parameter. * false (default): does not parse the extra fields. If you set PreserveOthers to true and ExpandOthers to false, the extra content is stored in the _decode_preserve_ field. Note If the extra field content contains non-standard content, you need to normalize it according to the CSV format before storing it.
ExpandKeyPrefix	String	No			The prefix for the extra field names. For example, if you configure it as `expand_`, the field names are expand_1, expand_2.
TrimLeadingSpace	Boolean	No	false	false	Specifies whether to ignore leading spaces in field values. true: ignores leading spaces. false (default): does not ignore leading spaces.
SplitSep	String	No	,	,	The separator. The default value is a comma (,).
KeepSource	Boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: keeps the source field. false (default): does not keep the source field.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.

Single-character separator mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_split_char	The plug-in type. Set the value to processor_split_char.
SourceKey	String	Yes			The source field.
SplitSep	String	Yes			The separator. It must be a single character and can be an invisible character, such as \u0001.
SplitKeys	String array	Yes		["ip", "time", "method"]	The names of the extracted fields, such as ["ip", "time", "method"].
PreserveOthers	Boolean	No	false	false	Specifies whether to keep the extra fields if the number of fields to be split is greater than the number of fields in the SplitKeys parameter. true: keeps the extra fields. false (default): does not keep the extra fields.
QuoteFlag	Boolean	No	false	false	Specifies whether to use a quote. true: uses a quote. false (default): does not use a quote.
Quote	String	No	/	\u0001	The quote. It must be a single character and can be an invisible character, such as \u0001. This parameter is valid only if you set QuoteFlag to true.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
NoMatchError	Boolean	No	false	false	Specifies whether to report an error if the specified separator does not match the separator in the log. true: reports an error. false (default): does not report an error.
KeepSource	Boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: keeps the source field. false (default): does not keep the source field.

Multi-character separator mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_split_string	The plug-in type. Set the value to processor_split_string.
SourceKey	String	Yes			The source field.
SplitSep	String	Yes			The separator. It must be a single character and can be an invisible character, such as \u0001\u0002.
SplitKeys	String array	Yes		["key1","key2"]	The names of the extracted fields, such as ["key1","key2"].Note If the number of fields to be split is smaller than the number of fields in the SplitKeys parameter, the extra fields in the SplitKeys parameter are ignored.
PreserveOthers	Boolean	No	false	false	Specifies whether to keep the extra fields if the number of fields to be split is greater than the number of fields in the SplitKeys parameter. true: keeps the extra fields. false (default): does not keep the extra fields.
ExpandOthers	Boolean	No	false	false	Specifies whether to use a quote. true: uses a quote. false (default): does not use a quote.
ExpandKeyPrefix	String	No	/	expand_	The prefix for the extra field names. For example, if you configure it as expand_, the field names are expand_1, expand_2.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
NoMatchError	Boolean	No	false	false	Specifies whether to report an error if the specified separator does not match the separator in the log. true: reports an error. false (default): does not report an error.
KeepSource	Boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: keeps the source field. false (default): does not keep the source field.

Key-value pair mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_split_key_value	The plug-in type. Set the value to processor_split_key_value.
SourceKey	string	Yes			The source field.
Delimiter	string	No	\t	\t	The separator between key-value pairs. The default value is the tab character `\t`.
Separator	string	No	:	:	The separator between the key and value in a single key-value pair. The default value is a colon (:).
KeepSource	Boolean	No	false	false	Specifies whether to keep the source field in the parsed log. true: keeps the source field. false (default): does not keep the source field.
ErrIfSourceKeyNotFound	Boolean	No	true	false	Specifies whether to report an error if the source log does not contain the specified source field. true (default): reports an error. false: does not report an error.
DiscardWhenSeparatorNotFound	Boolean	No	false	false	Specifies whether to discard the key-value pair if no matching separator is found. true: discards the key-value pair. false (default): does not discard the key-value pair.
ErrIfSeparatorNotFound	Boolean	No	true	false	Specifies whether to report an error if the specified separator does not exist. true (default): reports an error. false: does not report an error.
ErrIfKeyIsEmpty	Boolean	No	true	false	Specifies whether to report an error if the key is empty after splitting. true (default): reports an error. false: does not report an error.
Quote	String	No			The quote. If a value is enclosed in quotes, the value within the quotes is extracted. Multi-character quotes are supported. By default, the quote feature is disabled. Important * If the quote is a double quotation mark (""), you need to add an escape character, which is a backslash (\). When a backslash (\) is used with a quote inside the quotes, the backslash (\) is output as part of the value.

Grok mode

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_grok	The plug-in type. Set the value to processor_grok.
CustomPatternDir	String array	No			The directory where the custom Grok pattern file is located. The processor_grok plug-in reads all files in the directory. If this parameter is not added, no custom Grok pattern file is imported. Important After updating the custom Grok pattern file, you need to restart Logtail for the changes to take effect.
CustomPatterns	Map	No			The custom Grok pattern. The key is the rule name, and the value is the Grok expression. For information about the default supported expressions, see processor_grok. If the link does not contain the expression you need, enter a custom Grok expression in Match. If this parameter is not added, the custom Grok pattern is not used.
SourceKey	String	No	content	content	The source field. The default value is the content field.
Match	String array	Yes			The array of Grok expressions. The processor_grok plug-in matches the log against the expression list configured here from top to bottom and returns the first successful extraction result. Note Configuring multiple Grok expressions may affect performance. We recommend that you configure no more than 5 expressions.
TimeoutMilliSeconds	Long	No		0	The maximum time to try extracting fields with a Grok expression. Unit: milliseconds. If this parameter is not added or is set to 0, it means no timeout.
IgnoreParseFailure	Boolean	No	true	true	Specifies whether to ignore logs that fail to be parsed. true (default): ignores the logs. false: deletes the logs.
KeepSource	Boolean	No	true	true	Specifies whether to keep the source field after successful parsing. true (default): keeps the source field. false: discards the source field.
NoKeyError	Boolean	No	false	true	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
NoMatchError	Boolean	No	true	true	Specifies whether to report an error if none of the expressions set in the Match parameter match the log. true (default): reports an error. false: does not report an error.
TimeoutError	Boolean	No	true	true	Specifies whether to report an error if the matching times out. true (default): reports an error. false: does not report an error.

Add fields

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_add_fields	The plug-in type. Set the value to processor_add_fields.
Fields	Map	Yes			The field names and values to be added. Key-value pair format. Multiple fields can be added.
IgnoreIfExist	Boolean	No	false	false	Specifies whether to ignore duplicate fields if a field with the same name exists. true: ignores duplicate fields. false (default): does not ignore duplicate fields.

Drop fields

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_drop	The plug-in type. Set the value to processor_drop.
DropKeys	String array	Yes			The fields to be dropped. Multiple fields can be configured.

Rename fields

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_rename	The plug-in type. Set the value to processor_rename.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the log does not contain the specified source field. true: reports an error. false (default): does not report an error.
SourceKeys	String array	Yes			The source fields to be renamed.
DestKeys	String array	Yes			The renamed fields.

Pack fields

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_packjson	The plug-in type. Set the value to processor_packjson.
SourceKeys	String array	Yes			The source fields to be packed.
DestKey	String	No			The packed field.
KeepSource	Boolean	No	true	true	Specifies whether to keep the source field in the parsed log. true (default): keeps the source field. false: discards the source field.
AlarmIfIncomplete	Boolean	No	true	true	Specifies whether to report an error if the source log does not contain the specified source field. true (default): keeps the source field. false: discards the source field.

Expand JSON fields

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_json	The plug-in type. Set the value to processor_json.
SourceKey	String	Yes			The source field to be expanded.
NoKeyError	Boolean	No	true	true	Specifies whether to report an error if the source log does not contain the specified source field. true (default): reports an error. false: does not report an error.
ExpandDepth	Int	No	0	1	The depth of JSON expansion. The default value is 0, which indicates no limit. 1 indicates the current level, and so on.
ExpandConnector	String	No	_	_	The connector for JSON expansion. The default value is an underscore (_).
Prefix	String	No			The prefix to be added to the field names during JSON expansion.
KeepSource	Boolean	No	true	true	Specifies whether to keep the source field in the parsed log. true (default): keeps the source field. false: discards the source field.
UseSourceKeyAsPrefix	Boolean	No			Specifies whether to use the source field name as the prefix for all expanded JSON field names.
KeepSourceIfParseError	Boolean	No	true	true	Specifies whether to keep the source log if parsing fails. true (default): keeps the source log. false: discards the source log.
ExpandArray	Boolean	No	false	false	Specifies whether to expand arrays. This parameter is supported by Logtail 1.8.0 and later. false (default): does not expand arrays. true: expands arrays. For example, `{"k":["1","2"]}` is expanded to `{"k[0]":"1","k[1]":"2"}`.

Filter logs

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_filter_regex	The plug-in type. Set the value to processor_filter_regex.
Include	JSON Object	No			The key is the log field, and the value is the regular expression that the field value must match. The key-value pairs are combined using the AND operator. If the value of a log field matches the corresponding regular expression, the log is collected.
Exclude	JSON Object	No			The key is the log field, and the value is the regular expression that the field value must match. The key-value pairs are combined using the OR operator. If the value of any log field matches the corresponding regular expression, the log is discarded.

Extract log time

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_gotime	The plug-in type. Set the value to processor_gotime.
SourceKey	String	Yes			The source field.
SourceFormat	String	Yes			The format of the source time.
SourceLocation	Int	Yes			The time zone of the source time. If the parameter value is empty, it indicates the time zone of the host or container where Logtail resides.
DestKey	String	Yes			The destination field after parsing.
DestFormat	String	Yes			The time format after parsing.
DestLocation	Int	No			The time zone after parsing. If the parameter value is empty, it indicates the local time zone.
SetTime	Boolean	No	true	true	Specifies whether to set the parsed time as the log time. true (default): yes. false: no.
KeepSource	Boolean	No	true	true	Specifies whether to keep the source field in the parsed log. true (default): keeps the source field. false: does not keep the source field.
NoKeyError	Boolean	No	true	true	Specifies whether to report an error if the source log does not contain the specified source field. true (default): reports an error. false: does not report an error.
AlarmIfFail	Boolean	No	true	true	Specifies whether to report an error if extracting the log time fails. true (default): reports an error. false: does not report an error.

Convert IP addresses

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_geoip	The plug-in type. Set the value to processor_geoip.
SourceKey	String	Yes			The source field for IP address conversion.
DBPath	String	Yes		/user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb	The full path of the GeoIP database. For example, /user/data/GeoLite2-City_20180102/GeoLite2-City.mmdb.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.
NoMatchError	Boolean	No	true	true	Specifies whether to report an error if the IP address is invalid or not found in the database. true (default): reports an error. false: does not report an error.
KeepSource	Boolean	No	true	true	Specifies whether to keep the source field in the parsed log. true (default): keeps the source field. false: does not keep the source field.
Language	String	No	zh-CN	zh-CN	The language attribute. The default value is zh-CN. Important Make sure your GeoIP database contains the corresponding language.

Data masking

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_desensitize	The plug-in type. Set the value to processor_desensitize.
SourceKey	String	Yes			The name of the log field.
Method	String	Yes		const	The data masking method. Valid values: const: replaces sensitive content with a string. You can specify the target string with the ReplaceString parameter. md5: replaces sensitive content with its corresponding MD5 value.
Match	String	No	full	full	The method to extract sensitive content. Valid values: full (default): extracts all content, replacing all content in the target field value. regex: uses a regular expression to extract sensitive content.
ReplaceString	String	No			The string used to replace sensitive content. This parameter is required if you set Method to const.
RegexBegin	String	No			The regular expression to match the prefix of the sensitive content. This parameter is required if you set Match to regex.
RegexContent	String	No			The regular expression to match the sensitive content. This parameter is required if you set Match to regex.

Field value mapping

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_dict_map	The plug-in type. Set the value to processor_dict_map.
SourceKey	String	Yes			The source field.
MapDict	Map	No			The mapping dictionary. If the mapping dictionary is small, you can configure it directly with this parameter. You do not need to provide a local CSV dictionary file. Important If you set the DictFilePath parameter, the configuration in the MapDict parameter does not take effect.
DictFilePath	String	No			The dictionary file in CSV format. The separator for this CSV file is a comma (,), and the field reference is a double quotation mark (").
DestKey	String	No			The name of the mapped field.
HandleMissing	Boolean	No	false	false	Specifies whether to process the target field if it is missing from the source log. true: processes the field. The system fills in the value specified in the Missing parameter. false (default): does not process the field.
Missing	String	No	Unknown	Unknown	The fill value for the target field when it is missing from the source log. The default value is `Unknown`. This parameter takes effect when you set HandleMissing to true.
MaxDictSize	Int	No	1000	1000	The maximum size of the mapping dictionary. The default value is 1000, which means up to 1,000 mapping rules can be stored. To limit the memory usage of the plug-in on the server, you can reduce this value.
Mode	String	No	overwrite	overwrite	The processing method when the mapped field already exists in the source log. overwrite (default): overwrites the original field. fill: does not overwrite the original field.

Field encryption

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_encrypt	The plug-in type. Set the value to processor_encrypt.
SourceKey	String array	Yes			The source field.
EncryptionParameters	Object	Yes			The key-related configuration.
Key	String	Yes			The key. It must be 64 hexadecimal characters.
IV	String	No	00000000000000000000000000000000		The initialization vector for encryption. It must be 32 hexadecimal characters. The default value is `00000000000000000000000000000000`.
KeyFilePath	Boolean	No			The file path from which to read the encryption parameters. If not configured, it is read from Logtail Configuration - Input Configuration - File Path.
KeepSourceValueIfError	String	No	false	false	Specifies whether to keep the original field value if encryption fails. true: keeps the original value. false (default): does not keep the original value. If encryption fails, the field value is replaced with `ENCRYPT_ERROR`.

String replacement

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_string_replace	The plug-in type. Set the value to processor_string_replace.
SourceKey	String	Yes			The source field.
Method	String	Yes			The matching method. Valid values: `const`: replaces with a string. `regex`: replaces with a regular expression. `unquote`: removes escape characters.
Match	String	No			The content to match. If you set `Method` to `const`, enter the string that matches the content to be replaced. If multiple strings match, all are replaced. If you set `Method` to `regex`, enter the regular expression that matches the content to be replaced. If multiple strings match, all are replaced. You can also use regex grouping to match a specific group. If you set `Method` to `unquote`, you do not need to configure this parameter.
ReplaceString	String	No			The replacement string. The default value is "". If you set `Method` to `const`, enter the string to replace the original content. If you set `Method` to `regex`, enter the string to replace the original content. Replacement based on regex groups is supported. If you set `Method` to `unquote`, you do not need to configure this parameter.
DestKey	String	No			Specifies a new field for the replaced content. By default, no new field is added.

Data encoding and decoding

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_base64_encoding	The plug-in type. Set the value to processor_base64_encoding.
SourceKey	String	Yes			The source field.
NewKey	String	Yes			The name of the resulting field after encoding.
NoKeyError	Boolean	No	false	false	Specifies whether to report an error if the source log does not contain the specified source field. true: reports an error. false (default): does not report an error.

Convert logs to metrics

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_log_to_sls_metric	The plug-in type. Set the value to processor_log_to_sls_metric.
MetricTimeKey	String	No			Specifies the time field in the log. This field is mapped to the `__time_nano__` field in the time series data. By default, the value of the `__time__` field in the log is extracted. Make sure the specified field is a valid, formatted timestamp. Currently, Unix timestamps in seconds (10 digits), milliseconds (13 digits), microseconds (16 digits), and nanoseconds (19 digits) are supported.
MetricLabelKeys	[]String	Yes			Specifies the list of keys for the `__labels__` field. The keys must follow the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`. The value cannot contain a vertical bar (\|) or `#$#`. For more information, see Time series data (Metric). You cannot add the `__labels__` field in the MetricLabelKeys parameter. If the source field contains a `__labels__` field, its value is appended to the new `__labels__` field.
MetricValues	Map	Yes			Specifies the metric name and metric value. The metric name corresponds to the `__name__` field and must follow the regular expression `^[a-zA-Z_:][a-zA-Z0-9_:]*$`. The metric value corresponds to the `__value__` field and must be of the Double type. For more information, see Time series data (Metric) .
CustomMetricLabels	Map	No			The custom `__labels__` field. The key must follow the regular expression `^[a-zA-Z_][a-zA-Z0-9_]*$`, and the value cannot contain a vertical bar (\|) or `#$#`. For more information, see Time series data (Metric).
IgnoreError	Boolean	No			Specifies whether to output an error log when no log is matched. The default value is false, which indicates that no error log is output.

Convert logs to traces

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	processor_otel_trace	The plug-in type. Set the value to processor_otel_trace.
SourceKey	String	Yes			The source field.
Format	String	Yes		json	The format after conversion. Valid values: protobuf, json, and protojson.
NoKeyError	Boolean	No	false	true	Specifies whether to report an error if the log does not contain the corresponding source field. The default value is false.
TraceIDNeedDecode	Boolean	No			Specifies whether to Base64-decode the TraceID. The default value is false. If you set Format to protojson and the TraceID has been Base64-encoded, you must set TraceIDNeedDecode to true. Otherwise, the conversion fails.
SpanIDNeedDecode	Boolean	No			Specifies whether to Base64-decode the SpanID. The default value is false. If you set Format to protojson and the SpanID has been Base64-encoded, you must set SpanIDNeedDecode to true. Otherwise, the conversion fails.
ParentSpanIDNeedDecode	Boolean	No			Specifies whether to Base64-decode the ParentSpanID. The default value is false. If you set Format to protojson and the ParentSpanID has been Base64-encoded, you must set ParentSpanIDNeedDecode to true. Otherwise, the conversion fails.

Output plug-ins

SLS output plug-in

Parameter	Type	Required	Default value	Example	Description
Type	string	Yes	/	flusher_sls	The plug-in type. Set the value to flusher_sls.
Logstore	stirng	Yes	/	test-logstore	The name of the Logstore.

Response elements

Element

Type

Description

Example

None defined.

Examples

Success response

JSON format

{}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.