Process data during collection (plugins)
Plugins parse raw logs into structured data.
Background
Plugins are classified into native plugins and extended plugins.
Native plugins: Offer superior performance and are recommended for most use cases.
Extended plugins: Offer a broader range of features. Consider using extended plugins when your logs are too complex for native plugins. However, they can reduce performance.
Limitations
Performance limitations
When you use extended plugins for log processing, Logtail consumes more resources, primarily CPU. Adjust the Logtail parameters based on your actual needs. For more information, see Configure the startup parameters of Logtail.
When the raw data rate exceeds 5 MB/s, avoid complex plugin combinations. Perform simple processing with extended plugins and then use data transformation for more advanced tasks.
Log collection limitations
Extended plugins process text logs line by line. This means file-level metadata, such as
__tag__:__path__and__topic__, is added to each log.Adding an extended plugin affects tag-related features:
The contextual query and LiveTail features are disabled. To use these features, you must add an
aggregatorsconfiguration.The
__topic__field is renamed to__log_topic__. If you add anaggregatorsconfiguration, logs contain both the__topic__and__log_topic__fields. If you do not need the__log_topic__field, you can remove it by using the processor_drop plugin.Fields such as
__tag__:__path__no longer have a native field index. You must create field indexes for them.
Plugin combination limitations
For Logtail versions earlier than 2.0:
You cannot add native plugins and extended plugins at the same time.
Native plugins can be used only to collect text logs. When you use native plugins, the following requirements apply:
The first plugin must be a regex parsing, delimiter mode parsing, JSON parsing, Nginx mode parsing, Apache mode parsing, or IIS mode parsing plugin.
After the first processing plugin, you can add only one time parsing plugin, one data filtering plugin, and multiple data masking plugins.
For Logtail 2.0: Extended plugins can be added only after all native plugins.
Parameter combination limitations for native parsing plugins
For native parsing plugins in Logtail versions earlier than 2.0, such as regex parsing, JSON parsing, delimiter mode parsing, Nginx mode parsing, Apache mode parsing, and IIS mode parsing plugins, you must use specific parameter combinations for different scenarios. Other combinations are invalid, and Simple Log Service cannot guarantee the results.
Upload only successfully parsed logs:
Upload parsed logs on success and raw logs on failure:
If parsing is successful, the parsed log is uploaded with the raw log field appended. If it fails, the raw log is uploaded.
For example, if the raw log
"content": "{"request_method":"GET", "request_time":"200"}"is successfully parsed, appending the raw log field adds a new field to the parsed log. You can customize the new field's name. If unspecified, it defaults to the original field name. The field value is the raw log:{"request_method":"GET", "request_time":"200"}.
Plugin list
Native plugins
Plugin | Description |
Regex parsing | Native plugin: regex parsing |
JSON parsing | Native plugin: JSON parsing |
Delimiter mode parsing | Native plugin: delimiter mode parsing |
Nginx mode parsing | Native plugin: Nginx mode parsing |
Apache mode parsing | Native plugin: Apache mode parsing |
IIS mode parsing | Native plugin: IIS mode parsing |
Time parsing | Time parsing (native) |
Data filtering | Native plugin: data filtering |
Data masking | Native plugin: data masking |
Extended plugins
Function | Description |
Extract fields | Regex mode |
Delimiter mode | |
CSV mode | |
Single-character delimiter mode | |
Multi-character delimiter mode | |
Key-value pair mode | |
Grok mode | |
Add fields | Extended plugin: add fields |
Drop fields | Extended plugin: drop fields |
Rename fields | Extended plugin: rename fields |
Pack fields | Packs one or more fields into a single field in JSON object format. For more information, see Extended plugin: pack fields. |
Unpack a JSON field | Extended plugin: unpack a JSON field |
Filter logs | Filters logs by matching field values against a regular expression. For more information, see processor_filter_regex. |
Filters logs by matching field names against a regular expression. For more information, see processor_filter_key_regex. | |
Extract log time | Parses the time field from a raw log and sets the result as the log time. For more information, see Go language time format. |
Convert IP addresses | Converts IP addresses in logs into geographic locations, including country, province, city, and latitude/longitude. For more information, see Extended plugin: convert IP addresses. |
Data masking | Replaces sensitive data in logs with a specified string or an MD5 hash. For more information, see Extended plugin: data masking. |
Map field values | Extended plugin: map field values |
Encrypt fields | Extended plugin: encrypt fields |
Encode and decode data | BASE64 decoding |
BASE64 encoding | |
MD5 encoding | |
Log-to-metric conversion | Converts collected logs into Simple Log Service (SLS) metrics. For more information, see Extended plugin: log-to-metric conversion. |
Log-to-trace conversion | Converts collected logs into SLS traces. For more information, see Extended plugin: log-to-trace conversion. |
Add plugins
Modify an existing configuration
Log in to the Simple Log Service console.
In the Project list, click the target Project.
On the Log Storage > Logstores tab, click the > icon next to the target Logstore and select Data Collection > Logtail Configurations.
In the Logtail Configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.
Click Edit at the top of the page. In the Processor Configurations section, add a plugin and then click Save.
Create a new configuration
Log in to the Simple Log Service console.
On the right side of the console, click the Quick Data Import card.
In the Import Data dialog box, click any card and follow the wizard. In the Logtail Configuration step, add plugins. For more information, see Collect text logs from a host.
NoteConfiguring plugins follows the same process as modifying a Logtail configuration.
