If you have complex logs that cannot be parsed in basic modes such as full regex, NGINX, and JSON, you can use Logtail plug-ins to parse the logs. You can configure Logtail plug-ins for one or more processing methods. Then, Logtail executes the processing methods in sequence.
Limits
Performance limits
If a Logtail plug-in is used to process logs, Logtail consumes more resources. Most of these resources are CPU resources. You can modify the Logtail parameter settings based on your business requirements. For more information, see Configure the startup parameters of Logtail. If raw logs are generated at a speed higher than 5 MB/s, we recommend that you do not use complicated combinations of plug-ins to process logs. You can use Logtail plug-ins to preliminarily process logs and then use the data transformation feature to further process the logs.
Limits on text logs
Simple Log Service allows you to process text logs in basic modes. Simple Log Service also allows you to use Logtail plug-ins to process text logs. However, Logtail plug-ins have the following limits:
If you enable the plug-in processing feature, some advanced features of the text mode become unavailable. For example, you cannot configure filters, upload raw logs, specify the system time zone, drop logs that fail to be parsed, or upload some fields in delimiter mode. However, some of these features can be implemented by using related plug-ins.
The processors plug-ins use the line mode to process text logs. In this mode, metadata of files such as __tag__:__path__ and __topic__ is recorded in each log. If you use only processors plug-ins to process logs, the following limits apply to tag-related features:
By default, the context query and LiveTail features are unavailable.
If you want to use these features, you must add the aggregators configuration. For more information, see aggregators configuration.
By default, the name of the __topic__ field is renamed __log_topic__.
After you add the aggregators configuration, both the __topic__ field and the __log_topic__ field exist in logs. If the __log_topic__ field is not required, you can use the processor_drop plug-in to drop the field.
Fields such as __tag__:__path__ do not have original field indexes. You must configure indexes for these fields.
Configuration description
The configuration of a Logtail plug-in consists of a JSON object. The object contains the following keys related to data processing: processors and aggregators. The value of each key is a JSON array. The array includes an element that specifies the name of a Logtail plug-in. Each JSON array includes the type and detail fields. The type field specifies the name of the Logtail plug-in used for data processing, and the detail field specifies detailed configuration. Example:
{
...
"processors" : [
{
"type": "plugin_name",
"detail": {...}
},
...
],
"aggregators": [
{
"type": "plugin_name",
"detail": {...}
},
...
]
}processors configuration
In the configuration of a Logtail plug-in, the processors field is required. You can use the following plug-ins to process data.
Operation | Logtail plug-in | Description |
Extract fields | processor_regex | You can use the processor_regex plug-in to extract the fields that match a specified regular expression. For more information, see Regex mode. |
processor_anchor | You can use the processor_anchor plug-in to anchor strings and extract fields based on the start and stop keywords that you specify. For more information, see Anchor mode. | |
processor_csv | You can use the processor_csv plug-in to extract fields from CSV-formatted data. For more information, see CSV mode. | |
processor_split_char | You can use the processor_split_char plug-in to extract fields based on a specified single-character delimiter. For more information, see Single-character delimiter mode. | |
processor_split_string | You can use the processor_split_string plug-in to extract fields based on a specified multi-character delimiter. For more information, see Multi-character delimiter mode. | |
processor_split_key_value | You can use the processor_split_key_value plug-in to extract fields based on key-value pairs. For more information, see Key-value pair mode. | |
processor_grok | You can use the processor_grok plug-in to extract fields based on Grok patterns. For more information, see Grok mode. | |
Add fields | processor_add_fields | You can use the processor_add_fields plug-in to add fields to a log. For more information, see Add fields. |
Drop fields | processor_drop | You can use the processor_drop plug-in to drop specified fields. For more information, see Drop fields. |
Rename fields | processor_rename | You can use the processor_rename plug-in to rename specified fields. For more information, see Rename fields. |
Encapsulate fields | processor_packjson | You can use the processor_packjson plug-in to encapsulate one or more fields into a field in the JSON object format. For more information, see Encapsulate fields. |
Expand JSON fields | processor_json | You can use the processor_json plug-in to expand JSON fields. For more information, see Expand JSON fields. |
Filter logs | processor_filter_regex | You can use the processor_filter_regex plug-in to filter logs. For more information, see processor_filter_regex. |
processor_filter_key_regex | You can use the processor_filter_key_regex plug-in to filter logs. For more information, see processor_filter_key_regex. | |
Extract log time | processor_gotime | You can use the processor_gotime plug-in to parse the time field of raw logs into a time format that is supported by Go and use the parsed result as the log time. For more information, see Time format supported by Go. |
Convert IP addresses | processor_geoip | You can use the processor_geoip plug-in to convert IP addresses in logs to geographical locations. A geographical location includes the following information: country, province, city, longitude, and latitude. For more information, see Convert IP addresses. |
Mask data | processor_desensitize | You can use the processor_desensitize plug-in to replace sensitive data in logs with specified strings or MD5 hash values. For more information, see Mask sensitive data. |
Map field values | processor_dict_map | You can use the processor_dict_map plug-in to map field values. For more information, see Map field values. |
Encrypt fields | processor_encrypt | You can use the processor_encrypt plug-in to encrypt fields. For more information, see Encrypt fields. |
Encode and decode data | processor_base64_decoding | You can use the processor_base64_decoding plug-in to decode field values. For more information, see Base64 decoding. |
processor_base64_encoding | You can use the processor_base64_encoding plug-in to encode field values. For more information, see Base64 encoding. | |
processor_md5 | You can use the processor_md5 plug-in to encode field values by using the MD5 algorithm. For more information, see MD5 encoding. |
aggregators configuration
Logtail V1.2.1 and later support the aggregator_context plug-in.
If you use a Logtail plug-in to process logs and add only the processors configuration, the __topic__ field in logs is renamed __log_topic__. When you query and analyze logs, the context query and LiveTail feature are unavailable. If you want to use the features, you must add both the processors and aggregators configurations. In this example, you must add the aggregator_context plug-in. Configuration example:
{
...
"processors" : [...],
"aggregators": [
{
"type": "aggregator_context",
"detail": {}
}
]
}After you add the aggregators configuration, both the __topic__ and __log_topic__ fields exist in logs, and the values of the fields are the same.
Entry point
If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration.
Add a Logtail plug-in configuration when you modify a Logtail configuration
Log on to the Log Service console.
In the Projects section, click the required project.
Choose . Click > of the required Logstore. Then, choose .
In the Logtail Configurations list, click the required Logtail configuration.
On the Logtail Config page, click Modify.
Add a Logtail plug-in configuration and click Save.
The configuration items vary based on data sources.
If you use Logtail to collect text logs, turn on Enable Plug-in Processing in the Advanced Options section of your Logtail configuration. Then, add your Logtail plug-in configuration in the Plug-in Config field. For more information, see Advanced settings.
If you use Logtail to collect logs, add your Logtail plug-in configuration in the Plug-in Config field of your Logtail configuration. For more information, see Use Logtail plug-ins to collect data.
If you use Logtail to collect Kubernetes stdout and stderr, perform the following operations in the Simple Log Service console: In the Data Transformation section, click Add Field to add your Logtail plug-in configuration. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.
Add a Logtail plug-in configuration when you create a Logtail configuration
When you create a Logtail configuration, you can perform the following operations to add a Logtail plug-in configuration: In the Import Data section, click the On-premises Open Source/Commercial Software tab, select a data source, and then configure parameter settings. You can specify a project and a Logstore, create a machine group, and then configure the machine group. After you configure the settings, add your Logtail plug-in configuration in the Logtail Config or Specify Data Source step. For more information, see Collect text logs and Use Logtail plug-ins to collect data.
The Logtail plug-in configuration that you add when you create a Logtail configuration works in the same manner as the Logtail plug-in configuration that you add when you modify the Logtail configuration.