Simple Log Service:Overview of Logtail plug-ins for data processing

Background information

Logtail plug-ins for data processing are classified into native plug-ins and extended plug-ins.

• Native plug-ins provide high performance and are suitable for most business scenarios. We recommend that you use native plug-ins.

• Extended plug-ins provide more features. If you cannot process complex business logs by using native plug-ins, you can use extended plug-ins to parse the logs. However, system performance may be compromised in this case.

Limits

• Limits on performance

  • If you use extended plug-ins to process logs, Logtail consumes more resources. Most of the resources are CPU resources. You can modify the Logtail parameter settings based on your business requirements. For more information, see Configure the startup parameters of Logtail.

  • If raw logs are generated at a speed that exceeds 5 MB/s, we recommend that you do not use complicated combinations of plug-ins to process logs. You can use extended plug-ins to preliminarily process logs and then use the data transformation feature to further process the logs.

• Limits on log collection

  • Extended plug-ins use the line mode to process text logs. In this mode, the metadata of files such as __tag__:__path__ and __topic__ is recorded in each log.

  • If you add extended plug-ins to process logs, the following limits apply to tag-related features:

    • The contextual query and LiveTail features are unavailable. If you want to use the features, you must add the aggregators configuration.

    • The __topic__ field is renamed __log_topic__. After you add the aggregators configuration, logs contain the __topic__ and __log_topic__ fields. If you do not require the __log_topic__ field, you can use the processor_drop plug-in to delete the field.

    • For tag fields such as __tag__:__path__, the original field indexes no longer take effect. You must reconfigure indexes for the fields. For more information, see Create indexes.

• Limits on plug-in combinations

  • For plug-ins of Logtail earlier than V2.0:

    • You cannot add native plug-ins and extended plug-ins at the same time.

    • You can use native plug-ins only to collect text logs. When you add native plug-ins, take note of the following items:

      • You must add one of the following Logtail plug-ins for data processing as the first plug-in: Data Parsing (Regex Mode), Data Parsing (Delimiter Mode), Data Parsing (JSON Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode).

      • After you add the first plug-in, you can add one Time Parsing plug-in, one Data Filtering plug-in, and multiple Data Masking plug-ins.

    • You can add extended plug-ins only after you add native plug-ins.

  • For plug-ins of Logtail V2.0: You can add extended plug-ins only after you add native plug-ins.

• Limits on native plug-in-related parameter combinations

  For native plug-ins of Logtail earlier than V2.0, we recommend that you use only the following parameter combinations. The plug-ins refer to Data Parsing (Regex Mode), Data Parsing (JSON Mode), Data Parsing (Delimiter Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode). For other parameter combinations, Simple Log Service does not ensure configuration effects.

  • Upload logs that are parsed.

    

  • Upload logs that are obtained after parsing if the parsing is successful, and upload raw logs if the parsing fails.

    

  • Upload logs that are obtained after parsing and add a raw log field to the logs if the parsing is successful, and upload raw logs if the parsing fails.

    For example, if a raw log is "content": "{"request_method":"GET", "request_time":"200"}" and the raw log is successfully parsed, the system adds a raw log field to the log that is obtained after parsing. The raw log field is specified by the New Name of Original Field parameter. If you do not configure the parameter, the original field name is used. The field value is {"request_method":"GET", "request_time":"200"}.

    

Add plug-ins

Logtail plug-ins for data processing

Native plug-ins

Extended plug-ins