Log Service:Overview

Limits

• Performance limits

  If a Logtail plug-in is used to process logs, Logtail consumes more resources. Most of these resources are CPU resources. You can modify the Logtail parameter settings based on your business requirements. For more information, see Configure the startup parameters of Logtail. If raw logs are generated at a speed higher than 5 MB/s, we recommend that you do not use complicated combinations of plug-ins to process logs. You can use Logtail plug-ins to preliminarily process logs and then use the data transformation feature to further process the logs.

• Limits on text logs
  Log Service allows you to process text logs in basic modes. Log Service also allows you to use Logtail plug-ins to process text logs. However, Logtail plug-ins have the following limits:

  • If you enable the plug-in processing feature, some advanced features of the text mode become unavailable. For example, you cannot configure filters, upload raw logs, specify the system time zone, drop logs that fail to be parsed, or upload some fields in delimiter mode. However, some of these features can be implemented by using related plug-ins.
  • The processors plug-ins use the line mode to process text logs. In this mode, metadata of files such as __tag__:__path__ and __topic__ is recorded in each log. If you use only processors plug-ins to process logs, the following limits apply to tag-related features:
    • By default, the context query and LiveTail features are unavailable.

      If you want to use these features, you must add the aggregators configuration. For more information, see aggregators configuration.

    • By default, the name of the __topic__ field is renamed __log_topic__.

      After you add the aggregators configuration, both the __topic__ field and the __log_topic__ field exist in logs. If the __log_topic__ field is not required, you can use the processor_drop plug-in to drop the field.

    • Fields such as __tag__:__path__ do not have original field indexes. You must configure indexes for these fields.

Configuration description

{
  ...
  "processors" : [
    {
      "type": "plugin_name",
      "detail": {...}
        },
    ...
  ],
  "aggregators": [
    {
      "type": "plugin_name",
      "detail": {...}
        },
    ...
  ]
}

processors configuration

In the configuration of a Logtail plug-in, the processors field is required. You can use the following plug-ins to process data.

aggregators configuration

If you use a Logtail plug-in to process logs and add only the processors configuration, the __topic__ field in logs is renamed __log_topic__. When you query and analyze logs, the context query and LiveTail feature are unavailable. If you want to use the features, you must add both the processors and aggregators configurations. In this example, you must add the aggregator_context plug-in. Configuration example:

{
    ...
  "processors" : [...],
    "aggregators": [
      {
          "type": "aggregator_context",
            "detail": {}
      }
    ]
}

After you add the aggregators configuration, both the __topic__ and __log_topic__ fields exist in logs, and the values of the fields are the same.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration.