Logs can be identified based on log topics. You can specify a topic generation mode when you create a Logtail configuration in the Log Service console. You can specify log topics when you upload logs by calling API operations or by using SDKs. This topic describes the topic generation modes that you can specify in the Log Service console.

Null - Do not generate topic

If you set Topic Generation Mode to Null - Do not generate topic, no log topics are generated.

Machine Group Topic Attributes

This mode is used to distinguish the logs that are generated by different servers. If paths to the logs that are generated by different servers are the same or the files that store the logs are named the same, you can distinguish the logs based on log topics.

You can add servers to different machine groups and configure different topic attributes for the machine groups. Then, you can set Topic Generation Mode to Machine Group Topic Attributes when you create a Logtail configuration. When Logtail uploads the logs of servers in different machine groups to Log Service, Logtail includes the topic attributes of the machine groups as log topics. You can use log topics to query the logs.

File Path RegEx

This mode is used to distinguish the logs that are generated for different users or instances. If logs generated for different users or instances are stored in different directories but duplicate sub-directory names or log file names exist in these directories, Logtail cannot identify the user or instance for which the logs are generated when Logtail collects the logs.

Extract a value from a log file path

When you create a Logtail configuration, you can set Topic Generation Mode to File Path RegEx, specify a regular expression to match a log file path, and then use a capturing group to capture the content that you want to extract. You must specify a regular expression that can match the complete log file path. Only one capturing group is supported. When Logtail uploads the logs of different users or instances to Log Service, Logtail includes the names of the users or instances as log topics. You can use log topics to query the logs.
Note You must escape the forward slash (/) in a regular expression that is used to match a log file path.
For example, the logs that are generated for different users are stored in different directories. However, duplicate log file names exist in the directories. Example:
/logs
   - /userA/serviceA
     - service.log
   - /userB/serviceA
     - service.log
   - /userC/serviceA
     - service.log

If you specify only /logs for the log file path and specify service.log for the log file name in a Logtail configuration, Logtail collects logs from all files named service.log to a Logstore. The user for which the collected logs are generated cannot be identified. You can specify the following regular expression to extract a value from each log file path. Each value is used as a unique log topic.

  • Regular expression
    \/(.*)\/serviceA\/.*
  • Extraction results
    __topic__: userA
    __topic__: userB
    __topic__: userC

Extract multiple values from a log file path

If the source of a log cannot be identified by a single log topic, you can configure multiple capturing groups in the regular expression that you specify. This way, Logtail can extract multiple values from a log file path and use the values as log tags to identify the source of the log. Capturing groups include named capturing groups and unnamed capturing groups. A named capturing group is in the ?P<name> format. If the capturing groups in the specified regular expression are all named capturing groups, tag fields are generated in the __tag__:{name} format. If the capturing groups in the specified regular expression are all unnamed capturing groups, tag fields are generated in the __tag__:__topic_{i}__ format. {i} indicates the ordinal number of an unnamed capturing group.
Note If a specified regular expression includes multiple capturing groups, the __topic__ field is not generated.

For example, the log file path is /logs/userA/serviceA/service.log. You can specify one of the following regular expressions to extract multiple values from the log file path.

  • Example 1: Extract multiple values by using unnamed capturing groups in a regular expression.
    • Regular expression
      \/logs\/(.*?)\/(.*?)\/service.log
    • Extraction results
      __tag__:__topic_1__: userA
      __tag__:__topic_2__: serviceA
  • Example 2: Extract multiple values by using named capturing groups in a regular expression.
    • Regular expression
      \/logs\/(?P<user>.*?)\/(?P<service>.*?)\/service.log
    • Extraction results
      __tag__:user: userA
      __tag__:service: serviceA

Static topic generation

You can set Topic Generation Mode to File Path RegEx and specify customized:// + Custom topic name in the Custom RegEx field. This way, custom static topics are used.

Note This setting is supported on a Linux server on which Logtail V0.16.21 or later is installed.