Logs can be identified based on log topics. You can specify a topic generation mode when you create a Logtail configuration in the Log Service console. You can specify log topics when you upload logs by calling API operations or by using SDKs. This topic describes the topic generation modes that you can specify in the Log Service console.
Null - Do not generate topic
If you set Topic Generation Mode to Null - Do not generate topic, no log topics are generated.
Machine Group Topic Attributes
This mode is used to distinguish the logs that are generated by different servers. If paths to the logs that are generated by different servers are the same or the files that store the logs are named the same, you can distinguish the logs based on log topics.
You can add servers to different machine groups and configure different topic attributes for the machine groups. Then, you can set Topic Generation Mode to Machine Group Topic Attributes when you create a Logtail configuration. When Logtail uploads the logs of servers in different machine groups to Log Service, Logtail includes the topic attributes of the machine groups as log topics. You can use log topics to query the logs.
File Path RegEx
This mode is used to distinguish the logs that are generated for different users or instances. If logs generated for different users or instances are stored in different directories but duplicate sub-directory names or log file names exist in these directories, Logtail cannot identify the user or instance for which the logs are generated when Logtail collects the logs.
Extract a value from a log file path
/logs
- /userA/serviceA
- service.log
- /userB/serviceA
- service.log
- /userC/serviceA
- service.log
If you specify only /logs for the log file path and specify service.log for the log file name in a Logtail configuration, Logtail collects logs from all files named service.log to a Logstore. The user for which the collected logs are generated cannot be identified. You can specify the following regular expression to extract a value from each log file path. Each value is used as a unique log topic.
- Regular expression
\/(.*)\/serviceA\/.*
- Extraction results
__topic__: userA __topic__: userB __topic__: userC
Extract multiple values from a log file path
For example, the log file path is /logs/userA/serviceA/service.log. You can specify one of the following regular expressions to extract multiple values from the log file path.
- Example 1: Extract multiple values by using unnamed capturing groups in a regular
expression.
- Regular expression
\/logs\/(.*?)\/(.*?)\/service.log
- Extraction results
__tag__:__topic_1__: userA __tag__:__topic_2__: serviceA
- Regular expression
- Example 2: Extract multiple values by using named capturing groups in a regular expression.
- Regular expression
\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/service.log
- Extraction results
__tag__:user: userA __tag__:service: serviceA
- Regular expression
Static topic generation
You can set Topic Generation Mode to File Path RegEx and specify customized:// + Custom topic name
in the Custom RegEx field. This way, custom static topics are used.