Topic generation modes
- Null - Do not generate topic
The default log topic generation mode is Null - Do not generate topic when you configure Logtail to collect text files in the console. The topic is an empty string, and you can query logs without entering a topic.
- Machine Group Topic Attributes
The Machine Group Topic Attributes mode is used to differentiate log data of different servers. If the log data of different servers is stored in the same file or file path, you can divide the servers into different machine groups to differentiate the log data of different servers by topic. To do this, set different topics for different machine groups when creating machine groups, and select Machine Group Topic Attributes as the topic generation mode. Apply the previously created Logtail configuration to those machine groups to complete the configuration.
With the Machine Group Topic Attributes mode selected, Logtail uploads the topic attribute of the machine group where the current server belongs as the topic name to Log Service when reporting data. When querying logs, you need to specify a topic, that is, you need to specify the topic attribute of the target machine group as the query condition.
- File Path RegEx
Note In the regular expression of the file path, you need to escape backslashes (
- The File Path RegEx mode is used to differentiate log data generated by users and instances. If logs are stored under different directories based on different users or instances but their subdirectories and log file names are the same, Log Service cannot differentiate which user or instance generates the logs when collecting log files. To resolve this problem, you can select File Path RegEx as the topic generation mode, enter the regular expression for exact match of the file path, and set the topic name as the instance name.
- With the File Path RegEx mode selected, Logtail uploads the instance name as the topic name to Log Service
when reporting data. The topic generated varies depending on your directory structure
and configuration. You need to specify the topic name as the instance name when querying
logs. For example, the following directory setting stores logs in different directories
according to the users to which the logs belong:
/logs | - /userA/serviceA | - service.log | - /userB/serviceA | - service.log | - /userC/serviceA | - service.log
- If multiple fields in the file path need to be extracted separately, you can use the
multi-layer extraction method, that is,
? P<key>. The value of key can contain only lowercase letters and digits. An example is as follows:
The following custom tags are created for the logs:
/home/admin/serviceA/userB/access.log \/home\/admin\/(? P<service>[^\/]+)/(? P<user>[^/]+)/. *
"__tag__ : service : serviceA" "__tag__ : user : userB"Note Logtail 0.16.19 and later are supported.
- For the service.log files in /logs directory on the server, Log Service cannot differentiate which user or instance
generates the specified logs. In this case, you can select File Path RegEx as the topic generation mode, and enter the regular expression
\/(.*)\/serviceA\/. *to extract the instance names. After configuration, different topics are generated for logs in different directories, including userA, userB, and userC. Then, you can specify the topic to query logs.
- Static topic generation
Select File Path RegEx as the topic generation mode, and enter
customized:// + user-defined topic namein the Custom RegEx field.Note Logtail 0.16.21 and later are supported.
Set a log topic
- Configure Logtail in the Log Service console. For more information, see Collect text logs.
To select Machine Group Topic Attributes as the topic generation mode, set the machine group topic when creating or modifying a machine group.
- On the Logtail configuration page, unfold Advanced Options, and select a topic generation mode from the Topic Generation Mode drop-down list.
Modify a log topic
To change the log topic generation mode, modify the Topic Generation Mode option on the Logtail configuration page.