Note No topic can be set for a syslog.

Topic generation modes

You can set a topic when using Logtail to collect logs or using APIs or SDKs to upload data. Currently, the following topic generation modes are supported in the console: Null - Do not generate topic, Machine Group Topic Attributes, and File Path RegEx.
  • Null - Do not generate topic

    The default log topic generation mode is Null - Do not generate topic when you configure Logtail to collect text files in the console. The topic is an empty string, and you can query logs without entering a topic.

  • Machine Group Topic Attributes

    The Machine Group Topic Attributes mode is used to differentiate log data of different servers. If the log data of different servers is stored in the same file or file path, you can divide the servers into different machine groups to differentiate the log data of different servers by topic. To do this, set different topics for different machine groups when creating machine groups, and select Machine Group Topic Attributes as the topic generation mode. Apply the previously created Logtail configuration to those machine groups to complete the configuration.

    With the Machine Group Topic Attributes mode selected, Logtail uploads the topic attribute of the machine group where the current server belongs as the topic name to Log Service when reporting data. When querying logs, you need to specify a topic, that is, you need to specify the topic attribute of the target machine group as the query condition.

  • File Path RegEx
    • The File Path RegEx mode is used to differentiate log data generated by users and instances. If logs are stored under different directories based on different users or instances but their subdirectories and log file names are the same, Log Service cannot differentiate which user or instance generates the logs when collecting log files. To resolve this problem, you can select File Path RegEx as the topic generation mode, enter the regular expression for exact match of the file path, and set the topic name as the instance name.
    • With the File Path RegEx mode selected, Logtail uploads the instance name as the topic name to Log Service when reporting data. The topic generated varies depending on your directory structure and configuration. You need to specify the topic name as the instance name when querying logs. For example, the following directory setting stores logs in different directories according to the users to which the logs belong:
      /logs
        | - /userA/serviceA
          | - service.log
        | - /userB/serviceA
          | - service.log
        | - /userC/serviceA
          | - service.log
    • If multiple fields in the file path need to be extracted separately, you can use the multi-layer extraction method, that is, ? P<key> . The value of key can contain only lowercase letters and digits. An example is as follows:
      /home/admin/serviceA/userB/access.log
      \/home\/admin\/(? P<service>[^\/]+)/(? P<user>[^/]+)/. *
      The following custom tags are created for the logs:
      "__tag__ : service : serviceA"
      "__tag__ : user : userB"
      Note Logtail 0.16.19 and later are supported.
    • For the service.log files in /logs directory on the server, Log Service cannot differentiate which user or instance generates the specified logs. In this case, you can select File Path RegEx as the topic generation mode, and enter the regular expression \/(.*)\/serviceA\/. * to extract the instance names. After configuration, different topics are generated for logs in different directories, including userA, userB, and userC. Then, you can specify the topic to query logs.
    Note In the regular expression of the file path, you need to escape backslashes (/).
  • Static topic generation

    Select File Path RegEx as the topic generation mode, and enter customized:// + user-defined topic name in the Custom RegEx field.

    Note Logtail 0.16.21 and later are supported.

Set a log topic

  1. Configure Logtail in the Log Service console. For more information, see Collect text logs.

    To select Machine Group Topic Attributes as the topic generation mode, set the machine group topic when creating or modifying a machine group.

  2. On the Logtail configuration page, unfold Advanced Options, and select a topic generation mode from the Topic Generation Mode drop-down list.
Figure 1. Set the log topic


Modify a log topic

To change the log topic generation mode, modify the Topic Generation Mode option on the Logtail configuration page.

Note The modified configuration only applies to the data collected after the modification takes effect.