This topic describes how to configure a log topic to collect logs.

You can configure a topic when you use Logtail to collect logs or when you use API operations or SDKs to upload log data. The Log Service console supports the following topic generation modes: Null - Do not generate topic, Server Group Topic Attributes, and File Path RegEx.
  • Null - Do not generate topic

    In this mode, the topic is an empty string. You can query logs without configuring a topic.

  • Server Group Topic Attributes

    You can use this mode to identify logs that are generated on different servers. If the logs on the servers are saved in the files with same name or in the same directory, you can configure different topics to identify logs.

    You can add servers to different server groups, and specify different topic attributes for server groups. When you create a Logtail configuration, set Topic Generation Mode to Server Group Topic Attributes. When Logtail sends the logs of a server in a machine group to Log Service, Logtail uploads the topic attribute of the server group as the topic name. You can use the topic name as a filter to query logs.

  • File Path RegEx

    You can use this mode to identify logs that are generated by different users or instances. Log Service stores logs in different directories for different users or instances. However, duplicate sub-directory names or log file names may exist in these directories. Therefore, Log Service cannot identify which user or instance generates these logs.

    When you create a Logtail configuration, you can set Topic generation modes to File Path Regex. Then, enter the regular expression that can match the file directory. When Logtail sends logs to Log Service, Logtail uploads the username or instance name as the topic name. You can use the topic name as a filter to query logs.

    Logs that are generated by different users or instances may be stored in different files with the same name, respectively. Each file is under different directories. For example, three logs files are all named service.log. If you want to collect logs from these files and you only specify the service.log file under the /logs directory as the log destination, Log Service cannot identify users or instances that generate the logs in the destination service.log file. To resolve this issue, you can set Topic Generation Mode to File Path RegEx, and enter the \/(.*)\/serviceA\/.* * regular expression. Then, Log Service generates the following topics for logs in different directories: userA, userB, and userC.
    /logs
      | - /userA/serviceA
        | - service.log
      | - /userB/serviceA
        | - service.log
      | - /userC/serviceA
        | - service.log
    Note You must escape the forward slash (/) in the regular expression that is used to match file paths.
    To extract multiple fields from a file path, you can use the ?P<key> sub-expression to extract fields from the layers of the file path. The value of the key parameter can only contain lowercase letters and digits. The following section provides an example:
    /home/admin/serviceA/userB/access.log
    \/home\/admin\/(? P<service>[^\/]+)/(? P<user>[^/]+)/. *
    The following custom tags are created for logs:
    "__tag__ : service : serviceA"
    "__tag__ : user : userB"
  • Static topic generation

    You can set Topic Generation Mode to File Path RegEx, and enter customized:// + custom topic name in the Custom RegEx field.

    Note Static topic generation is supported by Logtail V0.16.21 (Linux) and later.