Logs can be identified by log topics. You can specify log topics when you collect logs.

You can specify a topic when you use Logtail to collect logs or when you call API operations or use an SDK to upload log data. In the Log Service console, you can set the topic generation mode to Null - Do not generate topic, Machine Group Topic Attributes, or File Path RegEx.
  • Null - Do not generate topic

    In this mode, the topic is an empty string. You can query logs without the need to specify a topic.

  • Machine Group Topic Attributes

    You can use this mode to identify logs that are generated on different servers. If the logs are saved in the files with same name or in the same directory, you can specify different topics to identify logs.

    You can add servers to different machine groups, and configure different topic attributes for machine groups. When you create a Logtail configuration, set Topic Generation Mode to Machine Group Topic Attributes. When Logtail sends logs of a server in a machine group to Log Service, Logtail uploads the topic attributes of the server group as topic names. You can use the log topics (that is, the topic attributes) as filters to query logs.

  • File Path RegEx

    You can use this mode to identify logs that are generated by different users or instances. Log Service stores logs in different directories for different users or instances. However, duplicate sub-directory names or log file names may exist in these directories. Therefore, Log Service cannot identify which user or instance generates these logs.

    To resolve this issue, you can set Topic generation modes to File Path Regex and enter the regular expression of the log file path when you create a Logtail configuration. The regular expression must fully match the log file path. When Logtail sends logs to Log Service, Logtail uploads the username or instance name as the topic name. You can use the topic name as a filter to query logs.

    Logs that are generated by different users or instances may be stored in different files with the same name, respectively. Each file is stored in a different directory. For example, three log files are all named service.log and you only specify the service.log file in the /logs directory as the log source when you collect logs from these files. After the logs are sent to Log Service, Log Service cannot identify which users or instances generate the logs. To resolve this issue, you can set Topic Generation Mode to File Path RegEx, and enter the \/(.*)\/serviceA\/.* regular expression. Then, Log Service generates the following topics for logs in different directories: userA, userB, and userC.
    /logs
      | - /userA/serviceA
        | - service.log
      | - /userB/serviceA
        | - service.log
      | - /userC/serviceA
        | - service.log
    Note You must escape the forward slash (/) in the regular expression that is used to match file paths.
    To extract multiple fields from a file path, you can use the ?P<key> sub-expression to extract fields from the layers of the file path. The value of the key parameter can only contain lowercase letters and digits.
    Note If the alias of the file already exists in the regular expression, the topic field is not generated. For example, in the following example, P<service> and P<user> are aliases. Therefore, the topic field is not generated. The topic field is generated only after one of the aliases is deleted.
    /home/admin/serviceA/userB/access.log
    \/home\/admin\/(?P<service>[^\/]+)/(?P<user>[^/]+)/.*
    The following custom tags are created for logs:
    "__tag__ : service : serviceA"
    "__tag__ : user : userB"
  • Static topic generation

    You can set Topic Generation Mode to File Path RegEx, and enter customized://+ custom topic name in the Custom RegEx field.

    Note Static topic generation is supported by Logtail V0.16.21 (Linux) and later.