When configuring Logtail to collect text logs, you must specify a regular expression based on your log sample if you parse and collect logs in full regex mode. This topic describes how to modify a regular expression.

To modify a regular expression that you specified in the Log Service console, you can click Validate to check the following items:
  • For the regular expression used to specify the starting header of a cross-line log, check whether the current regular expression can correctly match the expected number of log entries.
  • For the regular expression used to extract fields, check whether the value of each field meets your requirements.

If you need to verify more items and modify a regular expression, you can use online tools such as Regex101 and RegexTester. You can copy and paste the regular expression automatically generated in the console to a tool, and then enter actual logs for further verification and modification.

In full regex mode, Log Service automatically generates regular expressions, which may not be suitable for the message field of cross-line logs. This topic describes how to use Regex101 to verify a regular expression.

Procedure

  1. Copy the regular expression automatically generated by Log Service based on the log sample.
  2. Go to the website of Regex101.
  3. In the REGULAR EXPRESSION field, paste the automatically generated regular expression:
    \[([^]]+)]\s\[(\w+)]\s([^:]+:\s\w+\s\w+\s[^:]+:\S+\s[^:]+:\S+\s\S+). *

    The meaning of the regular expression appears in the right pane of the page.



  4. In the TEST STRING field, paste part of the log sample.

    In the following figure, some content after at is not included in the message field. The log entries included and not included in the message field are highlighted in orange and blue, respectively. Therefore, this regular expression does not fully match the log sample. That is, this regular expression is incorrect for the log sample and cannot be used to collect all required log data.



  5. Verify another error: The entered log sample contains only two colons.

    In the following figure, the regular expression fails to match the log sample.



  6. Replace the last element in the regular expression with [\S\s]+ and check whether the regular expression matches the log sample.

    In the following figure, the regular expression matches the content after at.



    In the following figure, the regular expression matches the log sample that contains only two colons.



You can use the preceding method to specify and modify your regular expression and apply it to a Logtail configuration.