All Products
Search
Document Center

How do I use the full regex mode to collect log entries in multiple formats?

Last Updated: Sep 27, 2020

Introduction

The full regular mode requires format consistency among all logs. However, some logs may contain content in multiple formats. This topic describes how to collect logs in multiple formats by using full regex mode.

 

Background

For example, a Java log is a program log that contains normal information and error information such as abnormal stacks, as shown below.

  • WARNING: indicates a multi-line log.
  • INFO: indicates a simple text log.
  • DEBUG: key value log.
[2018-10-01T10:30:31,000] [WARNING] java.lang.Exception: another exception happened
    at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
    at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
    at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
[2018-10-01T10:30:32,000] [INFO] info something
[2018-10-01T10:30:33,000] [DEBUG] key:value key2:value2

 

We recommend that you use either the Schema-On-Write or Schema-On-Read.

  • Schema-On-Write: the same log uses multiple collection configurations, with each collection configuration having a different regular expression, so that correct field extraction can be performed.
    Tips: Logtail does not allow you to use multiple Logtail Configs for a file. Therefore, you need to set up multiple symbolic links for the directory where the file is stored. Each Logtail config works on a different symbolic link, thereby indirectly collecting multiple Logtail Configs for the file at the same time.
  • Schema-on-read: you can use a common regular expression to collect log entries in different formats. For example, use a multi-line log collection, set the time and log level to line start regular expression, and the rest are all message fields. If you want to analyze the message field, you can create an index for this field and use the query and analysis features of log service, such as regular expression extraction. Yes. Extract required content from message fields, and perform analysis based on the content.
    Tips: this solution is recommended only when you need to analyze a small number of logs at the same time (for example, to process tens of millions of logs).

 

Application scope

  • Log Service for WAF