If you want to collect logs in full regex mode, the logs that you want to collect must be of the same type. If the logs that you want to collect are of different types, you can use the schema-on-read and schema-on-write approaches to process the logs before you collect the logs in full regex mode.

Java logs are program logs that contain normal information and errors such as stack exceptions. Java logs can be one of the following logs:
  • Multi-line WARNING logs
  • Single-line INFO logs
  • Key-value DEBUG logs
[2018-10-01T10:30:31,000] [WARNING] java.lang.Exception: another exception happened
    at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
    at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
    at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
[2018-10-01T10:30:32,000] [INFO] info something
[2018-10-01T10:30:33,000] [DEBUG] key:value key2:value2
You can use the following approaches to process logs before you collect the logs:
  • You can use the schema-on-write approach and specify different regular expressions for multiple Logtail configurations. This way, you can use multiple Logtail configurations to collect logs from a log file and extract the fields that you specify.
    Note By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?.
  • You can use the schema-on-read approach and specify a regular expression that contains the common fields of logs to collect the logs.

    For example, if you want to collect multi-line logs, you can specify a regular expression in which the timestamp and level of the logs are specified to match the beginning of the first line of a log, and the rest of the log is included in the message field. If you want to analyze content in the message field, you can create an index for the message field, specify a regular expression to extract the content that you want to analyze from the message field, and then analyze the content.

    Note We recommend that you use this approach only for scenarios in which you need to analyze tens of millions of logs or less.