If you want to collect logs in full regex mode, the logs that you want to collect must be of the same type. If the logs that you want to collect are of different types, you can use the schema-on-read and schema-on-write approaches to process the logs before you collect the logs in full regex mode.
- Multi-line WARNING logs
- Single-line INFO logs
- Key-value DEBUG logs
[2018-10-01T10:30:31,000] [WARNING] java.lang.Exception: another exception happened
at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
[2018-10-01T10:30:32,000] [INFO] info something
[2018-10-01T10:30:33,000] [DEBUG] key:value key2:value2
- You can use the schema-on-write approach and specify different regular expressions
for multiple Logtail configurations. This way, you can use multiple Logtail configurations
to collect logs from a log file and extract the fields that you specify.
Note By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?.
- You can use the schema-on-read approach and specify a regular expression that contains
the common fields of logs to collect the logs.
For example, if you want to collect multi-line logs, you can specify a regular expression in which the timestamp and level of the logs are specified to match the beginning of the first line of a log, and the rest of the log is included in the message field. If you want to analyze content in the message field, you can create an index for the message field, specify a regular expression to extract the content that you want to analyze from the message field, and then analyze the content.
Note We recommend that you use this approach only for scenarios in which you need to analyze tens of millions of logs or less.