edit-icon download-icon

Configure and parse text logs

Last Updated: Mar 13, 2018

Process of configuring Logtail to collect text logs

Logtail collects text logs in the following process:

Specify the file path > specify the method to separate log lines > extract log fields > specify the log time.

Specify log line separation method

Generally, a complete access log (for example, Nginx access log) occupies a line. Two logs are separated by a line break. For example, see the following two single-line access logs:

  1. 10.1.1.1 - - [13/Mar/2016:10:00:10 +0800] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"
  2. 10.1.1.1 - - [13/Mar/2016:10:00:11 +0800] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"

For Java applications, a program log usually spans several lines. The characteristic log header is used to separate two logs. For example, see the following Java program log:

  1. [2016-03-18T14:16:16,000] [INFO] [SessionTracker] [SessionTrackerImpl.java:148] Expiring sessions
  2. 0x152436b9a12aecf, 50000
  3. 0x152436b9a12aed2, 50000
  4. 0x152436b9a12aed1, 50000
  5. 0x152436b9a12aed0, 50000

The preceding Java log has a starting field in the time format. The regular expression is \[\d+-\d+-\w+:\d+:\d+,\d+]\s.*. You can complete the configurations in the console as follows.

1

Extract log fields

According to the Log Service data models, a log contains one or more key-value pairs. To extract specified fields for analysis, you must set a regular expression. If log content does not need to be processed, the log can be considered as a key-value pair. For the preceding access log:

  • When fields are extracted

    1. Regular expression: (\S+)\s-\s-\s\[(\S+)\s[^]]+]\s"(\w+).*
    2. Extracted contents: 1) 10.1.1.1; 2) 13/Mar/2016:10:00; 3) GET
  • When fields are not extracted

    1. Regular expression: (.*)
    2. Extracted contents: 1) 10.1.1.1 - - [13/Mar/2016:10:00:10 +0800] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"

Specify log time

According to the Log Service data models, a log must have a time field in UNIX timestamp format. Currently, the log time can be set to the system time when Logtail collects the log or the time field in the log content.

For the preceding access log:

  • Extract the time field in the log content

    1. Time: 13/Mar/2016:10:00:10
    2. Time expression: %d/%b/%Y:%H:%M:%S
  • The system time when the log is collected

    1. Time: Timestamp when the log is collected
Thank you! We've received your feedback.