Before you use Logstash to collect logs, you can modify the configuration file to parse log fields.

Use the system time as the log time for log upload

  • Log sample
    2016-02-25 15:37:01 [main] INFO com.aliyun.sls.test_log4j - single line log
    2016-02-25 15:37:11 [main] ERROR com.aliyun.sls.test_log4j - catch exception !
     java.lang.ArithmeticException: / by zero
        at com.aliyun.sls.test_log4j.divide(test_log4j.java:23) ~[bin/:?]
        at com.aliyun.sls.test_log4j.main(test_log4j.java:13) [bin/:?]
    2016-02-25 15:38:02 [main] INFO com.aliyun.sls.test_log4j - normal log
  • Collection configuration
    input {
      file {
        type => "common_log_1"
        path => ["C:/test/multiline/*.log"]
        start_position => "beginning"
        codec => multiline {
          pattern => "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"
          negate => true
          auto_flush_interval => 3
          what => previous
        }
      }
    }
    output {
      if [type] == "common_log_1" {
      logservice {
            codec => "json"
            endpoint => "***"
            project => "***"
            logstore => "***"
            topic => ""
            source => ""
            access_key_id => "***"
            access_key_secret => "***"
            max_send_retry => 10
        }
        }
    }
    Note
    • The configuration file must be encoded in UTF-8 without BOM. We recommend that you use Notepad++ to change the file encoding format.
    • The path parameter indicates the directory of a file. If you specify this parameter, you must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
    • The values of the type parameter must be kept consistent in the preceding configuration file. If a server has more than one Logstash configuration file, the type parameter in each configuration file must be unique. Otherwise, data cannot be processed correctly.
    Related plug-ins are file input plug-in and multiline. For a single-line log file, you can remove the codec => multiline configuration from the multiline plug-in.
  • Restart Logstash to make configurations take effect.

    Create a configuration file in the conf directory. For more information, see Configure Logstash as a Windows service.