edit-icon download-icon

Use Logstash to collect other logs

Last Updated: Mar 06, 2018

Log sample

  1. 2016-02-25 15:37:01 [main] INFO com.aliyun.sls.test_log4j - single line log
  2. 2016-02-25 15:37:11 [main] ERROR com.aliyun.sls.test_log4j - catch exception !
  3. java.lang.ArithmeticException: / by zero
  4. at com.aliyun.sls.test_log4j.divide(test_log4j.java:23) ~[bin/:?]
  5. at com.aliyun.sls.test_log4j.main(test_log4j.java:13) [bin/:?]
  6. 2016-02-25 15:38:02 [main] INFO com.aliyun.sls.test_log4j - normal log

Collection configuration

  1. input {
  2. file {
  3. type => "common_log_1"
  4. path => ["C:/test/multiline/*.log"]
  5. start_position => "beginning"
  6. codec => multiline {
  7. pattern => "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"
  8. negate => true
  9. auto_flush_interval => 3
  10. what => previous
  11. }
  12. }
  13. }
  14. output {
  15. if [type] == "common_log_1" {
  16. logservice {
  17. codec => "json"
  18. endpoint => "***"
  19. project => "***"
  20. logstore => "***"
  21. topic => ""
  22. source => ""
  23. access_key_id => "***"
  24. access_key_secret => "***"
  25. max_send_retry => 10
  26. }
  27. }
  28. }

Note:

  • The configuration file must be encoded in UTF-8 format without BOM. You can use Notepad++ to modify the file encoding format.
  • path indicates the file path, which must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
  • The type field must be modified unitedly and kept consistent in the file. If a machine has multiple Logstash configuration files, the type field in each configuration file must be unique. Otherwise, data cannot be processed properly.

Related plug-ins: file and multiline (for a single-line log file, remove the codec => multiline configuration).

Restart Logstash to apply configurations

Create a configuration file in the conf directory and restart Logstash to apply the file. For more information, see Set Logstash as a Windows service.

Thank you! We've received your feedback.