Use Logstash to collect csv logs

Last Updated: Jul 05, 2017

Use the system time as the uploaded log time

Sample log

  1. 10.116.14.201,-,2/25/2016,11:53:17,W3SVC7,2132,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv.log

Collection configuration

  1. input {
  2. file {
  3. type => "csv_log_1"
  4. path => ["C:/test/csv/*.log"]
  5. start_position => "beginning"
  6. }
  7. }
  8. filter {
  9. if [type] == "csv_log_1" {
  10. csv {
  11. separator => ","
  12. columns => ["ip", "a", "date", "time", "b", "latency", "status", "size", "method", "url", "file"]
  13. }
  14. }
  15. }
  16. output {
  17. if [type] == "csv_log_1" {
  18. logservice {
  19. codec => "json"
  20. endpoint => "***"
  21. project => "***"
  22. logstore => "***"
  23. topic => ""
  24. source => ""
  25. access_key_id => "***"
  26. access_key_secret => "***"
  27. max_send_retry => 10
  28. }
  29. }
  30. }

Note:

  • The configuration file must be encoded as UTF-8 without BOM. You can download Notepad++ to modify the file encoding format.
  • path indicates the file path, which must use Unix separators, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
  • The type field must be modified in a unified manner and kept consistent across the file. If a machine has multiple Logstash configuration files, the type field in each file must be unique. Otherwise, data cannot be processed properly.

Related plug-ins: file and csv.

Restart Logstash to apply configurations

Create a configuration file in the conf directory and restart Logstash to apply the file. For details, refer to Collect Windows logs through Logstash.

Use the log field content as the uploaded log time

Sample log

  1. 10.116.14.201,-,Feb 25 2016 14:03:44,W3SVC7,1332,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv_withtime.log

Collection configuration

  1. input {
  2. file {
  3. type => "csv_log_2"
  4. path => ["C:/test/csv_withtime/*.log"]
  5. start_position => "beginning"
  6. }
  7. }
  8. filter {
  9. if [type] == "csv_log_2" {
  10. csv {
  11. separator => ","
  12. columns => ["ip", "a", "datetime", "b", "latency", "status", "size", "method", "url", "file"]
  13. }
  14. date {
  15. match => [ "datetime" , "MMM dd YYYY HH:mm:ss" ]
  16. }
  17. }
  18. }
  19. output {
  20. if [type] == "csv_log_2" {
  21. logservice {
  22. codec => "json"
  23. endpoint => "***"
  24. project => "***"
  25. logstore => "***"
  26. topic => ""
  27. source => ""
  28. access_key_id => "***"
  29. access_key_secret => "***"
  30. max_send_retry => 10
  31. }
  32. }
  33. }

Note:

  • The configuration file must be encoded as UTF-8 without BOM. You can download Notepad++ to modify the file encoding format.
  • path indicates the file path, which must use Unix separators, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
  • The type field must be modified in a unified manner and kept consistent across the file. If a machine has multiple Logstash configuration files, the type field in each file must be unique. Otherwise, data cannot be processed properly.

Related plug-ins: file, csv, and date.

Restart Logstash to apply configurations

Create a configuration file in the conf directory and restart Logstash to apply the file. For details, refer to Collect Windows logs through Logstash.

Thank you! We've received your feedback.