edit-icon download-icon

Use Logstash to collect CSV logs

Last Updated: Mar 06, 2018

Upload the system time as the log time

Log sample

  1. 10.116.14.201,-,2/25/2016,11:53:17,W3SVC7,2132,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv.log

Collection configuration

  1. input {
  2. file {
  3. type => "csv_log_1"
  4. path => ["C:/test/csv/*.log"]
  5. start_position => "beginning"
  6. }
  7. }
  8. filter {
  9. if [type] == "csv_log_1" {
  10. csv {
  11. separator => ","
  12. columns => ["ip", "a", "date", "time", "b", "latency", "status", "size", "method", "url", "file"]
  13. }
  14. }
  15. }
  16. output {
  17. if [type] == "csv_log_1" {
  18. logservice {
  19. codec => "json"
  20. endpoint => "***"
  21. project => "***"
  22. logstore => "***"
  23. topic => ""
  24. source => ""
  25. access_key_id => "***"
  26. access_key_secret => "***"
  27. max_send_retry => 10
  28. }
  29. }
  30. }

Note:

  • The configuration file must be encoded in UTF-8 format without BOM. You can use Notepad++ to modify the file encoding format.
  • path indicates the file path, which must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
  • The type field must be modified unitedly and kept consistent in the file. If a machine has multiple Logstash configuration files, the type field in each configuration file must be unique. Otherwise, data cannot be processed properly.

Related plug-ins: file and csv.

Restart Logstash to apply configurations

Create a configuration file in the conf directory and restart Logstash to apply the file. For more information, see Set Logstash as a Windows service.

Upload the log field content as the log time

Log sample

  1. 10.116.14.201,-,Feb 25 2016 14:03:44,W3SVC7,1332,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv_withtime.log

Collection configuration

  1. input {
  2. file {
  3. type => "csv_log_2"
  4. path => ["C:/test/csv_withtime/*.log"]
  5. start_position => "beginning"
  6. }
  7. }
  8. filter {
  9. if [type] == "csv_log_2" {
  10. csv {
  11. separator => ","
  12. columns => ["ip", "a", "datetime", "b", "latency", "status", "size", "method", "url", "file"]
  13. }
  14. date {
  15. match => [ "datetime" , "MMM dd YYYY HH:mm:ss" ]
  16. }
  17. }
  18. }
  19. output {
  20. if [type] == "csv_log_2" {
  21. logservice {
  22. codec => "json"
  23. endpoint => "***"
  24. project => "***"
  25. logstore => "***"
  26. topic => ""
  27. source => ""
  28. access_key_id => "***"
  29. access_key_secret => "***"
  30. max_send_retry => 10
  31. }
  32. }
  33. }

Note:

  • The configuration file must be encoded in UTF-8 format without BOM. You can use Notepad++ to modify the file encoding format.
  • path indicates the file path, which must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
  • The type field must be modified unitedly and kept consistent in the file. If a machine has multiple Logstash configuration files, the type field in each configuration file must be unique. Otherwise, data cannot be processed properly.

Related plug-ins: file, csv, and date.

Restart Logstash to apply configurations

Create a configuration file in the conf directory and restart Logstash to apply the file. For more information, see Set Logstash as a Windows service.

Thank you! We've received your feedback.