Before you use Logstash to collect CSV logs, you must modify the configuration file to parse CSV log fields. Either the time reported by the system or the time recorded in the log content when a CSV log is collected can be used as the log time to upload a log to Log Service. You can configure Logstash to use two methods to collect CSV logs based on the type of the log time.

Use the system time as the log time

  • Sample log
    10.116.14.201,-,2/25/2016,11:53:17,W3SVC7,2132,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv.log
    					
  • Collection configuration
    input {
      file {
        type => "csv_log_1"
        path => ["C:/test/csv/*.log"]
        start_position => "beginning"
      }
    }
    filter {
      if [type] == "csv_log_1" {
      csv {
        separator => ","
        columns => ["ip", "a", "date", "time", "b", "latency", "status", "size", "method", "url", "file"]
      } 
      }
    }
    output {
      if [type] == "csv_log_1" {
      logservice {
            codec => "json"
            endpoint => "***"
            project => "***"
            logstore => "***"
            topic => ""
            source => ""
            access_key_id => "***"
            access_key_secret => "***"
            max_send_retry => 10
        }
        }
    }
    Note
    • The configuration file must be encoded in UTF-8 without BOM. We recommend that you use Notepad++ to change the file encoding format.
    • The path parameter indicates the directory of a log file. If you specify this parameter, you must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
    • The values of the type parameter must be kept consistent in the preceding collection configuration file. If a server has more than one Logstash configuration files, the type parameter in each configuration file must be unique. Otherwise, data cannot be processed correctly.
    Related plug-ins are file and csv.
  • Logstash restart

    Create a configuration file in the conf directory and restart Logstash to make the configuration take effect. For more information, see Configure Logstash as a Windows service.

Use the time in the log content as the log time

  • Sample log
    10.116.14.201,-,Feb 25 2016 14:03:44,W3SVC7,1332,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv_withtime.log
  • Collection configuration
    input {
      file {
        type => "csv_log_2"
        path => ["C:/test/csv_withtime/*.log"]
        start_position => "beginning"
      }
    }
    filter {
      if [type] == "csv_log_2" {
      csv {
        separator => ","
        columns => ["ip", "a", "datetime", "b", "latency", "status", "size", "method", "url", "file"]
      } 
      date {
        match => [ "datetime" , "MMM dd YYYY HH:mm:ss" ]
      }
      }
    }
    output {
      if [type] == "csv_log_2" {
      logservice {
            codec => "json"
            endpoint => "***"
            project => "***"
            logstore => "***"
            topic => ""
            source => ""
            access_key_id => "***"
            access_key_secret => "***"
            max_send_retry => 10
        }
        }
    }
    Note
    • The configuration file must be encoded in UTF-8 without BOM. We recommend that you use Notepad++ to change the file encoding format.
    • The path parameter indicates the path of a log file. If you specify this parameter, you must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
    • The values of the type parameter must be kept consistent in the preceding collection configuration file. If a server has more than one Logstash configuration files, the type parameter in each configuration file must be unique. Otherwise, data cannot be processed correctly.
    Related plug-ins are file and csv.
  • Logstash Restart

    Create a configuration file in the conf directory and restart Logstash to make the configuration take effect. For more information, see Configure Logstash as a Windows service.