Create Logstash configurations for log collection and processing

Plug-ins

• logstash-input-file plug-in

  The logstash-input-file plug-in collects logs by using the tail command. For more information, visit logstash-input-file.

• logstash-output-logservice plug-in

  The logstash-output-logservice plug-in processes the collected logs and uploads the logs to Log Service.

Procedure

1. Create a configuration file in the C:\logstash-2.2.2-win\conf\ directory.

   Replace the logstash-2.2.2-win version with your actual Logstash version. You can create a configuration file for each type of log. The file name is in the *.conf format.

2. Create configurations for log collection and processing.

   The following script shows sample log collection and processing configurations. The log collection configurations are included in the input parameter. The log processing configurations are included in the output parameter .

   Note

   • The configuration file must be encoded in UTF-8 without BOM. We recommend that you use the Notepad++ editor to modify the file encoding format.
   • The path field indicates the directory of a file. If you specify this field, you must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
   • The values of the type fields in a configuration file must be the same. If multiple Logstash configuration files are created for a server, the values of the type fields in the files must be the same.

   input {
     file {
       type => "iis_log_1"
       path => ["C:/inetpub/logs/LogFiles/W3SVC1/*.log"]
       start_position => "beginning"
     }
   }
   filter {
     if [type] == "iis_log_1" {
     #ignore log comments
     if [message] =~ "^#" {
       drop {}
     }
     grok {
       # check that fields match your IIS log settings
       match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"]
     }
       date {
       match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
         timezone => "Etc/UTC"
     }    
     useragent {
       source=> "useragent"
       prefix=> "browser"
     }
     mutate {
       remove_field => [ "log_timestamp"]
     }
     }
   }
   output {
     if [type] == "iis_log_1" {
     logservice {
           codec => "json"
           endpoint => "***"
           project => "***"
           logstore => "***"
           topic => ""
           source => ""
           access_key_id => "***"
           access_key_secret => "***"
           max_send_retry => 10
       }
       }
   }

   Table 1. Fields in the log processing configurations

   Field	Description
   endpoint	The endpoint of the Log Service project.
   project	The Log Service project.
   logstore	The Logstore.
   topic	The topic of the log entry.
   source	The source of the log entry. If the field is not specified, the IP address of the local server is automatically obtained.
   access_key_id	The AccessKey ID of your Alibaba Cloud account.
   access_key_secret	The AccessKey secret of your Alibaba Cloud account.
   max_send_retry	The maximum number of retries that are performed when a data packet fails to be sent to Log Service. Data packets that fail to be sent during the retry period are dropped. The retry interval is 200 milliseconds.

3. Restart Logstash. For more information, see Start the service.