Log Service:Create Logstash configurations for log collection and processing

Plug-ins

• logstash-input-file plug-in

  The logstash-input-file plug-in collects logs by using the tail command. For more information, see logstash-input-file.

• logstash-output-logservice plug-in

  The logstash-output-logservice plug-in processes the collected logs and uploads the logs to Log Service.

Procedure

1. Create a configuration file in the C:\logstash-2.2.2-win\conf\ directory.

   Replace logstash-2.2.2-win with your actual Logstash version. You can create a configuration file for each type of log. The file name is in the *.conf format.

2. Create configurations for log collection and processing.

   Create the following configurations for log collection and processing based on your business requirements and add the configurations to the configuration file. The configuration for log collection is specified by the input parameter. For more information, see Logstash documentation. The configuration for log processing is specified by the output parameter.

   Note

   • The configuration file must be encoded in UTF-8 without a byte order mark (BOM). You can use a text editor to modify the file encoding format.
   • The path parameter specifies the path to a configuration file. If you configure this parameter, you must use delimiters in the UNIX format. Example: C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
   • The values of the type parameters in a configuration file must be the same. If multiple Logstash configuration files are created for a server, the values of the type parameters in the files must be the same.

   input {
     file {
       type => "iis_log_1"
       path => ["C:/inetpub/logs/LogFiles/W3SVC1/*.log"]
       start_position => "beginning"
     }
   }
   filter {
     if [type] == "iis_log_1" {
     #ignore log comments
     if [message] =~ "^#" {
       drop {}
     }
     grok {
       # check that fields match your IIS log settings
       match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"]
     }
       date {
       match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
         timezone => "Etc/UTC"
     }    
     useragent {
       source=> "useragent"
       prefix=> "browser"
     }
     mutate {
       remove_field => [ "log_timestamp"]
     }
     }
   }
   output {
     if [type] == "iis_log_1" {
     logservice {
           codec => "json"
           endpoint => "***"
           project => "***"
           logstore => "***"
           topic => ""
           source => ""
           access_key_id => "***"
           access_key_secret => "***"
           to_json => true
           max_send_retry => 10
           max_buffer_items => 4000
           max_buffer_bytes => 2097152
           max_buffer_seconds => 3
       }
       }
   }

   Table 1. Parameters in the log processing configuration

   Parameter	Required	Description
   endpoint	Yes	The Log Service endpoint.
   project	Yes	The name of the Log Service project.
   logstore	Yes	The name of the Log Service Logstore.
   topic	Yes	The topic of logs.
   source	Yes	The source of logs. You can specify a custom source.
   access_key_id	Yes	The AccessKey ID of your Alibaba Cloud account. For more information, see AccessKey pair.
   access_key_secret	Yes	The AccessKey secret of your Alibaba Cloud account. For more information, see AccessKey pair.
   to_json	No	Specifies whether to parse logs in the JSON format. Valid values: true: parses logs in the JSON format. This is the default value. If a log is of the string type, the log is enclosed in double quotation marks (""). false: parses logs in the string format. If a log is in the JSON format, the log is escaped.
   max_send_retry	Yes	The maximum number of retries that you can perform when a packet fails to be sent to Log Service. Packets that fail to be sent after the retries are dropped. The retry interval is 200 milliseconds.
   max_buffer_items	No	The number of logs that are cached in a packet. If you do not configure this parameter, 4,000 logs are cached in a packet by default.
   max_buffer_bytes	No	The size of logs that are cached in a packet. Maximum value: 10485760. Unit: bytes. If you do not configure this parameter, 2,097,152 bytes of logs are cached in a packet by default.
   max_buffer_seconds	No	The maximum time period for which logs are cached. Unit: seconds. If you do not configure this parameter, logs are cached for up to 3 seconds by default.

3. Restart Logstash.
   For more information, see Start the service.