This topic describes how to create Logstash configurations for log collection and processing.
Plug-ins
logstash-input-file plug-in
The logstash-input-file plug-in collects logs by using the tail command. For more information, see logstash-input-file.
logstash-output-logservice plug-in
The logstash-output-logservice plug-in processes the collected logs and uploads the logs to Simple Log Service.
Procedure
Create a configuration file in the C:\logstash-2.2.2-win\conf\ directory.
Replace logstash-2.2.2-win with your actual Logstash version. You can create a configuration file for each type of log. The file name is in the *.conf format.
Create configurations for log collection and processing.
Create the following configurations for log collection and processing based on your business requirements and add the configurations to the configuration file. The configuration for log collection is specified by the input parameter. For more information, see Logstash documentation. The configuration for log processing is specified by the output parameter.
NoteThe configuration file must be encoded in UTF-8 without a byte order mark (BOM). You can use a text editor to modify the file encoding format.
The path parameter specifies the path to a configuration file. If you configure this parameter, you must use delimiters in the UNIX format. Example: C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.
The values of the type parameters in a configuration file must be the same. If multiple Logstash configuration files are created for a server, the values of the type parameters in the files must be the same.
input { file { type => "iis_log_1" path => ["C:/inetpub/logs/LogFiles/W3SVC1/*.log"] start_position => "beginning" } } filter { if [type] == "iis_log_1" { #ignore log comments if [message] =~ "^#" { drop {} } grok { # check that fields match your IIS log settings match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"] } date { match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ] timezone => "Etc/UTC" } useragent { source=> "useragent" prefix=> "browser" } mutate { remove_field => [ "log_timestamp"] } } } output { if [type] == "iis_log_1" { logservice { codec => "json" endpoint => "***" project => "***" logstore => "***" topic => "" source => "" access_key_id => "***" access_key_secret => "***" to_json => true max_send_retry => 10 max_buffer_items => 4000 max_buffer_bytes => 2097152 max_buffer_seconds => 3 } } }
Table 1. Parameters in the log processing configuration Parameter
Required
Description
endpoint
Yes
The Simple Log Service endpoint.
project
Yes
The name of the Simple Log Service project.
logstore
Yes
The name of the Simple Log Service Logstore.
topic
Yes
The topic of logs.
source
Yes
The source of logs. You can specify a custom source.
access_key_id
Yes
The AccessKey ID of your Alibaba Cloud account. For more information, see AccessKey pair.
access_key_secret
Yes
The AccessKey secret of your Alibaba Cloud account. For more information, see AccessKey pair.
to_json
No
Specifies whether to parse logs in the JSON format. Valid values:
true: parses logs in the JSON format. This is the default value.
If a log is of the string type, the log is enclosed in double quotation marks ("").
false: parses logs in the string format.
If a log is in the JSON format, the log is escaped.
max_send_retry
Yes
The maximum number of retries that you can perform when a packet fails to be sent to Simple Log Service. Packets that fail to be sent after the retries are dropped. The retry interval is 200 milliseconds.
max_buffer_items
No
The number of logs that are cached in a packet.
If you do not configure this parameter, 4,000 logs are cached in a packet by default.
max_buffer_bytes
No
The size of logs that are cached in a packet. Maximum value: 10485760. Unit: bytes.
If you do not configure this parameter, 2,097,152 bytes of logs are cached in a packet by default.
max_buffer_seconds
No
The maximum time period for which logs are cached. Unit: seconds.
If you do not configure this parameter, logs are cached for up to 3 seconds by default.
Restart Logstash.
For more information, see Start the service.
What to do next
Use PowerShell to launch the logstash.bat process. The logstash.bat process runs in the frontground. In most cases, the logstash.bat process is performed to test and debug log collection. After debugging, we recommend that you configure Logstash as a Windows service. You can run Logstash in the background and at startup. For more information, see Configure Logstash as a Windows service.