Create Logstash collection configurations - User Guide| Alibaba Cloud Documentation Center

Plug-in parameters

logstash-input-file

The plug-in is used to collect log files in tail mode. For details, refer to logstash-input-file.

Note: path indicates the file path, which must use Unix separators, for example, C:/test/multiline/*.log. Otherwise, fuzzy match is not supported.

logstash-output-logservice

You can use the plug-in to collect logs to Log Service.

Parameter	Description
endpoint	Example: `http://cn-shenzhen.log.aliyuncs.com`. For details, refer to Log Service endpoint.
project	Name of a Log Service project
logstore	Logstore name
topic	Log topic name. The default value “null” can be applied.
source	Log source. If this parameter is set to null, the IP address of the local machine is used as the log source.
access_key_id	Access key ID of your Alibaba Cloud account.
access_key_secret	Access key secret of your Alibaba Cloud account.
max_send_retry	The maximum number of retries performed when data packets cannot be sent to Log Service due to an exception. Data packets with retry failures are discarded. The retry interval is 200 ms.

Create collection configuration

You can create a configuration file for each log type. The file name format is xxx.conf. For easier management, it is recommended to create these configuration files in the C:\logstash-2.2.2-win\conf\ directory.

After you create a configuration file in the C:\logstash-2.2.2-win\conf\ directory, restart Logstash to apply the file.

Note: The configuration file must be encoded as UTF-8 without BOM. You can download Notepad++ to modify the file encoding format.

IIS log

For details, refer to IIS log configuration.
CSV log

The system time when logs are collected is used as the uploaded log time. For details, refer to CSV log configuration.
Default log time

For CSV logs, the time in the log content is used as the uploaded log time. For details, refer to CSV log configuration.
General log

By default, the system time when logs are collected is used as the uploaded log time. Log fields are not parsed. Single-line logs and multiline logs are supported. For details, refer to General log configuration.

Verify configuration syntax

Run PowerShell or cmd.exe to go to the Logstash installation directory.

PS C:\logstash-2.2.2-win\bin> .\logstash.bat agent --configtest --config C:\logstash-2.2.2-win\conf\iis_log.conf

Verify data collection

Modify the collection configuration file. Add the temporary configuration item rubydebug in the output phase to output collected results to the console. Set the type field according to your needs.

output {
if [type] == "***" {
  stdout { codec => rubydebug }
  logservice {
    ...
    }
  }
}

Run PowerShell or cmd.exe to go to the Logstash installation directory and launch the process.

PS C:\logstash-2.2.2-win\bin> .\logstash.bat agent -f C:\logstash-2.2.2-win\conf

After verification, end the logstash.bat process and delete the temporary configuration item rubydebug.

Subsequent operations

When logstash.bat is launched in PowerShell, the Logstash process operates in the foreground. It is used for testing configuration and debugging collections. It is recommended that Logstash be configured as a Windows service after debugging to enable Logstash to operate in the background and start automatically upon power-on. For details about how to configure Logstash as a Windows service, refer to Configure Logstash as a Windows service.