Create a Logstash collection configuration

Edit in GitHub

Last Updated: Apr 05, 2020 Edit in GitHub

This topic describes how to create a Logstash collection configuration.

Background information

Related plug-ins

logstash-input-file
This plug-in collects log files in tail mode. For more information, see logstash-input-file.

Note The path parameter indicates a file path. When you specify a file path, you must use delimiters in the UNIX format, for example, C:/test/multiline/*.log. Otherwise, fuzzy match cannot be used.

logstash-output-logservice

This plug-in sends the logs collected by the logstash-input-file plug-in to Log Service.


Parameter	Description
endpoint	The endpoint of Log Service, for example, http://regionid.example.com.
project	The name of a Log Service project.
logstore	The name of a Logstore.
topic	The topic of logs. The default value is NULL.
source	The log source. If the value of this parameter is NULL, the IP address of the current server is used as the log source. Otherwise, the log source is the specified parameter value.
access_key_id	The AccessKey ID of the Alibaba Cloud account.
access_key_secret	The AccessKey secret of the Alibaba Cloud account.
max_send_retry	The maximum number of retries performed when a data packet fails to be sent to Log Service. Data packets that fail to be sent within the retry period are discarded. The retry interval is 200 ms.

Procedure

Create a collection configuration.
Add a configuration file to the C:\logstash-2.2.2-win\conf\ directory. Restart Logstash to make the configuration take effect.
- IIS logs
  For more information, see Use Logstash to collect IIS logs.
- CSV logs
  The system time when logs are collected is used as the log upload time. For more information, see Use Logstash to collect CSV logs.
- Logs with time
  Assume that you collect CSV logs. The time in the log content is used as the log upload time. For more information, see Use Logstash to collect CSV logs.
- General logs
  By default, the system time when the logs are collected is used as the log upload time. Log fields are not parsed. Single-line logs and multi-line logs are supported. For more information, see Use Logstash to collect other logs.
You can create a configuration file for each type of logs. The file name format is *.conf. We recommend that you save all logs in the C:\logstash-2.2.2-win\conf\ directory to facilitate log management.

Note Configuration files must be encoded in UTF-8 without BOM. You can use Notepad++ to change the file encoding format.
Verify the configuration.
1. Run PowerShell or cmd.exe to go to the Logstash installation directory and run the following command to verify the configuration:
```
PS C:\logstash-2.2.2-win\bin> .\logstash.bat agent --configtest --config C:\logstash-2.2.2-win\conf\iis_log.conf
```
2. Modify the collection configuration file by adding the temporary configuration item rubydebug in the output phase to send the collected results to the console. Set the type field in the configuration as needed.
```
output {
if [type] == "***" {
  stdout { codec => rubydebug }
  logservice {
  ...
  }
 }
}
```
3. Run PowerShell or cmd.exe to go to the Logstash installation directory and run the following command:
```
PS C:\logstash-2.2.2-win\bin> .\logstash.bat agent -f C:\logstash-2.2.2-win\conf
```
After the verification, stop the logstash.bat process and remove the temporary configuration item rubydebug.

What to do next

Start logstash.bat in PowerShell. The logstash process works in the frontend. Configuring Logstash as a Windows service is usually required for configuration testing and collection debugging. We recommend that you configure Logstash as a Windows service after debugging so that Logstash can work in the backend and start automatically upon the power-on of the server where Logstash is installed. For more information about how to set Logstash to a Windows service, see Configure Logstash as a Windows service.