Logstash configuration files - Elasticsearch - Alibaba Cloud Documentation Center

Logstash uses pipelines to collect and process data. You must configure input and output plug-ins for pipelines. You can configure filter plug-ins based on your business requirements. The input and output plug-ins are used to configure input and output data sources, and the filter plug-ins are used to preprocess collected data. This topic describes pipeline configuration files of Alibaba Cloud Logstash.

For more information, see Structure of a Config File in open source Logstash documentation.

You can use a Logstash configuration file to configure a pipeline for data collection and processing. For more information, see Use configuration files to manage pipelines.

Configuration file structure

A Logstash pipeline configuration file allows you to configure each type of plug-in in a separate part.

input {
  ...
}

filter {
  ...
}

output {
  ...
}

Each part can contain one or more plug-ins. If you specify multiple filter plug-ins, Logstash applies the plug-ins in the order that they appear in the configuration file.

Important

If the configuration file contains a parameter similar to last_run_metadata_path, you must set this parameter to the file path of Alibaba Cloud Logstash. A path in the /ssd/1/<Logstash cluster ID>/logstash/data/ format is provided at the backend and is available for tests. The system does not delete the data in this path. Make sure that your disk has sufficient storage space when you use this path. After you specify a path, Logstash automatically generates a file in the path, but you cannot view the data in the file.
For security purposes, if you specify a JDBC driver when you configure a pipeline, you must add allowLoadLocalInfile=false&autoDeserialize=false at the end of the jdbc_connection_string parameter, such as jdbc_connection_string => "jdbc:mysql://xxx.drds.aliyuncs.com:3306/<Database name>?allowLoadLocalInfile=false&autoDeserialize=false". Otherwise, when you add a configuration file for the Logstash pipeline, the system displays an error message that indicates a check failure.

Plug-in configuration

The configuration of a plug-in consists of the plug-in name and a collection of plug-in properties. In the following configuration, the input part defines two beats plug-ins, and each plug-in contains the port and host properties.

input {
  beats {
    port => 8000
    host => "118.11.xx.xx"
  }

  beats {
    port => 8001
    host => "192.168.xx.xx"
  }
}

The properties supported by different plug-ins vary based on the plug-in types. For more information, see Input plugins, Output plugins, Filter plugins, and Codec plugins.

Value types

When you configure a plug-in, you can set the following types of values.

Array

This type is not recommended. We recommend that you use a standard type, such as string. You can define the :list => true property to facilitate type checks. This type is used when you want to process hash tables or lists of mixed types that do not require type checks. Example:

users => [ {id => 1, name => bob}, {id => 2, name => jane} ]

List

The list itself is not a type, but its attributes feature type characteristics. You can enter multiple values for type checks. If you specify :list => true when you declare a parameter, the plug-in enables list checks. Example:

path => [ "/var/log/messages", "/var/log/*.log" ]
uris => [ "http://elastic.co", "http://example.net" ]

In this example, path is configured as a list that contains two strings. uris is configured as a list of URLs. If one of the URLs is invalid, event processing fails.

Boolean

A value of the Boolean type must be true or false. The value is not enclosed in quotation marks. Example:

ssl_enable => true

Byte

A field of the byte type is a string field that represents a number of valid bytes. This is a convenient way to declare specific sizes in plug-in options. The byte type supports decimal (k M G T P E Z Y) and binary (Ki Mi Gi Ti Pi Ei Zi Yi). Decimal units are in base-1000 and binary units are in base-1024. The field is not case-sensitive and allows a space between the value and the unit. If no units are specified, an integer string indicates the number of bytes. Example:

 my_bytes => "1113"   # 1113 bytes
 my_bytes => "10MiB"  # 10485760 bytes
 my_bytes => "100kib" # 102400 bytes
 my_bytes => "180 mb" # 180000000 bytes

Codec

A codec is a type of data that is encoded or decoded. It can be used in both the input and output plug-ins. An input codec decodes data before the data enters the input plug-in. An output codec encodes data before the data leaves the output plug-in. If you use an input or output codec, you do not need to specify filter plug-ins for your Logstash pipeline.

For more information about available codecs, see Codec plugins. Example:

codec => "json"

Hash

A value of the hash type is a collection of key-value pairs, such as "field1" => "value1".

Important Multiple key-value pairs are separated by spaces instead of commas (,).

Example:

match => {
  "field1" => "value1"
  "field2" => "value2"
  ...
}
# Specify key-value pairs in a single line. Separate the key-value pairs with spaces instead of commas (,). 
match => { "field1" => "value1" "field2" => "value2" }

Numeric

A value of the numeric type must be a floating point or an integer. Example:

port => 33

Password

A password is a string with a single value that is not recorded or displayed. Example:

my_password => "password"

URI

A URI ranges from a full URL to a simple identifier, such as foobar. If a URI contains a password, such as http://user:paas@example.net, the password is not recorded or displayed. Example:

 my_uri => "http://foo:bar@example.net"

Path

A path is a string that represents a valid path in an operating system. Example:

my_path => "/tmp/logstash"

String

A string must be a single sequence of characters enclosed in double quotation marks (") or single quotation marks (').

Escape sequence

By default, Logstash does not enable escape sequences. If you want to use escape sequences in quoted strings, add config.support_escapes: true in logstash.yml. Then, the quoted strings (double- and single-precision) are converted based on the following table.


Escape character	Description	ASCII value (decimal)
\r	Carriage return	013
\n	Line break	010
\t	Tab	009
\\	Backslash	092
\"	Double quotation mark	034
\'	Single quotation mark	039

Example:

name => 'It\'s a beautiful day'

Comment

A comment starts with a number sign (#) and do not need to be at the beginning of a line. Example:

# A comment. 

input { # A comment. 
  # ...
}