Structure syntax and supported value types - Elasticsearch

A Logstash pipeline configuration file defines how data flows through a pipeline. Each file is divided into three sections — input, filter, and output — corresponding to the three plug-in types that control where data comes from, how it's transformed, and where it goes.

For the open source Logstash configuration format, see Structure of a Config File.

Configuration file structure

Each section holds one or more plug-in definitions:

# Use comments to describe parts of your configuration.
input {
  ...
}

filter {
  ...
}

output {
  ...
}

The input and output sections are required. The filter section is optional — include it only when you need to transform or preprocess data before it reaches the output.

Plug-in execution order: If you define multiple filter plug-ins, Logstash applies them in the order they appear in the configuration file.

Important

If your configuration includes a parameter like last_run_metadata_path, set it to a path in Alibaba Cloud Logstash. The backend provides a path in the format /ssd/1/<Logstash cluster ID>/logstash/data/ that you can use for testing. The system does not delete data in this path — make sure your disk has enough space. After you specify a path, Logstash automatically generates a file there, but you cannot view the file contents.

Important

If you configure a pipeline with a JDBC driver, append allowLoadLocalInfile=false&autoDeserialize=false to the jdbc_connection_string parameter for security: If you don't include these parameters, the system returns a check failure error when you add the configuration file.

jdbc_connection_string => "jdbc:mysql://xxx.drds.aliyuncs.com:3306/<Database name>?allowLoadLocalInfile=false&autoDeserialize=false"

Note

To use this configuration file in a pipeline, reference it in the console's pipeline management page. For more information, see Use configuration files to manage pipelines.

Plug-in configuration

Each plug-in definition consists of the plug-in name followed by its properties. The following example defines two beats plug-ins in the input section, each listening on a different port and host:

input {
  beats {
    port => 8000
    host => "118.11.xx.xx"
  }

  beats {
    port => 8001
    host => "192.168.xx.xx"
  }
}

Available properties vary by plug-in type. See the Elastic documentation for the full list:

Value types

Plug-in properties accept the following value types.

Array

Not recommended. Use the String type instead, or use :list => true to enable type checking on lists. Arrays are only useful when you need to handle hash tables or mixed-type lists that don't require type checks.

users => [ {id => 1, name => bob}, {id => 2, name => jane} ]

List

A List is not a standalone type — it's a property attribute. Declare a parameter with :list => true to allow multiple values with type checking. If any value in the list is invalid (for example, an invalid URL), event processing fails.

path => [ "/var/log/messages", "/var/log/*.log" ]
uris => [ "http://elastic.co", "http://example.net" ]

Boolean

Boolean values must be true or false, without quotation marks.

ssl_enable => true

Byte

A Byte value is a string that represents a number of bytes. Both decimal units (k M G T P E Z Y, base-1000) and binary units (Ki Mi Gi Ti Pi Ei Zi Yi, base-1024) are supported. The value is not case-sensitive and allows a space between the number and the unit. If you omit the unit, the integer is treated as a number of bytes.

my_bytes => "1113"   # 1113 bytes
my_bytes => "10MiB"  # 10485760 bytes
my_bytes => "100kib" # 102400 bytes
my_bytes => "180 mb" # 180000000 bytes

Codec

A Codec is a plug-in that encodes or decodes data. Codecs work in both input and output plug-ins:

An input codec decodes data before it enters the pipeline.
An output codec encodes data before it leaves the pipeline.

Using a codec eliminates the need for a separate filter plug-in. For available codecs, see Codec plug-ins.

codec => "json"

Hash

A Hash value is a collection of key-value pairs in the format "field1" => "value1".

match => {
  "field1" => "value1"
  "field2" => "value2"
  ...
}
# Or as a single line. No commas between entries:
match => { "field1" => "value1" "field2" => "value2" }

Numeric

Numeric values must be floating-point numbers or integers.

port => 33

Password

A Password is a single-value string that is never logged or displayed.

my_password => "password"

URI

A URI can range from a full URL to a simple identifier such as foobar. If the URI contains a password — for example, http://user:paas@example.net — the password is never logged or displayed.

my_uri => "http://foo:bar@example.net"

Path

A Path value is a string representing a valid operating system file path.

my_path => "/tmp/logstash"

String

A String value is a single sequence of characters enclosed in double quotation marks (") or single quotation marks (').

Escape sequences

Escape sequences are disabled by default. To enable them, add the following to logstash.yml:

config.support_escapes: true

With escape sequences enabled, quoted strings (both single- and double-quoted) support the following escape characters:

Escape character	Description	ASCII value (decimal)
`\r`	Carriage return	013
`\n`	Line break	010
`\t`	Tab	009
`\\`	Backslash	092
`\"`	Double quotation mark	034
`\'`	Single quotation mark	039

Example:

name => 'It\'s a beautiful day'

Comments

A comment starts with # and can appear anywhere on a line — not just at the beginning.

# This is a full-line comment.

input { # This is an inline comment.
  # ...
}