Elasticsearch:Use the logstash-output-oss plugin

Prerequisites

Before you begin, ensure that you have:

• Installed the logstash-output-oss plug-in. For more information, see Install and remove a plug-in.

• Activated OSS. For more information, see Activate OSS.

• Created a read/write OSS bucket and obtained the AccessKey ID and AccessKey secret of an account with write permissions on the bucket. For more information, see Create buckets.

• Prepared your data sources. The plug-in supports all Logstash input plug-ins. For a full list, see Input plugins.

Configure the plug-in

Create a pipeline using a configuration file, then configure the output section with the oss plug-in parameters. After you save and deploy the pipeline, Alibaba Cloud Logstash transfers data from your data sources to OSS.

For instructions on creating a pipeline, see Use configuration files to manage pipelines.

The following example configures a pipeline that reads from Beats and writes to OSS:

input {
  beats {
    port => "8044"                    # required
    codec => json {
      charset => "UTF-8"
    }
  }
}
output {
  oss {
    endpoint => "http://oss-cn-hangzhou-internal.aliyuncs.com"  # required
    bucket => "zl-log-output-test"                              # required
    access_key_id => "LTAI****************"                     # required
    access_key_secret => "yourAccessKeySecret"                  # required
    temporary_directory => "/ssd/1/<Logstash cluster ID>/logstash/data/22"  # required
    prefix => "oss/database"                                    # optional, default: empty
    recover => true                                             # optional, default: true
    rotation_strategy => "size_and_time"                        # optional, default: size_and_time
    time_rotate => 1                                            # optional, default: 15 (minutes)
    size_rotate => 1000                                         # optional, default: 31457280 (bytes)
    encoding => "gzip"                                          # optional, default: none
    additional_oss_settings => {
      max_connections_to_oss => 1024                            # optional, default: 1024
      secure_connection_enabled => false                        # optional, default: false
    }
    codec => json {
      charset => "UTF-8"
    }
  }
}

Alibaba Cloud Logstash currently supports data transfer only within the same virtual private cloud (VPC). To transfer data over the public network, you must configure NAT. For more information, see Configure NAT for public network data transfer.

Parameters

The following table describes all parameters supported by logstash-output-oss.

Temporary files

While transferring data, the plug-in caches data in a temporary file on your local machine and uploads it to OSS periodically. Use the temporary_directory parameter to specify where this file is stored.

A temporary file path follows this format:

/ssd/1/<Logstash cluster ID>/logstash/data/eaced620-e972-0136-2a14-02b7449b****/logstash/1/ls.oss.eaced620-e972-0136-2a14-02b7449b****.2018-12-24T14.27.part-0.data

The path segments are described below:

What's next

• To install or remove plug-ins, see Install and remove a plug-in.

• To manage pipelines using configuration files, see Use configuration files to manage pipelines.