Use the logstash-output-oss plug-in

Prerequisites

• The logstash-output-oss plug-in is installed.

  For more information, see Install a Logstash plug-in.

• OSS is activated.

  For more information, see Activate OSS.

• A read/write OSS bucket is created. The AccessKey ID and AccessKey secret of your account that has write permissions on the OSS bucket are obtained.

  For more information, see Create buckets.

• Data sources are prepared.

  The data sources can be selected from all the input plug-ins that are supported. For more information, see Input plugins.

logstash-output-oss application

After the prerequisites are met, you can create a pipeline based on the instructions provided in Use configuration files to manage pipelines. When you create a pipeline, configure the pipeline parameters based on the descriptions that are provided in the table of the Parameters section. After you configure the parameters, you must save the settings and deploy the pipeline. Then, Alibaba Cloud Logstash is triggered to transfer data from data sources to OSS.

input {
  beats {
    port => "8044"
    codec => json {
      charset => "UTF-8"

  }
}
output {
   oss {
    endpoint => "http://oss-cn-hangzhou-internal.aliyuncs.com"              
    bucket => "zl-log-output-test"                          
    access_key_id => "LTAIaxxxxx******"                 
    access_key_secret => "zuxxxx8hBpXs3e6i******"         
    prefix => "oss/database"                         
    recover => true                                      
    rotation_strategy => "size_and_time"                  
    time_rotate => 1                                     
    size_rotate => 1000
    temporary_directory => "/ssd/1/<Logstash cluster ID>/logstash/data/22"                            
    encoding => "gzip"                                 
    additional_oss_settings => {
      max_connections_to_oss => 1024                      
      secure_connection_enabled => false                  
    }
    codec => json {
      charset => "UTF-8"
    }
  }
}

Parameters

Temporary files

When the logstash-output-oss plug-in transfers data to OSS, the plug-in creates a temporary file on your on-premises machine. Data is temporarily stored in this file. The logstash-output-oss plug-in transfers data to OSS on a regular basis. You can use the temporary_directory parameter to specify the path of the temporary file if you have specific requirements on the path.

The following example shows a temporary file path:

/ssd/1/<Logstash cluster ID>/logstash/data/eaced620-e972-0136-2a14-02b7449b****/logstash/1/ls.oss.eaced620-e972-0136-2a14-02b7449b****.2018-12-24T14.27.part-0.data