Elasticsearch:Use the logstash-input-sls plug-in

Key capabilities

• Distributed consumption: Deploy one pipeline per server across multiple servers and have them consume the same Logstore in parallel. All servers share the same consumer_group and consumer_name, with consumer_name_with_ip set to true to make each consumer uniquely identifiable.

• High throughput: A single-core CPU can sustain 20 MB/s using the Java consumer group implementation.

• Checkpoint-based reliability: The plug-in saves the consumption progress on each server. If a server restarts after a failure, consumption resumes from the last checkpoint.

• Automatic shard rebalancing: Shards are redistributed across active consumers whenever a consumer joins or leaves the group.

Prerequisites

Before you begin, ensure that you have:

• The logstash-input-sls plug-in installed. See Install and remove a plug-in.

• An SLS project and Logstore with data collected. See Quick start.

• (If using a RAM user) Consumer group permissions granted to that user. See Use consumer groups to consume data.

Configure the pipeline

Create a pipeline configuration file following the instructions in Use configuration files to manage pipelines. After saving and deploying the pipeline, Alibaba Cloud Logstash starts retrieving data from SLS.

The following example pulls log data from an SLS Logstore and writes it to Alibaba Cloud Elasticsearch:

input {
  logservice{
    endpoint => "your project endpoint"
    access_id => "your access id"
    access_key => "your access key"
    project => "your project name"
    logstore => "your logstore name"
    consumer_group => "consumer group name"
    consumer_name => "consumer name"
    position => "end"
    checkpoint_second => 30
    include_meta => true
    consumer_name_with_ip => true
  }
}

output {
  elasticsearch {
    hosts => ["http://es-cn-***.elasticsearch.aliyuncs.com:9200"]
    index => "<your_index>"
    user => "elastic"
    password => "changeme"
  }
}

Parameters

Best practices

Performance benchmark

The following test results show throughput and resource usage on a 4-core Intel Xeon Platinum 8163 @ 2.50 GHz with 8 GB memory running Linux.

Test setup: A Java producer sends log entries (10 key-value pairs, ~500 bytes each) at increasing rates. Logstash consumes from the Logstore using the pipeline configuration below, writing output to Elasticsearch. The test verifies that consumption latency does not increase and consumption speed keeps pace with the ingest rate.

input {
  logservice{
    endpoint => "cn-hangzhou-intranet.log.aliyuncs.com"
    access_id => "***"
    access_key => "***"
    project => "test-project"
    logstore => "logstore1"
    consumer_group => "consumer_group1"
    consumer_name => "consumer1"
    position => "end"
    checkpoint_second => 30
    include_meta => true
    consumer_name_with_ip => true
  }
}
output {
  elasticsearch {
    hosts => ["http://es-cn-***.elasticsearch.aliyuncs.com:9200"]
    index => "myindex"
    user => "elastic"
    password => "changeme"
  }
}

Results:

CPU usage scales linearly with traffic. Memory usage remains constant at 1.3 GB across all traffic levels.

What's next

• Use consumer groups to consume data: Learn how to manage consumer group permissions and monitor consumption progress.

• Use configuration files to manage pipelines: Create and deploy Logstash pipeline configurations.