Elasticsearch:Use the logstash-output-datahub plug-in

Prerequisites

• The logstash-output-datahub plug-in is installed.

  For more information, see Install a Logstash plug-in.

• DataHub is activated, a project is created, and a topic is created for the project.

  For more information, see Get started with DataHub.

• The data source from which you want to read data is prepared.

  Available data sources are determined by the input plug-ins supported by Logstash. For more information, see Input plugins. In this example, an Alibaba Cloud Elasticsearch cluster is used.

Use logstash-output-datahub

Create a pipeline by following the instructions provided in Use configuration files to manage pipelines. When you create the pipeline, configure the pipeline parameters based on the descriptions that are described in the table of the Parameters section. After you configure the parameters, save the settings and deploy the pipeline. This way, Logstash can be triggered to read data from the data source and transfer the data to DataHub.

The following code provides a pipeline configuration example. For more information about the parameters, see Parameters.

input {
    elasticsearch {
      hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
      user => "elastic"
      index => "test"
      password => "your_password"
      docinfo => true
  }
}
filter{
    
}
output {
    datahub {
        access_id => "Your accessId"
        access_key => "Your accessKey"
        endpoint => "Endpoint"
        project_name => "project"
        topic_name => "topic"
        #shard_id => "0"
        #shard_keys => ["thread_id"]
        dirty_data_continue => true
        dirty_data_file => "/ssd/1/<Logstash cluster ID>/logstash/data/File name"
        dirty_data_file_max_size => 1000
    }
}

Parameters