Elasticsearch:Use the logstash-input-datahub plug-in

Prerequisites

• The logstash-input-datahub plug-in is installed.

  For more information, see Install and remove a plug-in.

• DataHub is activated, a project is created, a topic is created for the project, and data is imported to the topic.

  For more information, see Get started with DataHub.

Use logstash-input-datahub

Create a pipeline by following the instructions provided in Use configuration files to manage pipelines. When you create the pipeline, configure the pipeline parameters based on the descriptions that are described in the table of the Parameters section. After you configure the parameters, save the settings and deploy the pipeline. This way, Logstash can be triggered to read data from DataHub and transfer the data to the destination data source.

The following code provides a pipeline configuration example. For more information about the parameters, see Parameters.

input {
    datahub {
        access_id => "Your accessId"
        access_key => "Your accessKey"
        endpoint => "Endpoint"
        project_name => "test_project"
        topic_name => "test_topic"
        interval => 5
        #cursor => {
        #    "0"=>"20000000000000000000000003110091"
        #    "2"=>"20000000000000000000000003110091"
        #    "1"=>"20000000000000000000000003110091"
        #    "4"=>"20000000000000000000000003110091"
        #    "3"=>"20000000000000000000000003110000"
        #}
        shard_ids => []
        pos_file => "/ssd/1/<Logstash cluster ID>/logstash/data/File name"
    }
}
output {
    elasticsearch {
      hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
      user => "elastic"
      password => "your_password"
      index => "datahubtest"
      document_type => "_doc"
  }
}

Parameters