Use the logstash-input-maxcompute plug-in

Prerequisites

You have completed the following operations:

• The logstash-input-maxcompute plug-in is installed.

  For more information, see Install a Logstash plug-in.

• Alibaba Cloud MaxCompute is activated. A project is created. A table is created for the project. Data is imported to the table.

  For more information, see Prepare and Quick Start.

logstash-input-maxcompute application

After the prerequisites are met, you can create a pipeline task by following the instructions provided in Use configuration files to manage pipelines. For more information about the prerequisites, see Prerequisites. When you create a pipeline task, set the pipeline parameters by following the descriptions that are provided in the table of the Parameters section. After you set the parameters, you must save the settings and deploy the pipeline. Then, Alibaba Cloud Logstash is triggered to read data from MaxCompute to destination data sources.

The following example shows a configuration script. For more information about the parameters, see Parameters.

input {
    maxcompute {
        access_id => "Your accessId"
        access_key => "Your accessKey"
        endpoint => "maxcompute service endpoint"
        project_name => "Your project"
        table_name => "Your table name"
        partition => "pt='p1',dt='d1'"
        thread_num => 1
        dirty_data_file => "/ssd/1/tmp/xxxxx"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

Parameters