If you want to use Alibaba Cloud Logstash to collect data, you can use the configuration file of a Logstash cluster to create and configure pipelines. This method allows you to run a maximum of 20 pipelines at the same time. This topic describes how to use the configuration file of a Logstash cluster to manage pipelines.
Prerequisites
- An Alibaba Cloud Elasticsearch cluster is created, and the Auto Indexing feature is
enabled for the cluster.
For more information about how to create an Elasticsearch cluster, see Create clusters. For more information about how to enable the Auto Indexing feature, see Enable the Auto Indexing feature.Note To ensure data security, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. When you use Logstash to migrate data to an Elasticsearch cluster, indexes are created by submitting data instead of calling the create index operation. Therefore, before you use Logstash to migrate data, you must enable the Auto Indexing feature for the destination Elasticsearch cluster.
- An Alibaba Cloud Logstash cluster is created.
For more information, see Create an Alibaba Cloud Logstash cluster.
Background information
Logstash allows you to configure pipelines by using the following methods:
- Use the configuration file specified by
-f<path/to/file>
to configure a Logstash pipeline. - Use the pipelines.yml configuration file to enable the parallel running of pipelines in the same process.
- Use Kibana to access Logstash and configure a single-process pipeline.
Create a pipeline
Modify a pipeline
Warning If you click Save and Deploy after you modify a pipeline, the system restarts the Logstash cluster. Before you
can proceed, make sure that the restart does not affect your services.
- In the Pipelines section, find the pipeline that you want to modify and click Modify in the Actions column.
- In the Modify wizard, modify the settings in the Config Settings and Pipeline Parameters steps. You cannot change the value of Pipeline ID.
- Click Save or Save and Deploy. After the cluster is restarted, the pipeline modification task is complete.
Copy a pipeline
Warning If you click Save and Deploy after you copy a pipeline, the system restarts the Logstash cluster. Before you can
proceed, make sure that the restart does not affect your services.
- In the Pipelines section, find the pipeline that you want to copy, click More in the Actions column, and then select Copy.
- In the Copy wizard, specify Pipeline ID and retain other settings.
- Click Save or Save and Deploy. After the cluster is restarted, the pipeline copy task is complete.
Delete a pipeline
Warning
- After a pipeline is deleted, it cannot be recovered, and related ongoing pipeline tasks are stopped. Before you can proceed, make sure that the deletion does not affect your services.
- The deletion triggers an update of the Logstash cluster. Before you can proceed, make sure that the update does not affect your services.
- In the Pipelines section, find the pipeline that you want to delete and choose in the Actions column.
- In the Delete Pipeline wizard, check risk warnings.
- Click OK. After the cluster is updated, the pipeline deletion task is complete.