Logstash uses pipelines to collect and process data. You must configure input and output plug-ins for pipelines. You can configure filter plug-ins based on your business requirements. The input and output plug-ins are used to configure input and output data sources, and the filter plug-ins are used to preprocess collected data. Alibaba Cloud Logstash allows you to run a maximum of 20 pipelines at the same time. This topic describes how to use the configuration file of an Alibaba Cloud Logstash cluster to manage pipelines.
Prerequisites
The following operations are performed:
- An Alibaba Cloud Elasticsearch cluster is created.
For more information, see Create an Alibaba Cloud Elasticsearch cluster.
- The Auto Indexing feature is enabled for the Alibaba Cloud Elasticsearch cluster,
or an index is created in the Alibaba Cloud Elasticsearch cluster and configured with
mappings. In this example, the Auto Indexing feature is used.
For more information about how to enable the Auto Indexing feature, see Configure the YML file. For more information about how to create an index and configure mappings for the index, see Step 3: Create an index.Note To ensure data security, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. When you use Alibaba Cloud Logstash to transfer data to an Alibaba Cloud Elasticsearch cluster, indexes are created in the Elasticsearch cluster by submitting data instead of calling the create index API. Therefore, before you use Alibaba Cloud Logstash to transfer data, you must enable the Auto Indexing feature for the destination Elasticsearch cluster, or create an index in the destination Elasticsearch cluster and configure mappings for the index.
- An Alibaba Cloud Logstash cluster is created.
For more information, see Create an Alibaba Cloud Logstash cluster.
Limits
- Alibaba Cloud Logstash allows you to run a maximum of 20 pipelines at the same time.
- If you specify an Alibaba Cloud Elasticsearch cluster in the output configuration of a pipeline, you must make sure that the Auto Indexing feature is enabled for the cluster or an index is created in the cluster and configured with mappings.
- If the data sources that you specify in the configurations of a pipeline belong to other Alibaba Cloud services, you must make sure that the data sources reside in the same virtual private cloud (VPC) as your Alibaba Cloud Logstash cluster. Otherwise, you must configure network and security settings for your Logstash cluster. For more information, see Configure a NAT gateway for data transmission over the Internet.
- If the file_extend parameter is configured in the output configuration of a pipeline, you must make sure that the logstash-output-file_extend plug-in is installed for the Alibaba Cloud Logstash cluster. For more information, see Install and remove a plug-in.
Create a pipeline
Modify a pipeline
- On the Pipelines page, find the pipeline that you want to modify and click Modify in the Actions column.
- In the Modify wizard, modify the settings in the Config Settings and Pipeline Parameters steps. You cannot change the value of Pipeline ID.
- Click Save or Save and Deploy. After the cluster is restarted, the pipeline modification task is complete.
Copy a pipeline
- On the Pipelines page, find the pipeline that you want to copy and choose in the Actions column.
- In the Copy wizard, configure Pipeline ID and retain other settings.
- Click Save or Save and Deploy. After the cluster is restarted, the pipeline copy task is complete.
Delete a pipeline
- After a pipeline is deleted, it cannot be recovered, and tasks that are running on the pipeline are stopped. Before you can proceed, make sure that the deletion does not affect your services.
- The deletion triggers an update of the Logstash cluster. Before you can proceed, make sure that the update does not affect your services.
- On the Pipelines page, find the pipeline that you want to delete and choose in the Actions column.
- In the Delete Pipeline message, view risk warnings.
- Click Continue. After the cluster is updated, the pipeline deletion task is complete.
References
- API operation for creating a pipeline: CreatePipelines
- Reference for using Logstash to migrate data: Use Alibaba Cloud Logstash to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
- Reference for using Logstash to synchronize log data: Use the logstash-input-sls plug-in