Elasticsearch:Manage Logstash pipelines with configuration files

Limitations

• A maximum of 20 pipelines can run in parallel.

• If the output is an Alibaba Cloud Elasticsearch instance, you must enable automatic index creation or create the index and mappings beforehand.

• All Alibaba Cloud services must be in the same VPC. For cross-VPC access, Configure NAT public network data transfer.

• To use the file_extend output parameter, first install the logstash-output-file_extend plugin. Install or uninstall plugins.

Create a pipeline

1. Go to the Logstash Clusters page.

2. Navigate to the target cluster.

   1. In the top navigation bar, select the region where the cluster resides.

   2. On the Logstash Clusters page, find the cluster and click its ID.

3. In the left-side navigation pane, click Pipelines.

4. Click Create Pipeline.

5. Enter the Pipeline ID and Config Settings.

   Sample configuration:

   input {
       beats {
           port => 8000
       }
   }
   filter {
   
   }
   output {
       elasticsearch {
           hosts => ["http://es-cn-o40xxxxxxxxxx****.elasticsearch.aliyuncs.com:9200"]
           index => "logstash_test_1"
           password => "es_password"
           user => "elastic"
       }
       file_extend {
           path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
       }
   }

   Parameter	Description
   input	The data source. Supported types are listed in input plugins. Note Input plugins that listen on a port must use a port in the range 8000 to 9000. To add plugins, drivers, or other files to the input section, click Show Third-party Libraries. In the Third-party Libraries dialog box, click Upload to upload files. configure extended files.
   filter	Filters input data. Supported plugin types are listed in filter plugins.
   output	The output destination. Supported types are listed in output plugins. file_extend: Optional. Enables debug logging and sets the debug log output path with the path parameter. Use this parameter to view output directly in the console instead of verifying at the destination. Use the Logstash pipeline configuration debugging feature. Important The file_extend parameter requires the logstash-output-file_extend plugin. Install or uninstall plugins. The path parameter defaults to a system-specified path. Do not modify it. You can also retrieve the path by clicking Start Configuration Debug.

   Configuration file structure and supported data types may vary by version. Structure of a Config File.

   Important

   • When using a JDBC driver, append allowLoadLocalInfile=false&autoDeserialize=false to the jdbc_connection_string parameter to pass pipeline validation. Example: jdbc_connection_string => "jdbc:mysql://xxx.drds.aliyuncs.com:3306/<database_name>?allowLoadLocalInfile=false&autoDeserialize=false".

   • For parameters such as last_run_metadata_path, use the /ssd/1/<Logstash_instance_ID>/logstash/data/ path for testing. Data in this directory persists, so ensure sufficient disk space. Logstash auto-generates a file at the specified path, but you cannot view its content.

   • Logstash instances run in a VPC. Use resources in the same VPC when possible. For internet access, Configure NAT public network data transfer.

   • Use the file_extend plugin for debug logging instead of stdout.

6. Click Next step and configure the pipeline parameters.

   Parameter	Description
   Pipeline Workers	Number of worker threads for the filter and output stages. Increase this value if events are backlogged or CPU is underutilized. Default: the number of CPU cores.
   Pipeline Batch Size	Maximum events a worker collects before executing filters and outputs. Larger batches increase memory usage and may require a larger JVM heap size (LS_HEAP_SIZE). Default: 125.
   Pipeline Batch Delay	Wait time in milliseconds before dispatching an undersized batch to a worker thread. Default: 50 ms.
   Queue Type	Internal queuing model for event buffering. Valid values: MEMORY: Default. Uses an in-memory queue. PERSISTED: A disk-based persistent queue.
   Queue Max Bytes	The maximum amount of data the queue can store, in MB. The value must be an integer from 1 to 253-1. Default value: 1024. Note Make sure that this value is less than your total disk capacity.
   Queue Checkpoint Writes	Maximum events written before a checkpoint is forced (persistent queue only). 0 means no limit. Default: 1024.

   Warning

   Saving and deploying triggers an instance restart. Proceed only if this will not impact your business.

7. Click Save or Save and Deploy.

   • Save: Saves the pipeline configuration but does not apply it. After saving, you are returned to the Pipelines page. In the Pipelines section, you can click Deploy Now in the Actions column to restart the instance and apply the configuration.

   • Save and Deploy: Saves and deploys the configuration, restarting the instance to apply the changes.

8. In the success message, click OK. You can view the newly created pipeline in the Pipelines.

   The pipeline is created after the instance restarts.

Modify a pipeline

1. In the Pipelines section, find the target pipeline and click Modify in the Actions column.

2. On the Modify page, modify the Config Settings and Pipeline Parameters. The Pipeline ID cannot be modified.

3. Click Save or Save and Deploy. The pipeline is modified after the instance finishes restarting.

Clone a pipeline

1. In the Pipelines section, find the target pipeline and in the Actions column, click > Copy.

2. On the Copy page, change the Pipeline ID. The other settings remain unchanged.

3. Click Save or Save and Deploy. The pipeline is cloned after the instance finishes restarting.

Delete a pipeline

1. In the Pipelines section, find the target pipeline and in the Actions column, click > Delete.

2. In the Delete dialog box, review the risk warning.

3. Click Continue. The pipeline is deleted after the instance change is complete.

Related documents

• Create pipelines API: CreatePipelines

• Best practice for Logstash data migration: Migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster by using Alibaba Cloud Logstash

• Synchronize MySQL data with Logstash: Synchronize data from an ApsaraDB RDS for MySQL database to an Elasticsearch cluster by using Logstash

• Synchronize log data with Logstash: logstash-input-sls plugin user guide

Related topics

• Logstash FAQ

• Troubleshoot Logstash data write issues

• Troubleshoot Logstash performance issues