Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch - Elasticsearch

You can use an Alibaba Cloud Logstash pipeline to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch instance.

Limitations

The ECS instances that host your self-managed Elasticsearch cluster must be deployed in a VPC. ECS instances connected through ClassicLink are not supported.
Alibaba Cloud Logstash instances are deployed in a VPC. If your self-managed Elasticsearch cluster and Alibaba Cloud Logstash instance are in the same VPC, they can connect directly. If they are in different VPCs, you must use a NAT Gateway to connect over the public internet. For more information, see Configure data transmission over the public network by using a NAT Gateway.
The security group for the ECS instances that host your self-managed Elasticsearch cluster must allow traffic on port 9200 from all node IP addresses of the Logstash instance. You can find the node IP addresses on the Basic Information page.
This topic uses a self-managed Elasticsearch 8.17 cluster, an Alibaba Cloud Elasticsearch 8.17 instance, and an Alibaba Cloud Logstash 8.11.4 instance as an example. The provided scripts apply only to this scenario and may not be compatible with other configurations.

Procedure

Step 1: Prepare the environment
Step 2: Configure and run the Logstash pipeline
Step 3: View the migration results

Step 1: Prepare the environment

Set up a self-managed Elasticsearch cluster.

Deploy the self-managed Elasticsearch cluster on Alibaba Cloud ECS. This topic uses version 8.17 as an example. For more information, see Install and run Elasticsearch.
Create an Alibaba Cloud Logstash instance.

Create the Logstash instance in the same VPC as the ECS instances that host your self-managed Elasticsearch cluster. For more information, see Create an Alibaba Cloud Logstash instance.
Create the destination Alibaba Cloud Elasticsearch instance and enable auto indexing for it.
- Create an Elasticsearch instance in the same VPC and with the same version as your Logstash instance. This topic uses version 8.17 as an example. For more information, see Create an Alibaba Cloud Elasticsearch instance.
- To learn how to enable auto indexing, see Configure YML parameters.
  
  Note
  Logstash synchronizes only data, not data structures. Enabling auto indexing can lead to inconsistent data structures after migration. To ensure consistency, create an empty index in the destination Elasticsearch instance before you start. When you create the index, copy the mappings and settings from the source index and configure an appropriate number of shards.

Step 2: Configure and run the Logstash pipeline

Go to the Logstash Clusters page.
Navigate to the target cluster.
1. In the top navigation bar, select the region where the cluster resides.
2. On the Logstash Clusters page, find the cluster and click its ID.
In the left-side navigation pane, click Pipelines.
Click Create Pipeline.

On the Create page, enter a pipeline ID and configure the pipeline.

Use the following pipeline configuration.

input {
  elasticsearch {
    hosts => ["http://<IP address of the self-managed Elasticsearch master node>:9200"]
    user => "elastic"
    index => "*,-.monitoring*,-.security*,-.kibana*"
    password => "your_password"
    docinfo => true
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
    user => "elastic"
    password => "your_password"
    index => "%{[@metadata][input][elasticsearch][_index]}"
    document_id => "%{[@metadata][input][elasticsearch][_id]}"
  }    
  file_extend {
        path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
    }
}

Table 1. Parameters

Parameter	Description
hosts	The endpoint of the self-managed or Alibaba Cloud Elasticsearch service. In the input section, use the format `http://<IP address of the self-managed Elasticsearch master node>:<port>`. In the output section, use the format `http://<ID of the Alibaba Cloud Elasticsearch instance>.elasticsearch.aliyuncs.com:9200`. Important Replace <IP address of the self-managed Elasticsearch master node>, <port>, and <ID of the Alibaba Cloud Elasticsearch instance> with your actual values.
user	The username to access the self-managed or Alibaba Cloud Elasticsearch service. Important The user and password parameters are required. If your self-managed Elasticsearch cluster does not have X-Pack installed, you can leave these parameters empty. The default username for accessing an Alibaba Cloud Elasticsearch instance is elastic, which is used in this guide. To use a custom user, you must assign the necessary roles and permissions to the user. For more information, see Use Elasticsearch X-Pack for role-based access control.
password	The password to access the self-managed or Alibaba Cloud Elasticsearch service.
index	Specify the names of the indexes to synchronize. Set the input parameter to ,-.monitoring,-.security,-.kibana to synchronize all indexes except for system indexes that start with a `.`. Set the output parameter to %{[@metadata][input][elasticsearch][_index]} to match the index name from the metadata. This ensures that the index names in Alibaba Cloud Elasticsearch are the same as the index names in your self-managed Elasticsearch.
docinfo	When set to true, the `elasticsearch` input plugin extracts document metadata, such as index, type, and ID, from the self-managed Elasticsearch documents.
document_id	If set to %{[@metadata][input][elasticsearch][_id]}, the document ID is matched from the metadata. This ensures that document IDs in the destination instance match those in the source.
file_extend	Optional. Enables debug logging and uses the path parameter to set the output path for debug logs. We recommend that you configure this parameter. After you enable it, you can view the output directly in the console. If you do not enable it, you must verify the output on the destination and then return to the console to modify the pipeline, which is a time-consuming process. For more information, see Use pipeline configuration debugging. Important The file_extend parameter requires the logstash-output-file_extend plugin. Install or uninstall plugins. The path parameter defaults to a system-specified path. Do not modify it. You can also retrieve the path by clicking Start Configuration Debug.

The Elasticsearch input plugin is useful for tasks like bulk-importing test logs by reading documents from an Alibaba Cloud Elasticsearch cluster. By default, this process stops after reading all data. However, Alibaba Cloud Logstash instances run continuously and automatically restart stopped processes. In single-task scenarios, this behavior can cause duplicate data writes. To prevent this, use the schedule parameter with a cron expression to run the task only once. The pipeline stops after this single execution, preventing repetition. For more information, see the official Logstash documentation on Scheduling.

For example, to schedule the task to run at 13:20 on March 5:

schedule => "20 13 5 3 *"

For more information about pipeline configuration options, see Logstash configuration files.

Click Next step and configure the pipeline parameters.

Parameter	Description
Pipeline Workers	Number of worker threads for the filter and output stages. Increase this value if events are backlogged or CPU is underutilized. Default: the number of CPU cores.
Pipeline Batch Size	Maximum events a worker collects before executing filters and outputs. Larger batches increase memory usage and may require a larger JVM heap size (LS_HEAP_SIZE). Default: 125.
Pipeline Batch Delay	Wait time in milliseconds before dispatching an undersized batch to a worker thread. Default: 50 ms.
Queue Type	Internal queuing model for event buffering between stages. Valid values: MEMORY: Default. Uses an in-memory queue. PERSISTED: A disk-based persistent queue.
Queue Max Bytes	The maximum amount of data the queue can store, in `MB`. The value must be an integer from `1` to `2⁵³-1`. Default value: `1024`. Note Make sure that this value is less than your total disk capacity.
Queue Checkpoint Writes	Maximum events written before a checkpoint is forced (persistent queue only). 0 means no limit. Default: 1024.

Warning

Saving and deploying the configuration triggers an instance restart. Proceed only when this will not affect your business.

Click Save or Save and Deploy.
- Save: Saves the pipeline configuration but does not apply it. After saving, you are returned to the Pipelines page. In the Pipelines section, you can click Deploy Now in the Actions column to restart the instance and apply the configuration.
- Save and Deploy: Saves and deploys the configuration, restarting the instance to apply the changes.

Step 3: View the migration results

In Alibaba Cloud Elasticsearch, log on to the Kibana console. In the left-side navigation pane, click the icon and select Management > Dev Tools.

Note
This topic uses Alibaba Cloud Elasticsearch 8.17 as an example. The user interface may differ depending on your version.
In the Console, run the GET /_cat/indices?v command to view the successfully migrated indices.

The returned results show that the orders index has a health of green, a status of open, and a docs.count of 1, indicating that the data has been successfully migrated.

FAQ

Q: How do I establish network connectivity if the ECS instances hosting the self-managed Elasticsearch cluster and the Alibaba Cloud Logstash instance belong to different accounts?

A: If the ECS instances and the Logstash instance belong to different accounts, they must be in different VPCs. Connect the two VPCs by using Cloud Enterprise Network (CEN). For more information, see Step 3: Attach network instances.
Q: What do I do if an error occurs when Logstash writes data?

A: See Troubleshoot Logstash data write issues for diagnosis and resolution steps.

Q: Can I use Alibaba Cloud Logstash 8.11 to migrate data from a self-managed Elasticsearch 7.2 instance to an Alibaba Cloud Elasticsearch 7.10 instance?

A: Yes. However, you must meet the following key requirements:

Network and authentication

self-managed Elasticsearch → Alibaba Cloud Logstash
- The self-managed Elasticsearch instance must have a public IP and a configured allowlist.
- Configure authentication for the input plugin: user/password (Basic Auth) or an SSL certificate.
Alibaba Cloud Logstash → Alibaba Cloud Elasticsearch
- Use a direct private connection within the same VPC (recommended). This method provides password-free access authorized by instance association.
- If the instances are in different VPCs, configure a security group and an allowlist.

Example Logstash configuration

input {
  elasticsearch {
    hosts => ["http://<public_IP_of_self-managed_ES>:9200"]
    user => "self_es_user"
    password => "self_es_password"
    index => "source_index"
    query => '{ "query": { "match_all": {} } }'
    size => 500
    scroll => "5m"
    docinfo => true  # Preserves _source metadata, which is used to specify _id on write.
  }
}
filter {
  # Optional: Add field cleanup, mapping conversion, and more.
}
output {
  elasticsearch {
    hosts => ["https://<internal_endpoint_of_Alibaba_Cloud_ES>:9200"]
    # Alibaba Cloud Logstash typically injects authentication automatically after you select the destination Elasticsearch instance.
    index => "target_index"
    document_id => "%{[@metadata][_id]}"  # Ensures consistent document IDs.
    ssl => true
    ssl_certificate_verification => false # You can disable this in test environments. We recommend that you use a CA certificate in production.
  }
}