Elasticsearch:Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch over a private connection

Choose a migration path

Alibaba Cloud Elasticsearch (ES) provides two deployment modes: basic control (v2) architecture and cloud-native new control (v3) architecture. Check the deployment mode in the Basic Information section of your instance before proceeding.

Select a solution based on your scenario:

Prerequisites

Before you begin, ensure that you have:

Self-managed Elasticsearch cluster:

• An ECS instance in a virtual private cloud (VPC). ClassicLink is not supported.

• The ECS instance and the Alibaba Cloud Elasticsearch instance in the same VPC.

• Port 9200 open in the ECS security group for all node IP addresses of the Alibaba Cloud Elasticsearch instance. Find the node IP addresses in the Kibana console.

• A verified connection to both clusters. Run the following command from any machine that can reach port 9200 on both clusters:

  curl -XGET http://<host>:9200

• Index data prepared in the self-managed Elasticsearch cluster. The following figure shows the source index used as an example.

  

Alibaba Cloud Elasticsearch instance:

• Automatic index creation enabled, or the index mappings and settings already created on the instance.

• No whitelist restrictions configured.

Migration overview

Migrating data requires establishing a private connection between the self-managed cluster and the Alibaba Cloud ES instance before running the reindex API. The process has six steps:

1. Configure an SLB instance that supports PrivateLink — Set up a Server Load Balancer (SLB) instance as the service resource for the private connection.

2. Create an endpoint service — Expose the SLB instance as an endpoint service that the Alibaba Cloud ES instance can connect to.

3. Configure private network peering for Alibaba Cloud Elasticsearch — Associate the Alibaba Cloud ES instance with the endpoint service.

4. Get the endpoint domain name — Retrieve the domain name used to configure the reindex whitelist.

5. Configure the reindex whitelist — Authorize the Alibaba Cloud ES instance to pull data from the self-managed cluster via the private connection.

6. Migrate data — Run the reindex command to copy data to the target index.

Step 1: Configure an SLB instance that supports PrivateLink

Only SLB instances that support PrivateLink can serve as service resources for an endpoint service.

1. Create an SLB instance in the same region as the ECS instance that hosts your self-managed ES cluster. For details, see Create an SLB instance that supports PrivateLink.

2. Configure Protocol & Listener: set the load balancing protocol to TCP and the port to 9200. For details, see Configure protocol and listener.

   

3. Click Next. Configure the backend server group: add the ECS instance that hosts your self-managed ES cluster and set the port to 9200. For details, see Configure backend servers.

4. Click Next. Configure the health check settings as needed. The default settings are used in this procedure.

5. Click Next. Review the configuration, then click Submit and OK. On the Instance Management page, verify that the health check status of the backend ECS instance is Normal, which confirms the instance can process requests forwarded by the SLB instance.

Step 2: Create an endpoint service

1. Log on to the endpoint service console.

2. In the top menu bar, select the region where you want to create the endpoint service. The following steps use the China (Hangzhou) region as an example.

3. Click Create Endpoint Service and configure the parameters. For full instructions, see Create and manage an endpoint service. The following table describes the key parameters.

   Parameter	Description
   Select service resource	Select the zone to carry the traffic, then select the SLB instance created in Step 1. The SLB instance must have VPC as the network type and support PrivateLink.
   Automatically accept endpoint connections	Select Yes so the endpoint service automatically accepts connection requests.
   Enable zone affinity	Select Yes.

4. Click OK.

Step 3: Configure private network peering for Alibaba Cloud Elasticsearch

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, click Elasticsearch Clusters.

3. In the top navigation bar, select the resource group and region of your cluster. On the Elasticsearch Clusters page, click the cluster ID.

4. In the left-side navigation pane, choose Configuration and Management > Security.

5. In the Network Settings section, click Edit next to Configure Private Connection.

6. In the Configure Private Connection panel, click Add Private Connection. Select the endpoint service created in Step 2 and the target access zone, then select the system prompt checkbox.

   

7. Click OK. Once the connection is established, the status changes to Connected.

   

Step 4: Get the endpoint domain name

After the private connection is established, retrieve the endpoint domain name to use in the reindex whitelist configuration.

1. In the Configure Private Connection panel, click the Endpoint ID.

   

2. On the Endpoint Connections tab, click the icon next to the target endpoint ID to view the domain name.

   

Step 5: Configure the reindex whitelist

The Elasticsearch reindex API requires an explicit whitelist to allow the cluster to pull data from a remote host. Add the endpoint domain name to this whitelist to authorize the private connection.

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, click Elasticsearch Clusters.

3. In the top navigation bar, select the resource group and region of your cluster. On the Elasticsearch Clusters page, click the cluster ID.

4. In the left-side navigation pane, choose Configuration and Management > Cluster Configuration.

5. In the YML File Configuration section, click Modify Configuration.

6. In the YML File Configuration panel, click Modify Configuration and add the domain name obtained in Step 4:

   reindex.remote.whitelist: 'ep-bp1nitq0krp8yhcf****-cn-hangzhou-i.epsrv-bp1zczi0fgoc5qtv****.cn-hangzhou.privatelink.aliyuncs.com:9200'

   

7. Click OK.

Step 6: Migrate data

1. In Dev Tools on the Kibana console, run the following reindex command.

   For instructions on logging on to the Kibana console, see Log on to the Kibana console.

   POST /_reindex?pretty
   {
     "source": {
       "remote": {
         "host": "http://ep-bp1nitq0krp8yhcf****-cn-hangzhou-i.epsrv-bp1zczi0fgoc5qtv****.cn-hangzhou.privatelink.aliyuncs.com:9200",
         "username": "elastic",
         "password": "Elastic@123***"
       },
       "index": "source",
       "size": 5000
     },
     "dest": {
       "index": "dest"
     }
   }

   For more information, see the reindex API.

2. (Optional) While migration is running, check the status of all active reindex tasks:

   GET _tasks?detailed=true&actions=*reindex

3. After migration completes, verify the result:

   GET _cat/indices?

   If the dest index shows a normal health status and the data size matches expectations, the migration is successful.

   

FAQ

Migration is slow for a large index. How do I speed it up?

To reduce write overhead during migration, set the replica count to 0 and disable the refresh interval before starting. The defaults are number_of_replicas: 1 and refresh_interval: 1s. Run the following commands:

// Before migrating index data, set the number of replicas to 0 and disable refresh to speed up migration.
curl -u user:password -XPUT 'http://<host:port>/indexName/_settings' -d' {
        "number_of_replicas" : 0,
        "refresh_interval" : "-1"
}'

After migration completes, restore the default values:

// After migrating index data, reset the number of replicas to 1 and the refresh interval to 1s (the default value).
curl -u user:password -XPUT 'http://<host:port>/indexName/_settings' -d' {
        "number_of_replicas" : 1,
        "refresh_interval" : "1s"
}'

The reindex API uses scroll internally. To parallelize the operation, increase the scroll size parameter or configure scroll slices. For details, see the reindex API.

For very large datasets, consider using the Object Storage Service (OSS) snapshot method instead. See Advanced: Migrate a self-managed ES cluster to Alibaba Cloud ES using OSS.