Elasticsearch:Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch over a private connection

Prerequisites

• The self-managed Elasticsearch cluster must meet the following requirements:

  • The ECS instance that hosts the cluster must be in a virtual private cloud (VPC). ClassicLink is not supported. The ECS instance and the Alibaba Cloud Elasticsearch instance must be in the same VPC.

  • The security group of the ECS instance must allow access from all node IP addresses of the Alibaba Cloud Elasticsearch instance. You can find the node IP addresses in the Kibana console. Port 9200 must also be open.

  • A connection is established with the Alibaba Cloud Elasticsearch instance. You can run the curl -XGET http://<host>:9200 command from the machine where you run the script to verify the connection.

    Note

    You can run the scripts in this topic from any machine. The machine must be able to access port 9200 of both the self-managed Elasticsearch cluster and the Alibaba Cloud Elasticsearch cluster.

  • Prepare the index data in the self-managed Elasticsearch cluster. This topic uses the `source` index that is shown in the following figure as an example.

• The Alibaba Cloud Elasticsearch instance must meet the following requirements:

  • Enable the automatic index creation feature in advance. Alternatively, you can create the index mappings and settings on the Alibaba Cloud Elasticsearch instance.

  • No whitelist restrictions are configured.

Limitations

Procedure

1. Step 1: Configure an SLB instance that supports PrivateLink

   Currently, only Server Load Balancer (SLB) instances that support PrivateLink can be used as service resources for an endpoint service. Before you use PrivateLink to privately access a service between VPCs, you must create an SLB instance that supports PrivateLink and configure its listener.

2. Step 2: Create an endpoint service

   An endpoint service allows other VPCs to establish private connections by creating endpoints. After you configure the SLB instance, you must create an endpoint service.

3. Step 3: Configure private network peering for Alibaba Cloud Elasticsearch

   In the Elasticsearch console, you can associate the Alibaba Cloud Elasticsearch instance with the endpoint service that you created in Step 2.

4. Step 4: Obtain the endpoint domain name

   After the target instance is associated with the endpoint service, you must obtain the endpoint domain name. You will use this domain name to configure the reindex whitelist.

5. Step 5: Configure the reindex whitelist

   In the Elasticsearch console, add the domain name that you obtained in Step 4 to the reindex whitelist of the target instance for authorization.

6. Step 6: Migrate data

   After you complete these steps, a private connection is established between the source and target instances. You can now migrate the data.

Step 1: Configure an SLB instance that supports PrivateLink

1. Create an SLB instance.

   Make sure that the SLB instance is in the same region as the backend Elastic Computing Service (ECS) instances. For more information, see Create an SLB instance that supports PrivateLink.

2. Configure the Protocol & Listener by setting the load balancing protocol to TCP and the port to 9200.

   

   For more information, see Configure protocol and listener.

3. Click Next. Configure the backend server group by adding the ECS instance that hosts your self-managed ES cluster and setting the port to 9200.

   For more information, see Configure backend servers.

4. Click Next. In the Configure the Health Check section, configure the settings as needed. This topic uses the default settings.

5. Click Next. Review the configuration and click Submit. Click OK. Return to the Instance Management page and check the health check status of the backend ECS instance.

   If the health check status of the backend ECS instance is Normal, the instance can process requests that are forwarded by the SLB instance.

Step 2: Create an endpoint service

1. Log on to the endpoint service console.

2. In the top menu bar, select the region where you want to create the endpoint service.

   The example in this topic uses the China (Hangzhou) region.

3. Click Create Endpoint Service. On the Create Endpoint Service page, configure the endpoint service according to your requirements.

   For specific operations and detailed instructions, see Create and manage an endpoint service. The following table describes the configurations that are used in this topic. For parameters that are not mentioned, you can configure them as needed or keep the default values.

   Parameter	Description
   Select Service Resource	Select the zone to carry the traffic. Then, select the SLB instance to associate with the endpoint service. The SLB instance acts as the service resource and is associated with the endpoint service. The associated SLB instance accepts network access from your service users. The zone of the endpoint service is the same as the primary zone of the service resource. An SLB instance can be used as a service resource only if it meets two conditions: The network type is VPC. The feature is Supports PrivateLink.
   Automatically Accept Endpoint Connections	Select whether to automatically accept connection requests from endpoints. We recommend that you select Yes. The values are described as follows: Yes: The endpoint service automatically accepts connection requests from endpoints. The service can be accessed through the endpoints. No: The endpoint connection will be in the Disconnected state, waiting for the service administrator to process it: If the service administrator accepts the endpoint connection, the service can be accessed through the endpoint. If the service administrator rejects the endpoint connection, the service cannot be accessed through the endpoint.
   Enable Zone Affinity	Select whether to enable zone affinity. We recommend that you select Yes.

4. Click OK.

Step 3: Configure private network peering for Alibaba Cloud Elasticsearch

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, click Elasticsearch Clusters.

3. Navigate to the desired cluster.

   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.

   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the left-side navigation pane of the page that appears, choose Configuration and Management > Security.

5. In the Network Settings section, click Edit on the right side of Configure Private Connection.

6. In the Configure Private Connection panel, click Add Private Connection. Select the endpoint service created in Step 2 and the target access zone. Then, select the checkbox for the system prompt.

   

7. Click OK. The endpoint service then connects to the endpoint. Once the connection is established, the status changes to Connected.

   

Step 4: Obtain the endpoint domain name

After you complete the preceding steps, you must obtain the endpoint domain name to configure the reindex whitelist.

1. In the Configure Private Connection panel, click the desired Endpoint ID.

   

2. On the Endpoint Connections tab, click the icon next to the target endpoint ID to view the corresponding domain name.

   

Step 5: Configure the reindex whitelist

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left-side navigation pane, click Elasticsearch Clusters.

3. Navigate to the desired cluster.

   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.

   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the navigation pane on the left, choose Configuration and Management > Cluster Configuration.

5. In the YML File Configuration section, click Modify Configuration on the right.

6. In the YML File Configuration panel, click Modify Configuration and add the domain name that you obtained in Step 4.

   Sample code:

   reindex.remote.whitelist: 'ep-bp1nitq0krp8yhcf****-cn-hangzhou-i.epsrv-bp1zczi0fgoc5qtv****.cn-hangzhou.privatelink.aliyuncs.com:9200'

   

7. Click OK.

Step 6: Migrate data

1. In Dev Tools on the Kibana console, run the following command to migrate data.

   Note

   For more information about how to log on to the Kibana console, see Log on to the Kibana console.

   POST /_reindex?pretty
   {
   
     "source": {
   
       "remote": {
   
         "host": "http://ep-bp1nitq0krp8yhcf****-cn-hangzhou-i.epsrv-bp1zczi0fgoc5qtv****.cn-hangzhou.privatelink.aliyuncs.com:9200",
   
         "username": "elastic",
   
         "password": "Elastic@123***"
   
       },
   
       "index": "source",
       "size": 5000
   
     },
   
     "dest": {
   
       "index": "dest"
   
     }
   
   }

   For more information, see the reindex API.

2. Optional: During data migration, you can run the following command to obtain the status of all running reindex requests.

   GET _tasks?detailed=true&actions=*reindex

3. View the data migration result.

   After the data migration is complete, you can run the following command to view the result.

   GET _cat/indices?

   If the health status of the target index `dest` is normal and the data size is as expected, the data migration is successful.

FAQ

Question: What do I do if a single index has a large volume of data and the data synchronization is slow?