Elasticsearch:Migrate self-managed Elasticsearch data to Alibaba Cloud using the online migration tool

Self-managed Elasticsearch side

• You have deployed a self-managed Elasticsearch cluster on ECS. This cluster serves as the source cluster for migration.

  Note

  The ECS security group must allow the specified CIDR block to access the ECS instance through ports 9200 and 9300.

  • Port 9200: The HTTP API port for Elasticsearch. Clients—including browsers, applications, and Kibana—use this port to interact with the Elasticsearch cluster.

  • Port 9300: The TCP communication port for Elasticsearch. Nodes within the cluster use this port to communicate with each other.

• Your self-managed Elasticsearch cluster indexes have replicas.

• You have prepared a decrypted P12 file. A P12 file stores public and private keys and includes a digital certificate. You will use this file later to ensure normal communication between nodes during migration.

• (Optional) If your self-managed Elasticsearch has no authentication enabled, enable X-Pack Security by setting xpack.security.enabled: true to secure data during migration.

Alibaba Cloud Elasticsearch side

You have created an Alibaba Cloud Elasticsearch cluster. This cluster serves as the destination cluster to receive migrated data from your self-managed Elasticsearch. To ensure smooth migration, confirm that your Alibaba Cloud Elasticsearch cluster:

• Is deployed in Cloud-native new control plane (v3) mode.

• Is in the same region and VPC as your self-managed Elasticsearch ECS instance.

• Uses a version that matches your self-managed Elasticsearch version.

  Note

  Contact Alibaba Cloud Elasticsearch technical support to perform cross-version migration.

• Has instance specifications and node counts that are at least equal to those of your self-managed Elasticsearch cluster.

• Uses the same username and password as your self-managed Elasticsearch cluster.

• If your self-managed Elasticsearch uses custom plugins—for example, the elasticsearch-knn plugin for similarity search—install the same plugins on your Alibaba Cloud Elasticsearch cluster. For plugin-related operations, see Plugin configuration.

• Has cloud-based cluster features—such as cluster configuration and dictionaries—that meet your business expectations and are reachable over the network.

Step 1: Create a migration task

1. Access the online migration tool.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the navigation pane on the left, click Elastic Stack cloud-native PaaS version > Elasticsearch online migration tool to open the online migration tool.

2. Click Create to create a migration task and configure parameters.

   

   Parameter	Description
   Type	Select Self-managed Elasticsearch migration to migrate data from your self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch.
   Destination cluster	The Alibaba Cloud Elasticsearch instance that receives data from your self-managed Elasticsearch cluster. Select an instance in the same region as your self-managed Elasticsearch ECS instance. After selection, enter the virtual switch ID as prompted. Note You can select only instances deployed in Cloud-native new control plane (v3) mode. If no suitable instance exists, create an Alibaba Cloud Elasticsearch instance as needed. A virtual switch enables network isolation and access control. You can find its ID in the instance basic information.
   Self-managed ES VPC	The VPC ID of your self-managed Elasticsearch cluster.
   Self-managed ES ECS IP	The IP address of any ECS instance hosting a node in your self-managed Elasticsearch cluster. Note The ECS security group must allow the specified CIDR block to access the ECS instance through ports 9200 and 9300. Port 9200: The HTTP API port for Elasticsearch. Clients—including browsers, applications, and Kibana—use this port to interact with the Elasticsearch cluster. Port 9300: The TCP communication port for Elasticsearch. Nodes within the cluster use this port to communicate with each other.
   Self-managed ES P12 file	Upload the decrypted P12 file as prompted. A P12 file stores public and private keys and includes a digital certificate. You will use this file later to ensure normal communication between nodes during migration.
   Self-managed ES username	The username used to log on to your self-managed Elasticsearch cluster.
   Self-managed ES password	The password used to log on to your self-managed Elasticsearch cluster.

3. Click OK to complete task creation.

Step 2: Configure and run the migration task

In this step, configure and run the migration task as prompted.

1. Run migration checks and configure node fusion.

   On the Source and destination cluster settings tab, complete the following steps to check migration feasibility and configure node fusion.

   

   1. Feasibility check: Before migration, the system automatically checks network connectivity, version compatibility, and resource availability between the source and destination clusters to ensure safe and feasible migration.

      Note

      If the feasibility check fails, adjust configurations as prompted.

   2. Configure node fusion: The system detects configuration parameters from your source cluster—such as number of replica shards, sharding strategy, and user permissions—and adapts them to your destination cluster. This ensures seamless synchronization of configuration parameters, resource allocation, and sharding strategy for smooth migration and efficient operation.

      Note

      This step takes about 20 minutes. Please wait.

2. Migrate data.

   On the Data migration tab, follow the steps below to complete cluster fusion, replica shard migration, master node migration, and primary shard migration.

   Action	Description	Notes
   Cluster fusion	The system merges the source and destination clusters into one large cluster. After fusion, total nodes in the cluster = nodes in source cluster + nodes in destination cluster.	After cluster fusion, Kibana for the destination cluster is temporarily disabled. During migration, continue using Kibana for the source cluster until you complete the business cutover.
   Replica shard migration	The system migrates index replica shards from source cluster nodes to destination cluster nodes.	If unexpected failures occur or if migration affects your business, roll back the operation. After rollback, you cannot resume the current migration task. Create a new migration task instead.
   Master node migration	The master nodes of the merged cluster are migrated from the source cluster to the destination cluster.	Master node migration starts immediately and cannot be rolled back under any circumstances. Proceed with caution.
   Primary shard migration	The system migrates primary shards—which store raw data—from source cluster nodes to destination cluster nodes.	Primary shard migration cannot be rolled back under any circumstances. Proceed with caution.

3. Switch business traffic and isolate the network.

   In this step, switch your business traffic to the destination cluster and stop the source cluster from serving traffic using network isolation.

   1. Service switchover

      On the Business cutover tab, follow the prompts to update all query and write requests in your business application from the source cluster domain name to the destination cluster domain name. Use the destination cluster for all future services.

   2. Isolate the network.

      After switching business traffic, submit a network isolation request to prevent nodes from the source cluster from joining the destination cluster. This stops the source cluster from serving traffic.

      Note

      When the interface shows Unpublish self-managed cluster nodes, follow the prompt to unpublish self-managed cluster ECS nodes or stop the Elasticsearch process. We recommend stopping the process using kill -15 <ES process PID>. After completing this action, the system continues with network isolation. This process takes about 20 minutes. Please wait.

4. Complete migration.

   After successful network isolation, the migration task ends. Click Go to instance list to view instance details.

Step 3: Validate data

After the migration task completes, use the Kibana console to access your destination Alibaba Cloud Elasticsearch cluster. Run queries or write operations to verify that the cluster meets your expectations.