Elasticsearch:Migrate self-managed Elasticsearch data to Alibaba Cloud

Self-managed Elasticsearch

• A self-managed Elasticsearch cluster is deployed on ECS. This cluster serves as the source cluster for the migration.

  Note

  The security group of the ECS instance must have an allowlist configured for specific CIDR blocks to allow access to the ECS instance over ports 9200 and 9300.

  • Port 9200: The HTTP API port for Elasticsearch, used for interactions between clients (such as browsers, applications, and Kibana) and the Elasticsearch cluster.

  • Port 9300: The TCP transport port for Elasticsearch, used for communication between nodes within the cluster.

• Replicas have been created for the indexes in the self-managed Elasticsearch cluster.

• You must have a decrypted P12 file. The P12 file, which stores public and private keys and includes a digital certificate, ensures that nodes can communicate correctly during the data migration process.

• (Optional) If your self-managed Elasticsearch does not have authentication enabled, you must enable X-Pack Security (that is, set xpack.security.enabled: true) to ensure data security during the migration process.

Alibaba Cloud Elasticsearch

You must have an Alibaba Cloud Elasticsearch cluster. This cluster serves as the target cluster for the migration. To ensure a smooth migration, verify that the target cluster meets the following requirements:

• It is deployed in Cloud-native Control Architecture (v3) mode.

• It is in the same region and VPC as the ECS instance where your self-managed Elasticsearch cluster is deployed.

• We recommend using the same version as your self-managed Elasticsearch cluster.

  Note

  If you need to perform a cross-version data migration, contact the Alibaba Cloud Elasticsearch technical support team.

• The instance specifications and node count must meet or exceed those of the self-managed Elasticsearch cluster.

• We recommend using the same username and password as the self-managed Elasticsearch cluster.

• If you have installed custom plugins on your self-managed Elasticsearch instance, such as the elasticsearch-knn  plugin for similarity searches, you must also install the same plugins on your Alibaba Cloud Elasticsearch instance. For information about plugin operations, see Plugin Configuration.

• The target cluster's features, such as cluster configuration and dictionaries, must meet your business requirements, and the cluster must be reachable over the network.

Step 1: Create a migration task

1. Navigate to the online migration page.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the left navigation bar, click Elastic Stack Cloud Native PaaS > Online Migration Tool to go to the online migration page.

2. Click Create to create a migration task and configure its parameters.

   Parameter	Description
   Type	Select Data Migration from Self-managed Elasticsearch Cluster to migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch.
   Destination Cluster	The target Alibaba Cloud Elasticsearch instance. Select an instance in the same region as the ECS instance that hosts your self-managed cluster. After selecting an instance, enter its Virtual Switch ID as prompted. Note You can select only an instance deployed in Cloud-native Control Architecture (v3) mode. If no suitable instance is available, create an Alibaba Cloud Elasticsearch instance as needed. A vSwitch is used for network isolation and access control. You can find the ID by viewing the basic information of the instance.
   VPC of Self-managed Elasticsearch Cluster	The VPC ID of your self-managed Elasticsearch cluster.
   IP Address of ECS Instance for Self-managed Elasticsearch Cluster	The IP address of the ECS instance that hosts any node of your self-managed Elasticsearch cluster. Note The security group of the ECS instance must have an allowlist configured for specific CIDR blocks to allow access to the ECS instance over ports 9200 and 9300. Port 9200: The HTTP API port for Elasticsearch, used for interactions between clients (such as browsers, applications, and Kibana) and the Elasticsearch cluster. Port 9300: The TCP transport port for Elasticsearch, used for communication between nodes within the cluster.
   .p12 File of Self-managed Elasticsearch Cluster	Follow the on-screen instructions to upload the decrypted P12 file.A P12 file stores a public key, a private key, and a digital certificate. It is used to ensure proper communication between nodes during the subsequent data migration process.
   Username of Self-managed Elasticsearch Cluster	The username used to log on to the self-managed Elasticsearch cluster.
   Password of Self-managed Elasticsearch Cluster	The password used to log on to the self-managed Elasticsearch cluster.

3. Click OK to create the task.

Step 2: Configure and run the migration task

In this step, follow the on-screen instructions to configure the migration task and complete the data migration.

1. Perform migration checks and configure integration.

   On the Configure Source and Destination Clusters tab, follow the steps below to complete the feasibility check and integration configuration for the migration task.

   1. Feasibility Check: Before starting the migration, the system automatically checks the network connectivity, version compatibility, and resource adequacy between the source and target clusters to ensure the migration is safe and viable.

      Note

      If the feasibility check fails, modify the relevant configurations as prompted.

   2. Configuration Integration: The system automatically detects the configuration parameters of the source cluster (such as the number of shard replicas, sharding policies, and user permission settings) and applies them to the target cluster. This seamless synchronization ensures a smooth migration and efficient operation.

      Note

      This step takes about 20 minutes.

2. Perform data migration.

   On the Migrate Data tab, follow the steps shown in the figure to complete cluster integration, replica migration, master node migration, and primary shard migration in sequence.

   Actions	Description	Notes
   Cluster Integration	The system merges the source cluster and the target cluster into a single large cluster. After the merge, Total nodes in the cluster = Nodes in the source cluster + Nodes in the target cluster.	After cluster integration, Kibana for the target cluster is temporarily unavailable. During the data migration, you can continue to use Kibana on the source cluster until the service switchover is complete.
   Replica Migration	The system migrates the index replica shards from the source cluster nodes to the target cluster nodes.	If an unexpected failure occurs or the migration affects your workload, you can perform a rollback. After a rollback, the current migration task is aborted and cannot be resumed. You must create a new task to proceed.
   Master Node Migration	The system migrates the master node of the merged cluster from the source cluster to the target cluster.	After you start the master node migration, the change takes effect immediately. Rollback is not supported in any scenario. Proceed with caution.
   Primary Shard Migration	The system migrates the primary shards, which store the original data, from the source cluster nodes to the target cluster nodes.	After you start the primary shard migration, rollback is not supported in any scenario. Proceed with caution.

3. Perform service switchover and network isolation.

   In this step, you will switch your current services to the target cluster and use network isolation to take the original cluster offline.

   1. Service switchover.

      On the Switch Business tab, follow the on-screen instructions to switch all query and write requests in your business applications from the source cluster's endpoint to the target cluster's endpoint. The target cluster now provides all services.

   2. Network isolation.

      After the service switchover is complete, you must initiate a network isolation request. This prevents the nodes of the source cluster from rejoining the target cluster, effectively taking the source cluster offline.

      Note

      When the Take Offline Self-managed Cluster Node message is displayed, follow the prompts to take the self-managed cluster's ECS nodes offline or shut down the ES process (we recommend that you use the kill -15 <ES_process_PID> command to take the process offline). After the operation is complete, the system continues the network isolation process. This process is estimated to take 20 minutes. Please wait patiently.

4. Complete the migration.

   After network isolation succeeds, the migration task is complete. You can click Go to Cluster List to view instance details.

Step 3: Validate the data

After the data migration task is complete, access the target Alibaba Cloud Elasticsearch cluster through the Kibana console. Perform query or write operations to verify that the cluster is functioning as expected.