Elasticsearch:migrate self-managed elasticsearch data to alibaba cloud

Self-managed Elasticsearch

• A self-managed Elasticsearch cluster is deployed on an ECS instance. This cluster serves as the source cluster for the data migration.

  Note

  You must configure the security group of the ECS instance to add a whitelist for a specific CIDR block. This allows the specified CIDR block to access the ECS instance through ports 9200 and 9300.

  • Port 9200: The HTTP API port for Elasticsearch, used by clients such as browsers, applications, and Kibana to interact with the Elasticsearch cluster.

  • Port 9300: The TCP communication port for Elasticsearch, used for communication between nodes within the cluster.

• The indexes of the self-managed Elasticsearch cluster have replicas.

• A decrypted P12 file is prepared. A P12 file stores public and private keys and contains digital certificates. During data migration, the P12 file is used to ensure normal communication between nodes.

• (Optional) If authentication is not enabled for the self-managed Elasticsearch cluster, enable X-Pack Security by setting xpack.security.enabled: true to ensure data security during the migration.

Alibaba Cloud Elasticsearch

An Alibaba Cloud Elasticsearch cluster is created. This cluster serves as the destination cluster for the data migration and receives data from the self-managed Elasticsearch cluster. To ensure a smooth migration, make sure that the Alibaba Cloud Elasticsearch cluster meets the following requirements:

• It is deployed in the cloud-native new management (v3) mode.

• It is in the same region and VPC as the ECS instance on which the self-managed Elasticsearch cluster resides.

• The version should be consistent with your self-managed Elasticsearch.

  Note

  To perform a cross-version data migration, contact the Alibaba Cloud Elasticsearch helpdesk.

• The instance type and node count are not smaller than those of the self-managed Elasticsearch cluster.

• It has the same username and password as the self-managed Elasticsearch cluster.

• If the self-managed Elasticsearch cluster has custom plugins installed, such as the elasticsearch-knn plugin for similarity search, you must install the same plugins on the Alibaba Cloud Elasticsearch cluster. For more information about plugin operations, see Plugin configuration.

• The features of the cloud cluster, such as the cluster configuration and dictionaries, meet your business requirements and are accessible over the network.

Step 1: Create a migration task

1. Navigate to the online migration tool.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the navigation pane on the left, click Elastic Stack cloud-native PaaS Edition > Elasticsearch online migration tool to access the online migration page.

2. You can click Create to create a migration task and configure the required parameters.

   

   Parameter	Description
   Type	Select Self-managed Elasticsearch Migration to migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch.
   Destination Cluster	The Alibaba Cloud Elasticsearch instance that receives data from your self-managed Elasticsearch cluster. Select an instance in the same region as the ECS instance where your self-managed Elasticsearch is deployed. After you select an instance, enter its Virtual Switch ID as prompted on the page. Note You can select only instances deployed in Cloud-native New Control Plane (v3) mode. If no suitable instance exists, create one as needed by following Create an Alibaba Cloud Elasticsearch Instance. A virtual switch provides network isolation and access control. You can find the Virtual Switch ID on the instance details page by following View Basic Instance Information.
   Self-managed ES VPC	The VPC ID of your self-managed Elasticsearch cluster.
   Self-managed ES ECS IP	The IP address of any ECS instance that hosts a node in your self-managed Elasticsearch cluster. Note You must configure the security group of the ECS instance to add a whitelist for a specific CIDR block. This allows the specified CIDR block to access the ECS instance through ports 9200 and 9300. Port 9200: The HTTP API port for Elasticsearch, used by clients such as browsers, applications, and Kibana to interact with the Elasticsearch cluster. Port 9300: The TCP communication port for Elasticsearch, used for communication between nodes within the cluster.
   Self-managed ES P12 File	Upload the decrypted P12 file as prompted on the page. A P12 file stores a public key, a private key, and a digital certificate. It ensures secure communication between nodes during data migration.
   Self-managed ES Username	The username used to log on to your self-managed Elasticsearch cluster.
   Self-managed ES Password	The password used to log on to your self-managed Elasticsearch cluster.

3. Click OK to create the job.

Step 2: Configure and execute the migration task

In this step, you configure the migration task and complete the data migration by following the instructions on the page.

1. Perform a migration check and merge configurations.

   On the Source and Destination Cluster Settings tab, you can perform a feasibility check and specify the merging configuration for the migration task as follows.

   

   1. Feasibility check: Before the migration, the system automatically checks the network connectivity, version compatibility, and resource sufficiency of the source and destination clusters to ensure that the data migration is safe and feasible.

      Note

      If the feasibility check fails, modify the configurations as prompted on the page.

   2. Configure merging: The system automatically detects the configuration parameters of the source cluster, such as the number of replica shards, sharding strategy, and user permission configurations, and adapts them to the destination cluster. This ensures that the configuration parameters, resource allocation, and sharding strategy are seamlessly synchronized between the source and destination clusters. This enables a smooth migration and efficient operation.

      Note

      This step takes about 20 minutes to complete. Please wait.

2. Migrate the data.

   On the Data Migration tab, complete cluster merging, replica migration, master node migration, and primary shard migration in sequence, as shown in the following figure.

   Operation	Description	Note
   Cluster merging	The system merges the source and destination clusters into a single cluster. After the merge, total nodes in the cluster = source cluster nodes + destination cluster nodes.	After the clusters are merged, the Kibana console of the destination cluster is temporarily shut down. During the data migration, you can continue to use the Kibana console of the source cluster until the service is switched over.
   Replica migration	The system migrates the index replica shards from the nodes of the source cluster to the nodes of the destination cluster.	If an unexpected failure occurs or your services are affected during the migration, you can perform a rollback. After the rollback, the current migration task cannot continue. You must create a new migration task.
   Master node migration	The system migrates the master node in the merged cluster from the source cluster to the destination cluster.	The master node migration starts immediately after it is initiated. Rollback is not supported. Proceed with caution.
   Primary shard migration	The system migrates the primary shards that store raw data from the nodes of the source cluster to the nodes of the destination cluster.	After the primary shard migration starts, rollback is not supported. Proceed with caution.

3. Switch services and perform network isolation.

   In this step, you switch your services to the destination cluster. At the same time, you use network isolation to stop the source cluster from providing services.

   1. Switch services.

      On the Business Switch tab, follow the on-screen instructions to switch all query and write requests for your current business from the source cluster's domain name to the destination cluster's domain name. The destination cluster will then provide the related services.

   2. Network isolation.

      After the service switchover is complete, initiate a network isolation request to prevent the nodes of the source cluster from rejoining the destination cluster. This ensures that the source cluster no longer provides services.

      Note

      When the Take Offline Self-managed Cluster Nodes message appears, follow the prompts to take the self-managed cluster ECS nodes offline or shut down the ES process. (We recommend that you shut down the ES process by running the kill -15 <ES process PID> command.) After the operation is complete, the system proceeds with the network isolation. This process takes approximately 20 minutes.

4. Complete the migration.

   After network isolation is successful, the migration task is complete. Click Go to Instance List to view the instance details.

Step 3: Data validation

After the data migration task is complete, access the destination Alibaba Cloud Elasticsearch cluster from the Kibana console. Then, perform query or write operations to verify that the cluster works as expected.