Elasticsearch migration method selection - Elasticsearch

Alibaba Cloud Elasticsearch supports multiple data migration methods, including Logstash, reindex, Object Storage Service (OSS), and Data Online Migration. You can migrate data between Alibaba Cloud Elasticsearch instances, migrate data from a self-managed Elasticsearch cluster, or migrate data from a third-party Elasticsearch service. This topic compares each method's applicable scenarios and limitations to help you select the best option.

Important

In October 2020, Alibaba Cloud Elasticsearch upgraded its network architecture. Instances created before this date use the legacy network architecture, while instances created in or after October 2020 use the new network architecture. Cross-cluster operations, such as remote reindex, cross-cluster search, and cross-cluster replication, are not supported between instances on the new and legacy network architectures. To perform these operations, ensure that both instances use the same network architecture. For instances in the China (Zhangjiakou) and overseas regions, the exact time of the network architecture upgrade varies. To verify network connectivity for cross-cluster operations, submit a ticket to contact Alibaba Cloud Elasticsearch technical support. To check the network architecture of an instance, log on to the Alibaba Cloud Elasticsearch console and find the Basic Information section on the Control Architecture Type page. Cloud-native Control Architecture (v3) indicates the new network architecture, and Basic Control Architecture (v2) indicates the legacy network architecture.
Do not migrate system indexes. These indexes are prefixed with a period (.), such as .monitoring, .kibana, or .security. Migrating these indexes may cause Kibana to fail.
Data Transmission Service (DTS) does not support data synchronization between two Elasticsearch instances.
The migration duration depends on the volume of data to be migrated, the number of indexes, and the load on the source and target clusters. An exact duration cannot be estimated in advance. For the same volume of data, OSS snapshots are typically faster than the other migration methods. Before you perform the actual migration, run a trial migration with a small number of indexes to estimate the overall migration window.

Migration between Alibaba Cloud Elasticsearch instances

Migration method	Applicable scenarios	Limitations	Example
OSS snapshot	Migrating large datasets (GB, TB, or PB scale). Migrating snapshot data between instances in the same or different Alibaba Cloud accounts, within the same region or across regions. Note For cross-region data migration, you can use Elasticsearch snapshot and restore commands together with the cross-region replication (CRR) feature of OSS.	Migrating full data by using an OSS snapshot does not require service downtime. Migrating incremental data may require you to stop write operations on the source instance. When restoring incremental data on the new cluster, you must close the index. To use the cross-cluster OSS repository feature of Alibaba Cloud Elasticsearch, the source and target instances must meet the following requirements: Located in the same region. Belong to the same Alibaba Cloud account or RAM user. The version of the source instance must be earlier than or equal to the version of the target instance. For more information about version compatibility, see Compatibility matrixes. When using an API to migrate snapshot data across accounts, you must use the same AccessKey for both the source and target. Specifically, when adding the repository, use the AccessKey of the account that owns the Object Storage Service (OSS) bucket. Whether the OSS snapshot method is feasible depends mainly on the region of the OSS bucket rather than on the account that owns the instances. The preceding same-region and same-account requirements apply only to the cross-cluster OSS repository feature in the console. For cross-account scenarios, you can add the repository by using an API and the AccessKey of the account that owns the bucket. If the instances reside in the same region and the data volume is large, use OSS snapshots. For cross-region scenarios, first use the cross-region replication (CRR) feature of OSS to replicate the snapshot to a bucket in the destination region, and then restore the data on the target instance. For example, you can take a snapshot of an instance in the China (Shanghai) region to a bucket in the China (Shanghai) region, replicate the snapshot to a bucket in the China (Beijing) region by using CRR, and then restore the data to an instance in the China (Beijing) region.	Back up and restore data by using a cross-cluster OSS repository (for automatic snapshots) Create manual snapshots and restore data from manual snapshots (for manual snapshots)
Logstash	Migrating full data or syncing incremental data when real-time performance is not critical. Important For cross-account and cross-region migrations where the Elasticsearch and Logstash instances are not in the same VPC, you must configure a NAT gateway for the Logstash instance to transmit data over the public network. Migrating only query results. Filtering data before migration. Migrating data across major versions, such as from 5.x to 6.x or 7.x. For more information about version compatibility, see Compatibility matrixes.	The source Elasticsearch instance, Logstash instance, and target Elasticsearch instance must be in the same VPC. If not, you must configure a NAT gateway to enable public network connectivity. For more information, see Configure a NAT gateway for data transmission over the Internet. The versions of the source Elasticsearch, Logstash, and target Elasticsearch instances must be compatible. For more information, see Compatibility matrixes. To sync incremental data, ensure that the data IDs are consistent between the source and target, and configure a scheduled task.	Migrate data between Alibaba Cloud Elasticsearch clusters Use the reindex API to migrate data in a multi-type index of an earlier version
reindex	Migrating small datasets where migration speed is not a high priority. Migrating only query results (results from a query statement run in Kibana).	The two Elasticsearch clusters must use the same network architecture. For more information, see the notes at the beginning of this topic.	Use the reindex API to migrate data
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data
OSS snapshot and reindex	Migrating both full and incremental data. Migrating across major versions where the file format may be incompatible and direct snapshot restore is not supported. For example, to migrate from 6.x to 8.x, you can first restore a snapshot to a 7.x instance, reindex the data on the 7.x instance, and then restore a snapshot from the 7.x instance to the 8.x instance.	The intermediate version must be compatible with both the source and target versions. For more information, see Elastic version changes and compatibility.	Create manual snapshots and restore data from manual snapshots (for manual snapshots) Use the reindex API to migrate data
Data Online Migration	Zero-downtime upgrades from Alibaba Cloud Elasticsearch V2 to V3, with data migration and endpoint preservation.		Upgrade instance architecture

Migration from self-managed to Alibaba Cloud Elasticsearch

Migration method	Applicable scenarios	Limitations	Example
Data Online Migration	Migrating data from a self-managed Elasticsearch cluster deployed on ECS instances within an IDC. Smooth, highly available, zero-downtime migrations from a self-managed Elasticsearch cluster.	Cross-region and cross-VPC data migration are not supported. Cross-major-version migration is not supported (for example, from 7.x to 8.x). Indexes in the self-managed Elasticsearch cluster must have at least one replica.	Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch
OSS snapshot	Migrating large datasets (GB, TB, or PB scale). Migrating snapshot data in the same or different Alibaba Cloud accounts, within the same region or across regions.	You must obtain the elasticsearch-repository-oss plug-in that matches the source Elasticsearch version. Migrating incremental data by using an OSS snapshot may require service downtime. When restoring incremental data on the new cluster, you must close the index.	Use OSS to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
Logstash	Data migrations where real-time performance is not critical. Migrating only query results. Filtering data before migration. Migrating data across major versions, such as from 5.x to 6.x or 7.x. For more information about version compatibility, see Compatibility matrixes.	The source Elasticsearch instance, Logstash instance, and target Elasticsearch instance must be in the same VPC. If not, you must configure a NAT gateway to enable public network connectivity. For more information, see Configure a NAT gateway for data transmission over the Internet. The versions of the source Elasticsearch, Logstash, and target Elasticsearch instances must be compatible. For more information, see Compatibility matrixes. To sync incremental data, ensure that the data IDs are consistent between the source and target, and configure a scheduled task.	Use Alibaba Cloud Logstash to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster Use the reindex API to migrate data in a multi-type index of an earlier version
reindex	Migrating small datasets where migration speed is not a high priority. Migrating only query results (results from a query statement run in Kibana). Migrating from an earlier-version self-managed Elasticsearch cluster to a later-version Alibaba Cloud Elasticsearch cluster, such as from 6.x to 8.x. This requires using PrivateLink to establish network connectivity for remote reindexing.	The two Elasticsearch clusters must use the same network architecture. For more information, see the notes at the beginning of this topic.	Use the reindex API to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, the target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data

Migration from third-party to Alibaba Cloud Elasticsearch

Migration method	Applicable scenarios	Limitations	Example
OSS snapshot	Migrating large datasets (GB, TB, or PB scale).	Migrating incremental data by using an OSS snapshot may require service downtime. When restoring incremental data on the new cluster, you must close the index.	Migrate Elasticsearch index data from Amazon OpenSearch Service to Alibaba Cloud Elasticsearch Note This method is not limited to specific cloud vendors and relies on the Elasticsearch snapshot mechanism. For example, you can use Alibaba Cloud Data Online Migration (Migrate data) to migrate Elasticsearch snapshot data from Tencent Cloud COS to Alibaba Cloud OSS, and then restore the data from the OSS repository. For more information, see Create manual snapshots and restore data from manual snapshots. Elasticsearch snapshot migration supports restoring data only to the same or a later version. For more information, see Version compatibility of data restoration from snapshots.
elasticsearch-dump tool	Migrating small datasets with a limited number of indexes.	The source cluster, the target cluster, and the server where the elasticsearch-dump tool is installed must have network connectivity.	Use elasticsearch-dump to migrate data