Elasticsearch data migration solutions - Elasticsearch - Alibaba Cloud Documentation Center

You can use various methods, such as Logstash, reindex, Object Storage Service (OSS), and the online migration tool, to migrate data. Scenarios include migrating data between Alibaba Cloud Elasticsearch clusters, from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster, or from a third-party Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster. This topic describes the available migration solutions, their use cases, and their limitations to help you select the right solution for your needs.

Important

In October 2020, Alibaba Cloud Elasticsearch updated its network architecture. Clusters created before October 2020 use the original network architecture, whereas clusters created in or after October 2020 use the new network architecture. Operations like reindex, cross-cluster search, and cross-cluster replication are not supported between clusters with different network architectures. To perform these operations, ensure that both clusters use the same network architecture. For clusters in the China (Zhangjiakou) region and regions outside the Chinese mainland, the timing of the network architecture adjustment is uncertain. To verify network connectivity in these regions, submit a ticket to Alibaba Cloud Elasticsearch technical support. Log on to the Alibaba Cloud Elasticsearch console and go to the Basic Information page of the cluster. In the Deployment Mode section, check the network architecture. Cloud Native Control (v3) indicates the new network architecture, and Basic Control (v2) indicates the original network architecture.
You should not migrate system indices that start with a ., such as .monitoring, .kibana, or .security. Otherwise, Kibana may malfunction.

Scenario: Migrate data between Alibaba Cloud Elasticsearch clusters

Migration solution	Use cases	Limitations	Examples
OSS snapshot	Large-scale data migration (GB, TB, or PB). Migrate snapshot data between clusters under the same or different Alibaba Cloud accounts, in the same or different regions. Note For cross-region data migration, you can use Elasticsearch snapshot and restore commands together with the cross-region replication feature of OSS.	Migrating incremental data by using OSS snapshots may require you to stop the service, and you must also disable the index when restoring the incremental data on the new cluster. When configuring a shared OSS repository, the source and destination clusters must meet these requirements: Located in the same region. Belong to the same Alibaba Cloud account or RAM user. The source cluster version must be the same as or earlier than the destination cluster version. For more information, see Version compatibility. For cross-account API migration, you must use an AccessKey pair from the account that owns the OSS bucket to add the repository to both the source and destination clusters.	Back up and restore data by using a shared OSS repository (for automatic snapshots) Back up and restore data manually (for manual snapshots)
Logstash	Full migration or incremental synchronization when high real-time performance is not required. Important For cross-account or cross-region migration, the Elasticsearch and Logstash clusters may not be in the same VPC. If so, you must configure a NAT gateway for Logstash to enable data transfer over the internet. Migrate only query results. Filter data before migration. Migrate data across major versions, such as from 5.x to 6.x or 7.x. For more information, see Product compatibility.	The source cluster, Logstash cluster, and destination cluster must be in the same VPC. Otherwise, you must configure a NAT gateway for public network access. For more information, see Configure a NAT gateway for public data transfer. The versions of the source Elasticsearch cluster, Logstash cluster, and destination Elasticsearch cluster must be compatible. For more information, see Product compatibility. If you need to synchronize incremental data, you must ensure that the source and destination data IDs match, and you must also configure a scheduled task.	Synchronize data between Alibaba Cloud Elasticsearch clusters Use reindex to migrate data from multiple types in an earlier version
reindex	Small datasets where migration speed is not critical. Migrate only query results that are obtained by running query statements in Kibana.	Both Elasticsearch clusters must use the same network architecture. For more information, see Notes.	Use cross-cluster reindex between Alibaba Cloud ES clusters
elasticsearch-dump tool	Small datasets with a small number of indexes.	Network connectivity is required among the source Elasticsearch cluster, the target Elasticsearch cluster, and the server where the elasticsearch-dump tool is installed.	Use the elasticsearch-dump tool to migrate data
OSS snapshot and reindex	Full and incremental data migration. Use when a direct snapshot restore is not possible due to major version incompatibility. For example, to migrate data from 6.x to 8.x, you can first restore a snapshot to a 7.x cluster, reindex the data in the 7.x cluster, and then restore a snapshot from the 7.x cluster to the 8.x cluster.	The intermediate version must be compatible with both the source and destination versions. For more information, see Elastic Stack upgrade and migration guide.	Back up and restore data manually (for manual snapshots) Use cross-cluster reindex between Alibaba Cloud ES clusters
online migration tool	For a smooth, high-availability, zero-downtime upgrade from the V2 to V3 cluster architecture. Data is migrated automatically, and the cluster endpoint remains unchanged.		Upgrade cluster architecture

Scenario: Migrate self-managed Elasticsearch to Alibaba Cloud

Migration solution	Use cases	Limitations	Examples
online migration tool	Migrate data from a self-managed Elasticsearch cluster, whether deployed on-premises or on ECS instances. For a smooth, high-availability, zero-downtime data migration from a self-managed Elasticsearch cluster.	Cross-region and cross-VPC data migration are not supported. Migration across major versions, such as from 7.x to 8.x, is not supported. The source indexes must have replicas.	Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud
OSS snapshot	Large-scale data migration (GB, TB, or PB). Migrate snapshots between clusters under the same or different accounts, in the same or different regions.	You must install the elasticsearch-repository-oss plug-in that is compatible with your source Elasticsearch version. Migrating incremental data via OSS snapshots may require you to stop the service. Additionally, you need to close the indexes when you restore the incremental data on the new cluster.	Advanced: Use OSS to migrate a self-managed ES cluster to Alibaba Cloud ES
Logstash	Data migration where high real-time performance is not required. Migrate only query results. Filter data before migration. Migrate data across major versions, such as from 5.x to 6.x or 7.x. For more information, see Product compatibility.	The source cluster, Logstash cluster, and destination cluster must be in the same VPC. Otherwise, you must configure a NAT gateway for public network access. For more information, see Configure a NAT gateway for public data transfer. The versions of the source Elasticsearch cluster, Logstash cluster, and destination Elasticsearch cluster must be compatible. For more information, see Product compatibility. If you need to synchronize incremental data, ensure that the source and target data IDs are identical, and configure a scheduled task.	Use Alibaba Cloud Logstash to migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Use reindex to migrate data from multiple types in an earlier version
reindex	Small datasets where migration speed is not critical. Migrate only query results that are obtained by running query statements in Kibana. Migrate between major versions (e.g., from self-managed 6.x to Alibaba Cloud 8.x). In this case, use PrivateLink to establish network connectivity for remote reindexing.	Both Elasticsearch clusters must use the same network architecture. For more information, see Notes.	Use reindex to migrate data from a self-managed ES cluster to Alibaba Cloud
elasticsearch-dump tool	Small datasets with a small number of indexes.	The source Elasticsearch cluster, the destination Elasticsearch cluster, and the server where the `elasticsearch-dump` tool is installed must have network connectivity with each other.	Use the elasticsearch-dump tool to migrate data

Scenario: Migrate third-party Elasticsearch to Alibaba Cloud

Migration solution	Use cases	Limitations	Examples
OSS snapshot	Large-scale data migration (GB, TB, or PB).	Migrating incremental data by using OSS snapshots may require you to stop the service. Additionally, you must disable the index when you restore the incremental data on the new cluster.	Migrate indexes from Amazon OpenSearch Service to Alibaba Cloud ES Note This solution relies on the standard Elasticsearch snapshot mechanism and is not limited to a specific cloud provider. For example, you can use the Alibaba Cloud online migration tool to transfer Elasticsearch snapshot data from Tencent Cloud Object Storage (COS) to Alibaba Cloud OSS, and then restore the data from the OSS repository to your Alibaba Cloud Elasticsearch cluster. For instructions, see Back up and restore data manually. You can only restore a snapshot to a cluster of the same or a later version. For more information, see Snapshot and restore compatibility.
elasticsearch-dump tool	Small datasets with a small number of indexes.	The source Elasticsearch cluster, the target Elasticsearch cluster, and the server where the elasticsearch-dump tool is installed must be able to communicate with each other over the network.	Use the elasticsearch-dump tool to migrate data