You can use Logstash, the reindex API, or Object Storage Service (OSS) to migrate data between Alibaba Cloud Elasticsearch clusters, from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster, or from a third-party Elasticsearch source to an Alibaba Cloud Elasticsearch cluster. This topic describes the use scenarios and limits of each data migration solution. You can select a solution based on your business requirements.
The network architecture of Alibaba Cloud Elasticsearch was adjusted in October 2020. Elasticsearch clusters created before October 2020 are deployed in the original network architecture. Elasticsearch clusters created in October 2020 or later are deployed in the new network architecture. You are not allowed to perform cross-cluster operations, such as reindex, searches, or replication, between a cluster deployed in the original network architecture and a cluster deployed in the new network architecture. If you want to perform these operations between two clusters, you must make sure that the clusters are deployed in the same network architecture. The time when the network architecture in the China (Zhangjiakou) region and the regions outside China was adjusted is uncertain. If you want to perform the preceding operations between a cluster created before October 2020 and a cluster created in October 2020 or later in such a region, submit a ticket to contact Alibaba Cloud technical support to check whether the clusters can be connected.
We recommend that you do not migrate system indexes whose names start with a period (
.), such as .monitoring, .kibana, and .security indexes. If you migrate these indexes, Kibana may fail.
Data migration between Alibaba Cloud Elasticsearch clusters
Migration solution | Use scenario | Usage note | Example |
OSS snapshots |
|
|
|
Logstash |
|
| |
reindex API |
| The source and destination Elasticsearch clusters must be deployed in the same network architecture. For more information, see Use the reindex API to migrate data in a multi-type index of an earlier version. | |
elasticsearch-dump | You want to migrate small volumes of data in scenarios with a small number of indexes. | Network connections must be established among the source Elasticsearch object, destination Elasticsearch cluster, and the server on which the elasticsearch-dump tool is installed. | |
OSS snapshots and reindex API |
| The cluster of an intermediate version is compatible with both the source cluster and destination cluster. For more information, see Elastic version changes and compatibility. |
|
Data migration from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster
Migration solution | Use scenario | Usage note | Example |
OSS snapshots |
|
| |
Logstash |
|
| |
reindex API |
| The source and destination Elasticsearch clusters must be deployed in the same network architecture. For more information, see Use the reindex API to migrate data in a multi-type index of an earlier version. | |
elasticsearch-dump | You want to migrate small volumes of data in scenarios with a small number of indexes. | Network connections must be established among the source Elasticsearch object, destination Elasticsearch cluster, and the server on which the elasticsearch-dump tool is installed. |
Data migration from a third-party Elasticsearch source to an Alibaba Cloud Elasticsearch cluster
Migration solution | Use scenario | Usage note | Example |
OSS snapshots | The source stores gigabytes, terabytes, or petabytes of data. | If you want to use this solution to migrate incremental data, you may need to stop your Elasticsearch service and must disable the destination index before data migration. | Migrate Elasticsearch index data from Amazon OpenSearch Service to Alibaba Cloud Elasticsearch Note This solution is not limited to cloud service providers but depends on the Elasticsearch snapshot mechanism. For example, you can use the Data Online Migration service provided by Alibaba Cloud to migrate Elasticsearch data in snapshots from a Tencent Cloud Object Storage (COS) bucket to an Alibaba Cloud OSS bucket. Then, restore the data to the destination cluster. For information about how to migrate data from a Tencent COS bucket to an Alibaba Cloud OSS bucket, see Migrate data. For information about how to restore data in snapshots, see Create manual snapshots and restore data from manual snapshots. The version of the destination Elasticsearch cluster must be the same as or later than the source Elasticsearch object. For information about version compatibility, see Version compatibility of data restoration from snapshots. |
elasticsearch-dump | You want to migrate small volumes of data in scenarios with a small number of indexes. | Network connections must be established among the source Elasticsearch object, destination Elasticsearch cluster, and the server on which the elasticsearch-dump tool is installed. |