You can use the snapshots that are stored in Object Storage Service (OSS) to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster. To migrate data, call the snapshot API to create a snapshot for the self-managed Elasticsearch cluster and store the snapshot in OSS. Then, restore data from the snapshot to your Alibaba Cloud Elasticsearch cluster. This topic describes the procedure in detail.
Background information
Procedure
Step 1: Make preparations
Step 2: Install the elasticsearch-repository-oss plug-in
Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster
Connect to the ECS instance that hosts the self-managed Elasticsearch cluster and run the following command to create a snapshot repository:
curl -H "Content-Type: application/json" -XPUT localhost:9200/_snapshot/<yourBackupName> -d' {"type": "oss", "settings": { "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com", "access_key_id": "<yourAccesskeyId>", "secret_access_key":"<yourAccesskeySecret>", "bucket": "<yourBucketName>", "compress": true }}'
Parameter | Description |
---|---|
<yourBackupName> | The name of the repository, which can be customized. |
type | The type of the repository. Set this parameter to oss. |
endpoint | The endpoint of your OSS bucket. For more information, see Regions and endpoints.
Note If the ECS instance that hosts the self-managed Elasticsearch cluster resides in the
same region as your OSS bucket, use the internal endpoint of the OSS bucket. Otherwise,
use the public endpoint of the OSS bucket.
|
access_key_id | The AccessKey ID of the Alibaba Cloud account that is used to create the OSS bucket. For more information about how to obtain the AccessKey ID, see How can I obtain an AccessKey pair? |
secret_access_key | The AccessKey secret of the Alibaba Cloud account that is used to create the OSS bucket. For more information about how to obtain the AccessKey secret, see How can I obtain an AccessKey pair? |
bucket | The name of the OSS bucket. |
compress | Specifies whether to enable compression. Valid values:
|
If the repository is created, "acknowledge":true
is returned.
Step 4: Create a snapshot for specific indexes
curl -H "Content-Type: application/json" -XPUT localhost:9200/_snapshot/<yourBackupName>/snapshot_1?pretty -d'
{
"indices": "index1,index2"
}'
Replace <yourBackupName> with the name of the snapshot repository that you created in Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster. Replace index1 and index2 with the names of the indexes that you want to back up. If the snapshot is created,
"accepted" : true
is returned.
During snapshot creation, you can run the GET /_snapshot/<yourBackupName>/<yourSnapshotName>/_status
command to view the details of the snapshot. If the value of state in the response is SUCCESS, the snapshot is created.
Step 5: Create the same snapshot repository for the Alibaba Cloud Elasticsearch cluster
Step 6: Restore data to the Alibaba Cloud Elasticsearch cluster from the created snapshot
.
) from the created snapshot. Follow the instructions in Step 5: Create the same snapshot repository for the Alibaba Cloud Elasticsearch cluster to perform the operation. POST _snapshot/es_backup/snapshot_1/_restore
{"indices":"*,-.monitoring*,-.security_audit*","ignore_unavailable":"true"}
If the command is successfully run, "accepted" : true
is returned.
POST _snapshot/es_backup/snapshot_1/_restore
{
"indices":"index1",
"rename_pattern": "index(.+)",
"rename_replacement": "restored_index_$1"
}
Step 7: View restoration results
- View the restored indexes
GET /_cat/indices?v
- View the data in the restored indexes
GET /index1/_search
If the command is successfully run, the following result is returned:{ "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 1.0, "hits" : [ { "_index" : "index1", "_type" : "_doc", "_id" : "1", "_score" : 1.0, "_source" : { "productName" : "testpro", "annual_rate" : "3.22%", "describe" : "testpro" } } ] } }
FAQ
Q: How do I obtain the installation package of the elasticsearch-repository-oss plug-in of another version?
- version=Required plug-in version
- elasticsearch.version=Version of the self-managed Elasticsearch cluster
Note The version of the plug-in must be the same as that of the self-managed Elasticsearch cluster.
- java.version=1.8
Note
- Different versions of Elasticsearch clusters depend on different versions of JDKs. The actual JDK version is determined by open source Elasticsearch and the plug-in.
- Open source Elasticsearch provides a variety of cluster versions, and different versions of clusters are compiled in different ways. Therefore, before you install the elasticsearch-repository-oss plug-in for your cluster, you need to compile and debug the plug-in based on the cluster version. For example, you deploy a self-managed Elasticsearch V7.6.2 cluster, and the required JDK version is 1.8 or later. After compilation and debugging, the plug-in is elasticsearch-repository-oss-7.6.2.