Elasticsearch:Use OSS to migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster

Procedure

1. Step 1: Make preparations
   Prepare a self-managed Elasticsearch cluster, and create an OSS bucket and an Alibaba Cloud Elasticsearch cluster.
2. Step 2: Install the elasticsearch-repository-oss plug-in
   Install the elasticsearch-repository-oss plug-in on each node of the self-managed Elasticsearch cluster. You can create an OSS repository for the self-managed Elasticsearch cluster only after the plug-in is installed.
3. Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster
   Call the snapshot API to create a snapshot repository for the self-managed Elasticsearch cluster.
4. Step 4: Create a snapshot for specific indexes
   Create a snapshot for the indexes that you want to migrate and store the snapshot in the created snapshot repository.
5. Step 5: Create the same snapshot repository for the Alibaba Cloud Elasticsearch cluster
   In the Kibana console of the Alibaba Cloud Elasticsearch cluster, call the snapshot API to create a snapshot repository for the cluster. The snapshot repository must have the same name as the snapshot repository for the self-managed Elasticsearch cluster.
6. Step 6: Restore data to the Alibaba Cloud Elasticsearch cluster from the created snapshot
   Restore data from the snapshot in the snapshot repository of the self-managed Elasticsearch cluster to the Alibaba Cloud Elasticsearch cluster.
7. Step 7: View restoration results
   View the restored indexes and the data in the indexes.

Step 1: Make preparations

1. Prepare a self-managed Elasticsearch cluster.
   We recommend that you deploy an Elasticsearch cluster on Alibaba Cloud Elastic Compute Service (ECS) instances. For more information, see Installing and Running Elasticsearch.

   Note We recommend that you use Alibaba Cloud ECS instances in the same virtual private cloud (VPC) as the Alibaba Cloud Elasticsearch cluster to deploy a self-managed Elasticsearch cluster. If you use self-managed servers to deploy a cluster, a network connectivity issue may occur.

   In this example, a single-node Elasticsearch V6.7.0 cluster is used. In actual production, you can purchase multiple ECS instances that reside in the same VPC to deploy a multi-node Elasticsearch cluster. For more information about how to purchase an ECS instance, see Create an instance by using the wizard.

2. Activate OSS, and create a bucket in the region where the ECS instance that hosts the self-managed Elasticsearch cluster resides.
   For more information, see Activate OSS and Create buckets.

   Important The storage class of the bucket must be Standard. Elasticsearch does not support the Archive storage class.
3. Create an Alibaba Cloud Elasticsearch cluster in the region where the created bucket resides.
   For more information, see Create an Alibaba Cloud Elasticsearch cluster.

Step 2: Install the elasticsearch-repository-oss plug-in

1. Connect to the ECS instance that hosts the self-managed Elasticsearch cluster.
   For more information, see Connect to a Linux instance by using a password or key.

   Note In this example, a common user is used.
2. Download the installation package of the elasticsearch-repository-oss plug-in.
   In this example, the version of the plug-in is V6.7.0, which requires JDK 11.0 or later.

   wget https://github.com/aliyun/elasticsearch-repository-oss/releases/download/v6.7.0/elasticsearch-repository-oss-6.7.0.zip

   Note For more information about how to obtain the installation package of the elasticsearch-repository-oss plug-in of another version, see FAQ.
3. Decompress the installation package to the plugins folder in the installation path for the self-managed Elasticsearch cluster on the ECS instance.

   sudo unzip -d /usr/local/elasticsearch-6.7.0/plugins/elasticsearch-repository-oss elasticsearch-repository-oss-6.7.0.zip

   You can also use a command to install the plug-in.

   sudo ./bin/elasticsearch-plugin install file:///usr/local/elasticsearch-repository-oss-6.7.0.zip

4. Start the ECS instance that hosts the self-managed Elasticsearch cluster.

   sudo cd /usr/local/elasticsearch-6.7.0/bin
   sudo ./elasticsearch -d

Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster

Connect to the ECS instance that hosts the self-managed Elasticsearch cluster and run the following command to create a snapshot repository:

sudo curl -H "Content-Type: application/json" -XPUT localhost:9200/_snapshot/<yourBackupName> -d' {"type": "oss", "settings": { "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",  "access_key_id": "<yourAccesskeyId>",  "secret_access_key":"<yourAccesskeySecret>", "bucket": "<yourBucketName>", "compress": true }}'

If the repository is created, "acknowledge":true is returned.

Step 4: Create a snapshot for specific indexes

curl -H "Content-Type: application/json" -XPUT localhost:9200/_snapshot/<yourBackupName>/snapshot_1?pretty -d'
{
"indices": "index1,index2"
}'

Replace <yourBackupName> with the name of the snapshot repository that you created in Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster. Replace index1 and index2 with the names of the indexes that you want to back up. If the snapshot is created, "accepted" : true is returned.

During snapshot creation, you can run the GET /_snapshot/<yourBackupName>/<yourSnapshotName>/_status command to view the details of the snapshot. If the value of state in the response is SUCCESS, the snapshot is created.

Step 5: Create the same snapshot repository for the Alibaba Cloud Elasticsearch cluster

1. Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.
   For more information about how to log on to the Kibana console, see Log on to the Kibana console.

   Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
2. In the left-side navigation pane of the page that appears, click Dev Tools.
3. On the Console tab of the page that appears, run the following command to create a snapshot repository that has the same name as the snapshot repository for the self-managed Elasticsearch cluster.

   PUT _snapshot/<yourBackupName>
   {
       "type": "oss",
       "settings": {
           "endpoint": "oss-cn-hangzhou-internal.aliyuncs.com",
           "access_key_id": "<yourAccesskeyId>",
           "secret_access_key": "<yourAccesskeySecret>",
           "bucket": "<yourBucketName>",
           "compress": true
       }
   }

   Replace <yourBackupName> and <yourBucketName> with the repository name and bucket name that you specified in Step 3: Create a snapshot repository for the self-managed Elasticsearch cluster.

Step 6: Restore data to the Alibaba Cloud Elasticsearch cluster from the created snapshot

POST _snapshot/es_backup/snapshot_1/_restore
{"indices":"*,-.monitoring*,-.security_audit*","ignore_unavailable":"true"}

If the command is successfully run, "accepted" : true is returned.

POST _snapshot/es_backup/snapshot_1/_restore
{
  "indices":"index1",
  "rename_pattern": "index(.+)",
  "rename_replacement": "restored_index_$1"
}

Step 7: View restoration results

FAQ

Q: How do I obtain the installation package of the elasticsearch-repository-oss plug-in of another version?