Elasticsearch:Migrate Amazon OpenSearch indexes to Alibaba Cloud Elasticsearch

How it works

This cloud-agnostic solution uses the Elasticsearch snapshot mechanism. Use the Alibaba Cloud Migration Implementation tool to migrate indexes from AWS ES or snapshot data from Tencent Cloud COS to Alibaba Cloud OSS, then restore from the OSS repository. Manual Backup and Restoration.

Snapshots can only be restored to clusters of the same or later version. Snapshot compatibility.

Background

Concepts

• Elasticsearch: a distributed, RESTful search and analytics engine. As the core of the Elastic Stack, it centrally stores data for searching and analysis.

• Kibana: a visualization tool for searching and analyzing Elasticsearch data.

• Amazon OpenSearch Service: a managed AWS service that provides Elasticsearch APIs and real-time analytics.

• Alibaba Cloud Elasticsearch: a managed service based on open source Elasticsearch, designed for data analysis and search scenarios. Provides enterprise-grade access control, security monitoring and alerting, and automated reporting.

• Snapshot and restore: creates snapshots of individual indexes or an entire cluster in a remote repository (shared file system, Amazon S3, or HDFS) for restoration to a compatible Elasticsearch version.

  • A snapshot of an index created in version 5.x can be restored to version 6.x.

  • A snapshot of an index created in version 2.x can be restored to version 5.x.

  • A snapshot of an index created in version 1.x can be restored to version 2.x.

Solution overview

Prerequisites

Ensure the following:

• An Amazon OpenSearch Service domain of Elasticsearch 7.1.0.0 is created in the Singapore region.

  Create an Amazon OpenSearch Service domain.

• An Alibaba Cloud Elasticsearch cluster of version 7.1.0.0 is created in the China (Hangzhou) region.

  Create an Alibaba Cloud Elasticsearch cluster.

• An OSS bucket is created.

  This guide uses a bucket in the China (Hangzhou) region with Standard storage class and private ACL. All other parameters use default settings. Create buckets.

• Have the index that you want to migrate ready. This topic uses an index named movies as an example.

Prerequisites for manual snapshots on AWS

Amazon OpenSearch Service automatically creates daily snapshots of primary index shards, stored in a preconfigured S3 bucket for 14 days at no extra charge. These snapshots can restore a domain but cannot migrate data to a new domain. For migration, create manual snapshots in your own S3 bucket. Standard S3 storage fees apply.

To manually create and restore index snapshots, you need IAM and Amazon S3. Ensure the following prerequisites are met before taking a snapshot.

• Create an Amazon S3 bucket

  Create an S3 bucket to store manual snapshots. Note its ARN for use in:

  • The Resource statement of the IAM policy attached to the IAM role.

  • The Python client used to register the snapshot repository.

  Example S3 bucket ARN:

  arn:aws:s3:::eric-es-index-backups

• Create an IAM role

  Ensure the IAM role specifies Amazon OpenSearch Service (es.amazonaws.com) in the Service statement of its trust relationship:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
          "Service": "es.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }

  View the trust relationship details in the IAM console.

  

  Note

  Amazon OpenSearch Service is not available in the Select role type drop-down list when creating a role for an AWS service. Select AWS EC2, create the role, then edit the trust relationship to change ec2.amazonaws.com to es.amazonaws.com.

• Create an IAM policy

  Create an IAM policy and attach it to the IAM role. The following example specifies the ARN of the eric-es-index-backups bucket:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "s3:ListBucket"
              ],
              "Effect": "Allow",
              "Resource": [
                  "arn:aws:s3:::eric-es-index-backups"
              ]
          },
          {
              "Action": [
                  "s3:GetObject",
                  "s3:PutObject",
                  "s3:DeleteObject"
              ],
              "Effect": "Allow",
              "Resource": [
                  "arn:aws:s3:::eric-es-index-backups/*"
              ]
          }
      ]
  }

  1. Copy the policy content into the policy editor.

  2. Review the policy.

  3. Attach the IAM policy to the IAM role.

Step 1: Register a manual snapshot repository

Register a snapshot repository with Amazon OpenSearch Service before creating manual snapshots. Sign the registration request using credentials for the user or role specified in the IAM role's trust relationship. Prerequisites for manual snapshots on AWS.

1. Download the register_snapshot_repository.py file.

2. Modify the sample Python client file.

   Update the highlighted values to match your configuration, then save the file as snapshot.py.

   Parameters in the sample Python client file:

   Parameter	Description
   region	The AWS region where the snapshot repository is created.
   host	The endpoint of the Amazon OpenSearch Service domain.
   aws_access_key_id	The access key ID of your IAM credential.
   aws_secret_access_key	The secret access key of your IAM credential.
   path	The path to the snapshot repository.
   data	Must include the name and ARN of the S3 bucket created for the IAM role. Prerequisites for manual snapshots on AWS. Important To enable server-side encryption with S3-managed keys for the snapshot repository, add "server_side_encryption": true to the settings JSON. If the Amazon S3 bucket is in the ap-southeast-1 region, use "endpoint": "s3.amazonaws.com" instead of "region": "ap-southeast-1".

3. Install boto-2.48.0.

   The sample Python client requires boto 2.x:

   # wget https://pypi.python.org/packages/66/e7/fe1db6a5ed53831b53b8a6695a8f134a58833cadb5f2740802bc3730ac15/boto-2.48.0.tar.gz#md5=ce4589dd9c1d7f5d347363223ae1b970
   # tar zxvf boto-2.48.0.tar.gz
   # cd boto-2.48.0
   # sudo python setup.py install

4. Run the Python client to register the snapshot repository.

   # python snapshot.py

5. In the Kibana console for your Amazon OpenSearch Service domain, go to Dev Tools > Console and run:

   GET _snapshot

   

Step 2: Create and restore the initial snapshot

1. Manually create a snapshot on Amazon OpenSearch Service.

   Note

   Run the following commands in the Kibana console or by using curl from a Linux or macOS command line.

   • Create a snapshot named snapshot_movies_1 for the movies index in the eric-snapshot-repository repository.

     PUT _snapshot/eric-snapshot-repository/snapshot_movies_1
     {
     "indices": "movies"
     }

   • Check the snapshot status.

     GET _snapshot/eric-snapshot-repository/snapshot_movies_1

     

   • In the Amazon S3 console, view the snapshot files.

2. Transfer the snapshot data from Amazon S3 to Alibaba Cloud OSS.

   Migrate applications from Amazon S3 to OSS.

   After transfer, verify the snapshot data in the OSS console.

   

3. Restore the snapshot to the Alibaba Cloud Elasticsearch cluster.

   1. Create a snapshot repository.

      Important

      During an Elasticsearch snapshot migration, you must ensure that each Elasticsearch cluster uses a dedicated snapshot repository or path. If multiple clusters share the same repository, data restoration may fail and data may be lost.

      In the Kibana console of the destination Alibaba Cloud Elasticsearch cluster (Log on to the Kibana console), go to Dev Tools > Console and create a snapshot repository with the same name:

      PUT _snapshot/eric-snapshot-repository
      {
      "type": "oss",
      "settings": {
                  "base_path": "my/snapshot/directory"
                  "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com", 
                  "access_key_id": "your AccessKeyID",
                  "secret_access_key": "your AccessKeySecret ",
                  "bucket": "eric-oss-aws-es-snapshot-s3",
                  "compress": true
            }
      }

   2. Check the status of the snapshot named snapshot_movies_1.

      GET _snapshot/eric-snapshot-repository/snapshot_movies_1

      

      Note

      Note the start and end times for incremental snapshot migration with ossimport:

      • "start_time_in_millis": 1519786844591

      • "end_time_in_millis": 1519786846236

4. Restore the snapshot.

   POST _snapshot/eric-snapshot-repository/snapshot_movies_1/_restore
   {
       "indices": "movies"
   }

   Check the availability of the movies index:

   GET movies/_recovery

   After successful restoration, the movies index contains three documents, matching the Amazon OpenSearch Service domain.

Step 3: Create and restore the final snapshot

1. Insert data into the movies index in the Amazon OpenSearch Service domain.

   The movies index already has three documents. Insert two more.

   

   Use the GET movies/_count command to check the number of documents in the index.

2. Manually create another snapshot.

   Create another snapshot. See Manually create a snapshot on Amazon OpenSearch Service.

   PUT _snapshot/eric-snapshot-repository/snapshot_movies_2
   {
   "indices": "movies"
   }

   Check the snapshot status:

   GET _snapshot/eric-snapshot-repository/snapshot_movies_2

   View the files that are listed in the Amazon S3 bucket.

   

3. Transfer the incremental snapshot data from Amazon S3 to Alibaba Cloud OSS.

   Use ossimport to migrate data from S3 to OSS. To migrate only new files, set isSkipExistFile to true in local_job.cfg.

   When isSkipExistFile is true, objects are skipped if their size and LastModifiedTime match. When false (default), existing objects are overwritten. This option is invalid when jobType is audit.

   After migration, verify the new files in OSS.

   

4. Restore the incremental snapshot.

   Follow the snapshot restoration procedure in Step 2: Create and restore the first snapshot. Close the movies index before restoration, then reopen the movies index afterward.

   • Close the movies index.

     POST /movies/_close

   • Check the status of the movies index.

     GET movies/_stats

   • Restore the incremental snapshot.

     POST _snapshot/eric-snapshot-repository/snapshot_movies_2/_restore
     {
         "indices": "movies"
     }

   • Open the movies index.

     POST /movies/_open

   After restoration, the movies index contains 5 documents, matching the Amazon OpenSearch Service domain.

   

Summary

Migrate index data from Amazon OpenSearch Service to Alibaba Cloud Elasticsearch by using snapshots. Close indexes on the destination cluster before restoration to prevent writes and requests during migration.

Related resources:

• Elasticsearch documentation

• Snapshot module

• Working with Index Snapshots in Amazon Elasticsearch Service

• Seamlessly switch applications from Amazon S3 to OSS

• ossimport description and configurations

FAQ

Why is repository uniqueness required?

If multiple Elasticsearch clusters share the same snapshot repository, the following issues can occur:

1. Metadata conflicts: Multiple clusters operating on the same repository can overwrite or corrupt metadata files, preventing snapshots from being correctly identified.

2. Data overwrites: Snapshots from different clusters may overwrite each other's data files, causing data loss or corruption.

3. Restoration failures: Elasticsearch cannot determine which data belongs to which cluster, causing restore operations to fail or restore incorrect data.

Why do OSS snapshot restores fail?

This error may occur if a snapshot object name in OSS contains a special character such as a forward slash (/).

Alibaba Cloud Elasticsearch does not support object names with special characters during OSS snapshot restoration. Use ossbrowser to move the snapshot data to a folder with a standard name.