Elasticsearch:Migrate Amazon OpenSearch indexes to Alibaba Cloud Elasticsearch

Prepare the source AWS resources

• Create an AWS domain. Amazon OpenSearch Service has retired early Elasticsearch versions such as 7.1.0.0. Use one of the versions that are currently available in the console. This topic uses Elasticsearch 7.10.2 as an example. You can use any region; this topic uses us-east-1. For more information, see Creating and managing Amazon OpenSearch Service domains.

  Note

  If fine-grained access control is enabled for the domain and the master user type is IAM, or if the access policy of the domain allows only IAM principals (for example, es:ESHttp* is granted only to the account root user), you cannot log on to the Kibana console of the domain from a browser. The request returns the 403 error User: anonymous is not authorized to perform: es:ESHttpGet. In this case, run all AWS-side commands in this topic as requests that are signed with AWS Signature Version 4 (SigV4). For example, reuse the Python client described in Step 2.

• Create an AWS S3 bucket to store the manual snapshots, and record the Amazon Resource Name (ARN) of the bucket. The ARN is used in two places: the Resource statement of the IAM policy, and the Python client that registers the snapshot repository.

  The following example shows an ARN.

  arn:aws:s3:::eric-es-index-backups

• Prepare the index that you want to migrate in AWS ES. This topic uses an index named movies.

Create an IAM role and an IAM policy

The IAM role authorizes AWS ES to access the S3 bucket, and the IAM policy limits the actions that the role can perform. You must create both and attach the policy to the role.

• Create an IAM role to authorize AWS ES. The role is required when you register the snapshot repository, and only an IAM user that has permissions to assume this role can register the repository.

  The Service statement in the trust relationship of the role must be set to es.amazonaws.com, as shown in the following example.

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
          "Service": "es.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
  }

  In the IAM console, choose Roles in the left-side navigation pane, find the target role, click the Trust relationships tab, and then click Edit trust relationship to view and edit the trust policy.

  Note

  AWS ES is not available in the Select role type drop-down list when you create a role for an AWS service in the IAM console. Select AWS EC2, create the role, and then change ec2.amazonaws.com to es.amazonaws.com.

• Create an IAM policy and attach it to the preceding IAM role to limit the actions that the role can perform on the S3 bucket.

  The Resource statement of the policy must specify the ARN of the S3 bucket that stores the snapshots. The following example specifies the eric-es-index-backups bucket.

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "s3:ListBucket"
              ],
              "Effect": "Allow",
              "Resource": [
                  "arn:aws:s3:::eric-es-index-backups"
              ]
          },
          {
              "Action": [
                  "s3:GetObject",
                  "s3:PutObject",
                  "s3:DeleteObject"
              ],
              "Effect": "Allow",
              "Resource": [
                  "arn:aws:s3:::eric-es-index-backups/*"
              ]
          }
      ]
  }

  After the policy is created, you can verify it in the IAM console: choose Policies in the left-side navigation pane, click the policy name to open the details page, and confirm on the JSON view of the Permissions tab that the policy content matches the preceding example. Policy summary shows that the policy grants Limited: List, Read, Write access to S3.

  Then, click Roles in the left-side navigation pane, select the target role, and confirm on the Permissions tab that the policy is attached.

Prepare the destination Alibaba Cloud resources

• Create an Alibaba Cloud Elasticsearch cluster in the China (Hangzhou) region. The version of the cluster must be no earlier than the exact patch version of the source AWS ES cluster. If the source cluster runs 7.10.2, select 7.16.2. For more information, see Create an Alibaba Cloud Elasticsearch cluster.

• Create an OSS bucket to receive the snapshot data that is transferred from AWS S3. The bucket used in this topic resides in the China (Hangzhou) region, uses the Standard storage class and the private ACL, and keeps the default values for all other parameters. For more information, see Create buckets.

Prepare the registration script

1. Download the register_snapshot_repository.py file.

2. Modify the parameter values that are marked by the comments in the script, and save the script as snapshot.py.

   The following table describes the parameters.

   Parameter	Description
   region	The AWS region in which the snapshot repository is created.
   host	The endpoint of the AWS ES domain.
   aws_access_key_id	The ID of the IAM credential.
   aws_secret_access_key	The key of the IAM credential.
   service	The AWS service identifier that is used for signing. For Amazon OpenSearch Service, including its Elasticsearch versions, the value is always es.
   url	The request URL that registers the snapshot repository, in the https://{host}/_snapshot/{repository-name} format. The repository name must be the same as the name that you use when you create and restore snapshots.
   payload	Must include the name and ARN of the S3 bucket that you created for the IAM role in Step 1: Prepare the resources required for the migration. Important We recommend that you add "base_path": "<custom-prefix>" to settings so that the snapshot files of this cluster are stored under the specified prefix and do not share a path with the snapshots of other clusters in the same S3 bucket. For more information, see the description of repository uniqueness in the FAQ section. Record the prefix: the base_path value that you specify when you register the OSS repository in Alibaba Cloud Elasticsearch must be the same as the actual prefix of the snapshot data in OSS. To enable server-side encryption with S3-managed keys for the snapshot repository, add "server_side_encryption": true to the settings JSON. If the S3 bucket resides in the ap-southeast-1 region, use "endpoint": "s3.amazonaws.com" instead of "region": "ap-southeast-1".

Install the dependencies and run the registration

1. Install the dependencies that the sample Python client requires.

   The sample Python client sends requests by using requests and signs the requests by using requests-aws4auth. Install both dependencies on the machine that runs the registration, in a Python 3 environment.

   pip install requests requests_aws4auth

2. Run the Python client to register the snapshot repository. If the registration succeeds, HTTP status code 200 and {"acknowledged" : true} are returned.

   python snapshot.py

Verify the registration result

1. Log on to the Kibana console of the AWS ES domain and run the following command on the Dev Tools > Console page to view the result. If fine-grained access control is enabled for the domain, or if the access policy allows only IAM principals and you cannot log on to Kibana, run the same command as a signed request. For example, reuse the preceding Python client and change the request method to GET.

   GET _snapshot

   The response returns the configuration of the registered snapshot repositories. The repository that the system automatically creates is named cs-automated. If encryption at rest is enabled for the domain, the repository is named cs-automated-enc.

   {
     "eric-snapshot-repository": {
       "type": "s3",
       "settings": {
         "bucket": "eric-es-index-backups",
         "base_path": "eric-aws",
         "region": "us-east-1",
         "role_arn": "arn:aws:iam::xxx:role/eric-iam-role-es"
       }
     },
     "cs-automated-enc": {
       "type": "s3"
     }
   }

Create a full snapshot in AWS ES

• Create a snapshot named snapshot_movies_1 for the movies index in the eric-snapshot-repository repository.

  PUT _snapshot/eric-snapshot-repository/snapshot_movies_1
  {
  "indices": "movies"
  }

• Check the snapshot status. If state is SUCCESS, the snapshot is created.

  GET _snapshot/eric-snapshot-repository/snapshot_movies_1

  {
    "snapshots": [
      {
        "snapshot": "snapshot_movies_1",
        "uuid": "BlgKLvgoSpSgwBnbD4hIWg",
        "version_id": 7100299,
        "version": "7.10.2",
        "indices": [
          "movies"
        ],
        "data_streams": [],
        "include_global_state": true,
        "state": "SUCCESS",
        "start_time": "2018-02-28T03:00:44.591Z",
        "start_time_in_millis": 1519786844591,
        "end_time": "2018-02-28T03:00:46.236Z",
        "end_time_in_millis": 1519786846236,
        "duration_in_millis": 1645,
        "failures": [],
        "shards": {
          "total": 5,
          "failed": 0,
          "successful": 5
        }
      }
    ]
  }

• Confirm in the AWS S3 console that the snapshot files are generated.

  After the snapshot is created, the S3 bucket eric-es-index-backups contains the generated snapshot files, including the indices folder, index-0, index.latest, meta-BlgKLvgoSpSgwBhbD4hTWg.dat, and snap-BlgKLvgoSpSgwBhbD4hTWg.dat.

Transfer the snapshot data to Alibaba Cloud OSS

Transfer the snapshot data from AWS S3 to Alibaba Cloud OSS. For more information, see Migrate applications from Amazon S3 to OSS.

After the transfer is complete, confirm in the OSS console that the snapshot data is in place.

The OSS bucket contains the following snapshot files: indices/ (folder), index-0, index.latest, meta-BlgKLvgoSpSgwBhbD4hTWg.dat, and snap-BlgKLvgoSpSgwBhbD4hTWg.dat.

Register an OSS snapshot repository in Alibaba Cloud Elasticsearch

1. Create a snapshot repository.

   During an Elasticsearch snapshot migration, you must ensure that each Elasticsearch cluster uses a dedicated snapshot repository or path. If multiple clusters share the same repository, data restoration may fail and data may be lost.

   Log on to the Kibana console of the destination Alibaba Cloud Elasticsearch cluster (Log on to the Kibana console) and run the following command on the Dev Tools > Console page to create a snapshot repository that has the same name. The base_path value must be the same as the actual prefix of the snapshot data after the data is transferred to OSS. If no base_path is configured for the repository on the AWS side and the snapshot files reside in the root directory of the bucket, do not configure base_path here either.

   PUT _snapshot/eric-snapshot-repository
   {
   "type": "oss",
   "settings": {
               "base_path": "my/snapshot/directory",
               "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
               "access_key_id": "your AccessKeyID",
               "secret_access_key": "your AccessKeySecret",
               "bucket": "eric-oss-aws-es-snapshot-s3",
               "compress": true
         }
   }

2. Check the status of the snapshot named snapshot_movies_1.

   GET _snapshot/eric-snapshot-repository/snapshot_movies_1

   If state in the response is SUCCESS, Alibaba Cloud Elasticsearch can read the snapshot from the OSS repository. If snapshot_missing_exception is returned, check whether base_path is the same as the actual prefix of the snapshot data in OSS.

   Note

   Record the start time and the end time of this snapshot. The values are required when you migrate the incremental snapshot data. Example:

   • "start_time_in_millis": 1519786844591

   • "end_time_in_millis": 1519786846236

Restore the indexes and verify the data

Run the following command in the Kibana console of the Alibaba Cloud Elasticsearch cluster to restore the snapshot.

POST _snapshot/eric-snapshot-repository/snapshot_movies_1/_restore
{
    "indices": "movies"
}

GET movies/_recovery

You can also verify the data in the Kibana console of the Alibaba Cloud Elasticsearch cluster: on the Discover page, select the movies index. In this example, three documents are returned after the restoration (The Manchurian Candidate, Mars Attacks!, and U.S. Marshals), which is the same as the source cluster.

Create an incremental snapshot in AWS ES

1. Write incremental data to the movies index in AWS ES.

   In this example, 2 documents are written. After the write operation, the movies index contains 5 documents whose _id values are 1 to 5. Run GET movies/_count to check the current number of documents, which is used for comparison with the destination cluster.

2. Create the incremental snapshot.

   Run the following command to manually create a snapshot. For more information, see Create a full snapshot in AWS ES.

   PUT _snapshot/eric-snapshot-repository/snapshot_movies_2
   {
   "indices": "movies"
   }

   After the snapshot is created, run the following command to check the snapshot status.

   GET _snapshot/eric-snapshot-repository/snapshot_movies_2

   View the new snapshot files in the AWS S3 console.

   The S3 bucket lists all files of the snapshot repository. The new files that correspond to snapshot_movies_2 are snap-CWhlF7ShQZaKQlJasPE70A.dat, index.latest, index-1, and meta-CWhlF7ShQZaKQlJasPE70A.dat.

Transfer the incremental snapshot data to Alibaba Cloud OSS

Transfer the incremental snapshot data from AWS S3 to Alibaba Cloud OSS by using ossimport. The S3 bucket already contains the files of the first snapshot. We recommend that you set isSkipExistFile to true in the local_job.cfg configuration file to transfer only the new snapshot files.

isSkipExistFile is a boolean parameter. The default value false indicates that existing objects are overwritten. If the value is true, the objects that are already migrated are skipped based on the size and LastModifiedTime values of the objects. This parameter does not take effect when jobType is audit.

After the transfer is complete, the new snapshot files appear in OSS.

The new files are index-1, index.latest, meta-CWhIF7ShQZaKQUasPE70A.dat, and snap-CWhIF7ShQZaKQUasPE70A.dat.

Restore the incremental snapshot and switch service traffic

After the restoration is complete, run GET movies/_count to verify the number of documents. In this example, the destination cluster returns 5, which is the same as the source AWS ES cluster. You can then switch service traffic to the Alibaba Cloud Elasticsearch cluster.