PolarDB:Migrate data to PolarSearch using Reindex

Prerequisites

Ensure the following conditions are met:

• Environment dependency: The source Elasticsearch or OpenSearch cluster and the PolarSearch nodes must be in the same Virtual Private Cloud (VPC). If they are in different VPCs, you must first establish network connectivity between them before proceeding.

• Version compatibility: The PolarSearch version must be compatible with the source cluster version. For detailed compatibility information, see the Version Selection Guide.

Step 1: Create the destination index

Before you run the Reindex task, create a destination index in the PolarSearch cluster with the same Settings and Mappings as the source index.

1. Query the Settings of the source index.

   curl -XGET "http://<source_address>/<source_index>/_settings?pretty"

2. Query the Mappings of the source index.

   curl -XGET "http://<source_address>/<source_index>/_mapping?pretty"

3. Create the destination index on a PolarSearch node. Replace <source_settings> and <source_mappings> with the actual Settings and Mappings you obtained in the previous steps.

   curl -X PUT "http://<dest_address>/<dest_index>?pretty" \
     -u <dest_username>:<dest_password> \
     -H 'Content-Type: application/json' \
     -d'{
     "settings": {
       <source_settings>
     },
     "mappings": {
       <source_mappings>
     }
   }'

Step 2: Configure the Reindex allowlist

To allow PolarSearch nodes to access the remote source cluster, add the source cluster address to the Reindex allowlist. Otherwise, the Reindex task fails with one of the following errors:

• [address] not allowlisted in reindex.remote.allowlist

• [address] not allowlisted in reindex.remote.whitelist

Step 3: Run the Reindex task

1. Run the following command on a PolarSearch node to start the Reindex task. This command pulls data from the remote source cluster to the destination index in PolarSearch.

   curl -X POST "http://<dest_address>/_reindex?pretty" \
     -u <dest_username>:<dest_password> \
     -H 'Content-Type: application/json' \
     -d'{
     "source": {
       "remote": {
         "host": "http://<source_address>",
         "username": "<source_username>",
         "password": "<source_password>"
       },
       "index": "<source_index>" // Name of the source index to migrate
     },
     "dest": {
       "index": "<dest_index>" // Name of the destination index in PolarSearch
     }
   }'

   Parameters

   Parameter	Description
   source_address	The access address of the source Elasticsearch or OpenSearch cluster in host:port format.
   source_username	The username to access the source cluster. If authentication is not required, you can omit the username and password fields.
   source_password	The password to access the source cluster.
   source_index	The name of the source index to migrate.
   dest_address	The access address of the destination PolarSearch cluster in host:port format.
   dest_username	The username to access PolarSearch.
   dest_password	The password to access PolarSearch.
   dest_index	The name of the destination index in PolarSearch.

2. After the Reindex task is complete, verify the migration by comparing the document counts of the source and destination indexes.

   # Query the document count of the source index
   curl -XGET "http://<source_address>/<source_index>/_count?pretty"
   
   # Query the document count of the destination index
   curl -XGET "http://<dest_address>/<dest_index>/_count?pretty" \
     -u <dest_username>:<dest_password>

FAQ

• Q1: Can a single Reindex task migrate multiple indexes?

  No. The Reindex API supports only one-to-one index migration. To migrate multiple indexes, you must run a separate Reindex task for each index. Do not migrate data from multiple source indexes to a single destination index (a many-to-one migration), as this can cause data or field conflicts. Instead, run multiple one-to-one Reindex tasks. For example, migrate test_index1 to test_index_copy1, and then migrate test_index2 to test_index_copy2.

  curl -X POST "http://<dest_address>/_reindex?pretty" \
    -u <dest_username>:<dest_password> \
    -H 'Content-Type: application/json' \
    -d'{
    "source": {
      "remote": {
        "host": "http://<source_address>",
        "username": "<source_username>",
        "password": "<source_password>"
      },
      "index": "test_index1" // Name of the source index to migrate
    },
    "dest": {
      "index": "test_index_copy1" // Name of the destination index in PolarSearch
    }
  }'
  
  curl -X POST "http://<dest_address>/_reindex?pretty" \
    -u <dest_username>:<dest_password> \
    -H 'Content-Type: application/json' \
    -d'{
    "source": {
      "remote": {
        "host": "http://<source_address>",
        "username": "<source_username>",
        "password": "<source_password>"
      },
      "index": "test_index2" // Name of the source index to migrate
    },
    "dest": {
      "index": "test_index_copy2" // Name of the destination index in PolarSearch
    }
  }'

• Q2: What should I do if the Reindex task fails with an illegal_argument_exception error?

  This error indicates that a field type in the source index is incompatible with the Mappings of the destination index. Before you run the Reindex task, ensure the Mappings of the destination index match those of the source index.

  Error example:

  "cause" : {
    "type" : "illegal_argument_exception",
    "reason" : "mapper [...] cannot be changed from type [float] to [long]"
  }

• Q3: How do I configure the allowlist for multiple source clusters?

  If you need to migrate data from multiple source clusters, you can add multiple addresses to the allowlist parameter, separated by commas, when you submit a ticket to contact us. For example: reindex.remote.allowlist: "host1:9200, host2:9200".

Appendix: Available Reindex parameters

Sample request

curl -X POST "http://<dest_address>/_reindex?pretty&refresh=false&timeout=30s&wait_for_active_shards=1&wait_for_completion=false&requests_per_second=-1&require_alias=false&scroll=5m&slices=1&max_docs=100000"
-u <dest_username>:<dest_password>
-H 'Content-Type: application/json'
-d'
{
  "source": {
    "remote": {
      "host": "http://<source_address>",
      "username": "<source_username>",
      "password": "<source_password>"
    },
    "index": "test_index" // The name of the index to migrate
  },
  "dest": {
    "index": "test_index_copy"  // The name of the destination index in PolarSearch
  }
}'

Parameters