Elasticsearch:Using Reindex API to migrate multi-type data

Considerations

Workflow

1. Prerequisites

   Prepare Alibaba Cloud Elasticsearch and Logstash instances. Make sure that they are in the same Virtual Private Cloud (VPC).

   • Alibaba Cloud Elasticsearch instance: Used to store index data.

   • Alibaba Cloud Logstash instance: Used to migrate the processed data using a pipeline.

2. Step 1: Convert index types

   Use the reindex API to convert multi-type indices in an Alibaba Cloud Elasticsearch 5.x instance to single-type indices. The following methods are supported:

   • Merge types: Use a reindex script to merge data from a multi-type index in an Elasticsearch 5.x instance into a new single-type index.

   • Split types: Use the reindex API to split data from a multi-type index in an Elasticsearch 5.x instance into multiple single-type indices based on the original type.

3. Step 2: Migrate data using Logstash

   Use an Alibaba Cloud Logstash instance to migrate the processed index data to an Elasticsearch 6.x instance.

4. Step 3: Verify the data migration

   View the migrated indices in Kibana.

Prerequisites

1. Prepare earlier-version (5.5.3) and later-version (6.7.0) Alibaba Cloud Elasticsearch instances, and the multi-type data to be migrated.

   For more information about how to create an instance, see Create an Alibaba Cloud Elasticsearch instance.

2. Create an Alibaba Cloud Logstash instance in the same Virtual Private Cloud as your Alibaba Cloud Elasticsearch instances.

   For more information, see Step 2: Create an Alibaba Cloud Logstash instance.

Step 1: Convert index types

The following steps merge types to consolidate data from a multi-type index into a single-type index.

1. Enable automatic index creation for your Elasticsearch instance.

   1. Log on to the Alibaba Cloud Elasticsearch console.

   2. In the left-side navigation pane, click Elasticsearch Clusters.

   3. In the top navigation bar, select a resource group and a region.

   4. In the instance list, click the ID of the earlier-version instance.

   5. In the left-side navigation pane, click Cluster Configuration.

   6. Click YML File Configuration next to Modify Configuration.

   7. On the YML File Configuration page, set Auto Indexing to Enable.

      Warning

      Changing the setting for Auto Indexing restarts the instance. Make sure that this action does not adversely affect your services before you proceed.

   8. Select the This operation restarts the instance. Confirm to proceed. checkbox, and then click OK.

2. Log on to the Kibana console of the earlier-version Elasticsearch instance.

   For more information, see Log on to the Kibana console.

3. In the left-side navigation pane, click Dev Tools.

   If the earlier-version instance does not contain multi-type index data, run the following commands in the Console to create test data.

   PUT twitter/tweet/1
   {"user": "kimchy", "message": "trying out Elasticsearch"}
   PUT twitter/tweet/2
   {"user": "kimchy", "message": "another tweet"}
   PUT twitter/user/1
   {"name": "kimchy", "email": "kimchy@elastic.co"}
   PUT twitter/user/2
   {"name": "elastic", "email": "info@elastic.co"}

4. In the Console, run the following command to merge data from a multi-type index into a single-type index.

   POST _reindex
   {
     "source": {
       "index": "twitter"
     },
     "dest": {
       "index": "new1"
     },
     "script": {
       "inline": """
       ctx._id = ctx._type + "-" + ctx._id;
       ctx._source.type = ctx._type;
       ctx._type = "doc";
       """,
       "lang": "painless"
     }
   }

   In the preceding example, a custom type is specified by using ctx._source.type. This adds a type field to the new1 index and sets its value to the value of the original _type. In addition, the _id of the new1 index is constructed from _type-_id to prevent conflicts from occurring if documents of different types have the same ID.

5. Run the GET new1/_mapping command to view the merged Mapping structure.

6. Run the following command to view the data in the merged index.

   GET new1/_search
   {
      "query":{
        "match_all":{
         }
     }
   }

The following steps split types to reindex a multi-type index into multiple single-type indices.

1. In the Console, run the following commands to split a multi-type index into single-type indices.

   POST _reindex
   {
     "source": {
       "index": "twitter",
       "type": "tweet",
       "size": 10000
     },
     "dest": {
       "index": "twitter_tweet"
     }
   }
   POST _reindex
   {
     "source": {
       "index": "twitter",
       "type": "user",
       "size": 10000
     },
     "dest": {
       "index": "twitter_user"
     }
   }

   The example above splits the twitter index into the twitter_tweet and twitter_user indices based on different types.

2. Run the following commands to view the data in the split indices.

   GET twitter_tweet/_search
   {
      "query":{
        "match_all":{
   
         }
     }
   }

   GET twitter_user/_search
   {
      "query":{
        "match_all":{
   
         }
     }
   }

Step 2: Migrate data using Logstash

1. Go to the Logstash Clusters page.

2. Navigate to the target cluster.

   1. In the top navigation bar, select the region where the cluster resides.

   2. On the Logstash Clusters page, find the cluster and click its ID.

3. In the left-side navigation pane, click Pipelines.

4. Click Create Pipeline.

5. On the Create page, enter a Pipeline ID and configure the pipeline.

   Sample pipeline configuration:

   input {
       elasticsearch {
       hosts => ["http://es-cn-0pp1f1y5g000h****.elasticsearch.aliyuncs.com:9200"]
       user => "elastic"
       index => "*"
       password => "your_password"
       docinfo => true
     }
   }
   filter {
   }
   output {
     elasticsearch {
       hosts => ["http://es-cn-mp91cbxsm000c****.elasticsearch.aliyuncs.com:9200"]
       user => "elastic"
       password => "your_password"
       index => "test"
     }
   }

   In the preceding example, the index parameter of the output section is set to a fixed value test. Replace it with the actual target index name as needed.

   For more information about pipeline configuration parameters, see Logstash configuration file reference.

6. Click Next step and configure the pipeline parameters.

   Parameter	Description
   Pipeline Workers	Number of worker threads for the filter and output stages. Increase this value if events are backlogged or CPU is underutilized. Default: the number of CPU cores.
   Pipeline Batch Size	Maximum events a worker collects before executing filters and outputs. Larger batches increase memory usage and may require a larger JVM heap size (LS_HEAP_SIZE). Default: 125.
   Pipeline Batch Delay	Wait time in milliseconds before dispatching an undersized batch to a worker thread. Default: 50 ms.
   Queue Type	Internal queuing model for event buffering. Valid values: MEMORY: Default. Uses an in-memory queue. PERSISTED: A disk-based persistent queue.
   Queue Max Bytes	The maximum amount of data the queue can store, in MB. The value must be an integer from 1 to 253-1. Default value: 1024. Note Make sure that this value is less than your total disk capacity.
   Queue Checkpoint Writes	Maximum events written before a checkpoint is forced (persistent queue only). 0 means no limit. Default: 1024.

   Warning

   Saving and deploying triggers an instance restart. Proceed only if this will not impact your business.

7. Click Save or Save and Deploy.

   • Save: Saves the pipeline configuration but does not apply it. After saving, you are returned to the Pipelines page. In the Pipelines section, you can click Deploy Now in the Actions column to restart the instance and apply the configuration.

   • Save and Deploy: Saves and deploys the configuration, restarting the instance to apply the changes.

Step 3: Verify data migration

1. Log on to the Kibana console of the later-version Elasticsearch instance.

   For more information, see Log on to the Kibana console.

2. In the left-side navigation pane, click Dev Tools.

3. In the Console, run the following command to view the migrated indices.

   GET _cat/indices?v