Elasticsearch:Shrink indices with the _shrink API

Prerequisites

• Cluster health: Your Elasticsearch cluster must be healthy and operating under normal load (green status).

• Index health: The index must have a green health status.

• Elasticsearch version: Elasticsearch V5.X+.

Usage notes

Before performing a shrink operation, be aware of these critical points:

• Shard count for shrink:

  • The source index must have more primary shards than the target index.

  • The source index’s primary shard count must be a factor of the source index shard number.

    • Example: An index with 8 primary shards can shrink to 4, 2, or 1. An index with 15 primary shards can shrink to 5, 3, or 1. If the source index has a prime number of primary shards (e.g., 7), it can only shrink to 1 primary shard.

• Disk space: The single node that will hold all shards of the source index must have sufficient disk space. This space should be greater than the total size of the data stored across all primary shards of the source index. For guidance on shard evaluation and capacity planning, see Assess shards.

• Maximum documents: The source index must contain no more than 2,147,483,519 documents.

• Target index name: No index with the same name as your intended new index must exist in your Elasticsearch cluster.

• Off-peak operation: The shrink operation temporarily increases CPU usage and load on the node performing the shrink. We strongly recommend performing shrink operations during off-peak hours to minimize impact on cluster performance.

Procedure

1. Log on to the Kibana console of your Elasticsearch cluster and go to the Kibana homepage.

2. In the left navigation menu, click Dev tools.

3. In Console, disable writes for the source index.

   PUT /my_original_index/_settings
   {
       "index.routing.allocation.require._name": "es-cn-zvp25yhyy000y****-1ab7****-0001",
       "index.blocks.write": true,
       "index.number_of_replicas": 0
   }                     

   • Replace my_original_index with the name of the source index.

   • Replace es-cn-zvp25yhyy000y****-1ab7****-0001 with the actual name of one of your data nodes. You can get node names by running GET _cat/nodes?v in Dev Tools.

   • index.routing.allocation.require._name: This forces all primary shards of my_original_index to move to the specified node.

   • index.blocks.write: true: Disables write operations to the source index.

   • index.number_of_replicas: 0: Ensures there are no replica shards, simplifying the relocation process.

4. Execute the _shrink API:

   This command initiates the shrink operation, creating my_shrunk_index with fewer primary shards. It also configures the target index.

   POST /my_original_index/_shrink/my_shrunk_index
   {
     "settings": {
       "index.blocks.write": null,
       "index.number_of_shards": 5,
       "index.number_of_replicas": 0,
       "index.routing.allocation.require._name": null
     }
   }

   • Replace my_original_index and my_shrunk_index with your index names.

   • index.blocks.write: null: Clears the write block from the new index, allowing writes to it immediately after creation.

   • index.number_of_shards: Specify the desired number of primary shards for my_shrunk_index. Ensure this number meets the divisibility rules described in the Prerequisites section.

   • index.number_of_replicas: Sets the number of replica shards for the target index.

   • index.routing.allocation.require._name: null: Clears the shard allocation requirement for the target index, allowing its shards to be distributed across the cluster.

5. Verify shrink progress and cluster health.

   After initiating the shrink, it's crucial to monitor its progress and confirm the cluster's health.

   1. Check shrink progress

      Use the _cat recovery API to see if any shard recoveries related to the shrink operation are still active.

      GET _cat/recovery?v&active_only

      If the response shows no indices in the "index" column related to the shrink, it indicates that the shrink operation (and its associated recoveries) is complete.

   2. Check cluster health

      Confirm that your Elasticsearch cluster's overall health status is green.

      GET _cluster/health

      A response containing "status" : "green" confirms the cluster is healthy.

FAQ

Q: Why use hard links instead of symbolic links?

A: Hard links ensure index independence. If you use symbolic links, deleting the source index after writing data to the target index also deletes data in the target index. Hard links prevent this.