Use the _shrink API to shrink an index into a new index with fewer primary shards - Elasticsearch

Before you create an index in an Elasticsearch cluster, you must determine the number of primary shards to be configured for the index based on the actual volume of business data. If you have only a small amount of business data but configure a large number of primary shards for the index, resources may be excessively consumed and QPS or write throughput may be affected. In this case, we recommend that you reduce the number of primary shards configured for the index. This topic describes how to use the _shrink API to shrink an existing index into a new index with fewer primary shards.

Background information

When you use an Elasticsearch cluster, you must take note of the total number of shards configured for indexes in the cluster and shard configuration for each index in the cluster. The larger the total number of shards configured for indexes in the cluster, the more the file handles occupied by the shards, and the more the resources consumed in the cluster. In addition, inappropriate shard configuration such as excessive primary shards affects query and write operations.

If you configure an excessive number of primary shards for an index, you can use the reindex API to reduce the number of primary shards. However, this method requires a long period of time. To reduce time costs, open source Elasticsearch provides the _shrink API. When you use the _shrink API, a shrink operation is performed. You must complete the following steps when you use the _shrink API to reduce the number of primary shards for an index:

Create a new index. The number of primary shards for the new index is less than that for the original index, but the other settings for the new index are the same as those for the original index. Relocate all shards for the original index to the same node in the Elasticsearch cluster and make sure that the reserved disk space for the node is greater than the size of data stored on all primary shards for the original index.
Create hard links to link segments from the original index to the new index.
Recover the new index. This operation is similar to opening a closed index.

The following descriptions provide information about performance tests that are performed on the reindex API and _shrink API:

Test environment:
- Data nodes: five data nodes, each of which offers 8 vCPUs and 16 GiB of memory
- Data volume: 182 GiB of data stored in an index
- Number of shards: 30 primary shards for the original index, 5 primary shards for the new index, and no replica shards for both indexes

Test results


Method	Consumed time	Resource usage
reindex API	3 hours and 22 minutes	The write QPS in the cluster is excessively high, and the resource usage of the data nodes is high.
`_shrink` API	15 minutes	The computing resource usage of the node on which the shrink operation is performed is high.

Prerequisites

The Elasticsearch cluster is healthy, and the load of the cluster is normal.
The number of primary shards to be obtained after the shrink operation is evaluated based on the number of data nodes in the Elasticsearch cluster and the disk space of the Elasticsearch cluster. For more information, see Shard evaluation.
The original index is in a normal state indicated by the color green.
The number of documents stored in the original index does not exceed 2,147,483,519.
The Elasticsearch cluster does not contain an index that is named the same as the new index.

Procedure

Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.
For more information about how to log on to the Kibana console, see Log on to the Kibana console.

Note In this example, an Elasticsearch V7.10.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
In the upper-right corner of the page that appears, click Dev tools.

On the Console tab of the page that appears, run the following command to disable data write operations for the original index, set the number of replica shards for the original index to 0, and relocate all shards for the original index to the same node in the Elasticsearch cluster.

In this example, an original index named shrink5 is used. You must replace the name of the original index in the following command based on your business requirements.

PUT shrink5/_settings
{
    "index.routing.allocation.require._name": "es-cn-zvp25yhyy000y****-1ab7****-0001",
    "index.blocks.write": true,
    "index.number_of_replicas": 0
}


Parameter	Description
index.routing.allocation.require._name	The name of the node to which you want to relocate shards. You can run the `GET _cat/nodes?v` command to obtain the name. Note Before you call the `_shrink` API to shrink an existing index into a new index with fewer primary shards, you must relocate all shards for the original index to the same node in the Elasticsearch cluster.
index.blocks.write	Specifies whether to disable data write operations for the original index. Set this parameter to true. The value true indicates that data write operations are disabled for the original index. Note Before you call the `_shrink` API to shrink an existing index into a new index with fewer primary shards, you must disable data write operations for the original index.
index.number_of_replicas	The number of replica shards for the original index.

Call the _shrink API to shrink the original index into a new index with fewer primary shards.

The following command provides an example on how to shrink the original index shrink5 that has 30 primary shards to the new index shrink_hk5e_cn that has five primary shards. You must replace the name of the original index and that of the new index in the following command based on your business requirements.

POST shrink5/_shrink/shrink_hk5e_cn
{
  "settings": {
    "index.blocks.write": null,
    "index.number_of_shards": 5,
    "index.number_of_replicas": 0,
    "index.routing.allocation.require._name": null
  }
}


Parameter	Description
index.blocks.write	Specifies whether to disable data write operations for the new index. Set this parameter to null. This way, the settings that are copied from the original index are cleared.
index.number_of_shards	The number of primary shards for the new index. Notice After the shrink operation is triggered, the CPU utilization and minute-average load of the node on which the shrink operation is performed are high. We recommend that you shrink indexes during off-peak hours. The number of primary shards for the original index must be greater than that for the new index. The number of primary shards for the original index must be divisible by the number of primary shards for the new index. For example, if the number of primary shards for the original index is 8, the number of primary shards for the new index can be 4, 2, or 1. If the number of primary shards for the original index is 15, the number of primary shards for the new index can be 5, 3, or 1. If the number of primary shards for the original index is a prime number, the number of primary shards for the new index can only be 1.
index.number_of_replicas	The number of replica shards for the new index.
index.routing.allocation.require._name	The name of the node to which you want to relocate shards. Set this parameter to null. This way, the settings that are copied from the original index are cleared.

View the result.
Call the _cat recovery API to query the index shrink progress. If no recoveries about index shrink are returned and the Elasticsearch cluster is healthy, the index shrink is complete.
- Query the index shrink progress
```
GET _cat/recovery?v&active_only
```
  If no index that is waiting to be shrunk is displayed in the index column in the returned result, no recoveries about index shrink exist.
- Query the health status of the Elasticsearch cluster
```
GET _cluster/health
```
  If the returned result contains "status" : "green", the Elasticsearch cluster is healthy.

FAQ

Q: Why are hard links instead of symbolic links used?

A: Hard links ensure the independence of the new index. If you use symbolic links, and you delete the original index after data is written to the new index, data in the new index is also deleted. Hard links ensure that data in the new index is not deleted.