Elasticsearch:Evaluate specifications and storage capacity

Worked example: 200 GiB source data

Scenario: 200 GiB of source data, one replica shard per primary shard.

Cluster storage = 200 GiB x (1 + 1) x 1.7
                = 200 x 2 x 1.7
                = 680 GiB

Storage consumed outside the formula

Beyond the factors in the formula, these items also consume storage:

• X-Pack monitoring indexes -- Used for exception analysis:

  • .monitoring-es-6-*: Consumes significant storage. Retains the last 7 days by default.

  • .monitoring-kibana-6-*: Grows with the number of indexes. Retains the last 7 days by default.

  • .watcher-history-3-*: Consumes minimal storage. Delete manually when no longer needed.

• Cluster logs -- Include run logs, access logs, and slow logs. Retained for the last 7 days by default. This retention period cannot be changed. Log volume increases with the number of queries and data pushes the cluster receives.

Data nodes

Two rules determine the maximum scale per data node:

• Maximum nodes per cluster = vCPUs per node x 5

• Maximum storage per node = Memory per node (GiB) x a scenario-specific multiplier

The following table shows the maximum node count and maximum storage per node for each specification:

Total cluster storage = Storage per node x Number of nodes

Select node specifications based on the maximum storage per node and the maximum number of nodes for your target specification.

• The number of data nodes affects the total shard count. Complete the shard evaluation below before finalizing node specifications.

• For aggregation-heavy queries, select specifications with a 1:2 vCPU-to-memory ratio and enable client nodes.

Dedicated master nodes

Enable dedicated master nodes for clusters with many data nodes to maintain cluster stability. Select the specification based on your data node count:

If the cluster has many indexes and shards, or data changes frequently, select higher specifications for dedicated master nodes.

Client nodes

Client nodes (coordinating node in Elasticsearch) handle the reduce phase of distributed queries. Dedicated client nodes isolate garbage collection (GC) impact from data nodes.

Example: For 10 data nodes at 8 vCPUs, 32 GiB each, configure 2 client nodes at 8 vCPUs, 32 GiB each.

Shard size guidelines

Number of shards per index

Determine the shard count based on data volume:

• Large data volume, high write throughput: Configure multiple primary shards per index with one replica per primary shard.

• Small data volume, low write throughput: Configure one primary shard per index with one or more replica shards.

Default shard configuration varies by version:

• V7.X and later: 1 primary shard, 1 replica shard per index

• Earlier than V7.X: 5 primary shards, 1 replica shard per index

Load balancing with small indexes: If the data volume per index is less than 30 GiB, use one primary shard with multiple replicas to distribute load across nodes. For example, a 20 GiB index on a 5-node cluster can use 1 primary shard and 4 replica shards.

Shard distribution guidelines

• Keep the total shard count equal to the data node count, or an integer multiple of it.

• Place a maximum of 5 shards per index on a single node.

Total shards per node

Calculate the maximum number of shards a single data node can hold:

• The default maximum shard count per node in V7.X clusters is 1,000. Do not change this limit. If more shards are needed, add more nodes instead.

• Excessive shards increase performance overhead and may exhaust file handles, leading to cluster faults. Configure shards based on actual business requirements.

For more guidance, see How to size your shards.