Separate hot and cold data with ILM | Elasticsearch - Elasticsearch

Use Index Lifecycle Management (ILM) to automatically move indices through hot, warm, cold, and delete phases in a hot-warm cluster, reducing storage costs. Available in Elasticsearch 6.6.0 and later.

Phase	Description
hot	Handles real-time writes for time-series data. The rollover API creates a new index when the current index reaches a specified document count, size, or age.
warm	The index becomes read-only and serves queries only.
cold	The index is no longer updated and is queried infrequently. Query speed may decrease.
delete	The index is permanently deleted.

You can apply an ILM policy to an index in two ways:

Apply a policy to an index template. The policy covers all indices under the alias. The following example uses this method.
Apply a policy to a single index. The policy affects only the current index. New indices created by rollover are not affected.

The following hot-warm scenario demonstrates this lifecycle:

Write data in real time. When the index reaches a threshold, a rollover creates a new index.
After rollover, the old index stays in the hot phase for 30 minutes, then enters warm.
After merge and shrink complete, the index enters cold 1 hour after rollover.
Data moves to warm nodes. The index is deleted 2 hours after rollover.

Procedure

Step 1: Create a hot-warm cluster and check attributes
Set hot and warm node attributes during cluster creation.
Step 2: Configure an ILM policy
Define an ILM policy and apply it to indices under an alias.
Step 3: Verify data distribution
Verify that cold-phase index shards reside on warm nodes.
Step 4: Update an ILM policy
Update an existing policy.
Step 5: Switch an ILM policy
Switch between different policies for a rollover.

Step 1: Create a hot-warm cluster and check attributes

A hot-warm cluster contains hot nodes for real-time writes and warm nodes for historical data.

Node type	Data requirements	Read/write performance	Specifications	Storage requirements
Hot node (hot)	Recent data, such as logs from the last 2 days.	High	High (for example, 32-core, 64 GB).	SSD cloud disks are recommended.
Warm node (warm)	Historical data, such as logs older than 2 days.	Low	Low (for example, 8-core, 32 GB).	Ultra disks recommended. Use OpenStore for serverless storage of large cold datasets.

In Alibaba Cloud Elasticsearch, warm nodes have a box_type value of warm (not cold), matching the warm tier in native Elasticsearch.

When you create an Alibaba Cloud Elasticsearch cluster, enable warm nodes to create a hot-warm cluster.
After warm nodes are enabled, the system adds the -Enode.attr.box_type parameter to node startup arguments:
- Hot nodes: -Enode.attr.box_type=hot
- Warm nodes: -Enode.attr.box_type=warm
Data nodes are designated as hot nodes only after you enable warm nodes.
Log on to the Kibana console of the cluster. Connect to an Elasticsearch cluster using Kibana.
In the left-side navigation pane, click Dev Tools.
In the Console, run the following command to check the node attributes.
```
GET _cat/nodeattrs?v&h=host,attr,value
```
If the response lists both hot and warm nodes, the cluster supports hot-warm architecture.

Step 2: Configure an ILM policy

In the Kibana console, run the following command to define an ILM policy.

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Parameter	Description
hot	A rollover triggers when any condition is met: index reaches 1 GB (`max_size`), exceeds one day (`max_age`), or contains 1,000 documents (`max_docs`). The old index waits 30 minutes before entering warm.
warm	The index shrinks to one shard and force-merges to one segment. It enters cold 1 hour after rollover.
cold	The index moves to a warm node. It enters delete 2 hours after rollover.
delete	The index is deleted.

Policy names cannot be changed after creation. You can also create policies in the Kibana console, but the UI limits time units to hours. Use the API for finer-grained units such as seconds.

Create an index template that routes new indices to hot nodes.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-policy",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

Parameter	Description
index.routing.allocation.require.box_type	Node type for index allocation.
index.lifecycle.name	Name of the ILM policy to apply.
index.lifecycle.rollover_alias	Alias used for rollover.

Create the initial index with a sequence number.

PUT gamestabes-000001
{
"aliases": {
    "gamestabes":{
       "is_write_index": true
        }
      }
}

You can also create a time-based index with date math.

Write data through the alias. The index rolls over when it meets the policy conditions and the next ILM check runs.

PUT gamestabes/_doc/1
{
    "EU_Sales" : 3.58,
    "Genre" : "Platform",
    "Global_Sales" : 40.24,
    "JP_Sales" : 6.81,
    "Name" : "Super Mario Bros.",
    "Other_Sales" : 0.77,
    "Platform" : "NES",
    "Publisher" : "Nintendo",
    "Year_of_Release" : "1985",
    "na_Sales" : 29.08
}

By default, ILM checks indices that match the policy criteria every 10 minutes. You can modify the check interval using the indices.lifecycle.poll_interval parameter.

(Optional) View the index lifecycle status in Kibana.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch section, click Index Management.
3. Click the Lifecycle phase drop-down list and select a lifecycle phase to filter indices.
4. Click the name of a filtered index to view its detailed configuration.

Step 3: Verify data distribution

After an index enters the cold phase, verify that its shards reside on warm nodes.

In the Kibana console, find the index that entered the cold phase.
Run the following command to check the shard distribution. Replace shrink-gamestabes-000012 with your index name.
```
GET _cat/shards/shrink-gamestabes-000012
```
If the shard's node has box_type set to warm, the index resides on a warm node.

Step 4: Update an ILM policy

Run the following command to update the game-policy. This example changes the delete phase duration:

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "3h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

View the updated policy version.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch section, click Index Lifecycle Policies.
3. Check the policy version number. It increments with each update. Active indices keep the old version until the next rollover.

Step 5: Switch an ILM policy

Create a new policy.

PUT /_ilm/policy/game-new
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "3GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply the new policy to the template.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-new",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

FAQ

How do I adjust the ILM check interval?

By default, ILM checks indices against the policy every 10 minutes. Data may exceed the threshold between checks. For example, an index may contain more than 1,000 documents before rollover triggers even if max_docs is 1,000.

Adjust the check frequency with the indices.lifecycle.poll_interval parameter:

Important

A shorter interval increases node load. Set this value based on your workload.

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}

Does the Alibaba Cloud Elasticsearch console automatically clean up indices?

No. The Alibaba Cloud Elasticsearch console cannot automatically clean up indices. You must use Index Lifecycle Management (ILM) to manage the index lifecycle.

By configuring an ILM policy, you can automatically delete indices on a schedule after creation (the delete phase). In a hot-warm architecture, ILM can also automatically migrate data from hot nodes to warm nodes. For configuration steps, see Step 2 in this topic.

Why does deleting a large amount of data cause CPU spikes or cluster stalls, and how can ILM help?

When Elasticsearch deletes data, Lucene creates a .del file to mark the corresponding documents as deleted. During background segment merging, Elasticsearch reads old segments, skips marked documents, and writes new segments. Deleting a large amount of data can therefore consume significant CPU resources and cause cluster stalls.

Use an ILM policy to manage data in phases and avoid the segment merging pressure caused by concentrated deletion. You can also directly delete an index that is no longer needed. Deleting an index is a metadata operation and does not create segment merging pressure, but ensure that no workloads still use the index.

ILM distributes background segment merging pressure by automatically performing delete or merge operations across the lifecycle, instead of marking a large number of documents as deleted at the same time. Configure ILM from the command line when required because the Alibaba Cloud Elasticsearch console might not support every configuration detail.

Can ILM accidentally delete system indices such as security indices?

Whether an index is deleted depends on the ILM policy associated with that index. If you modify the default policy for a system index, such as a security index, the index might be deleted. If you do not modify the default policy, the index is not accidentally deleted.

When you configure an ILM policy, distinguish system indices from business indices. Do not apply a custom lifecycle policy to a system index.

How long does automatic segment merging take to clean up data, and how can I release space promptly?

Automatic segment merging has no fixed duration. It does not run in real time, and its duration depends on the current write workload. If you must release storage space promptly, run the _forcemerge command manually or automate the operation with an ILM policy.

The warm-phase forcemerge action in Step 2: Configure an ILM policy shows how to force merge an index to one segment as part of the ILM policy.