Separate hot and cold data with ILM | Elasticsearch - Elasticsearch

Use Index Lifecycle Management (ILM) to automatically move indices through hot, warm, cold, and delete phases in a hot-warm cluster, reducing storage costs. Available in Elasticsearch 6.6.0 and later.

Phase	Description
hot	Handles real-time writes for time-series data. The rollover API creates a new index when the current index reaches a specified document count, size, or age.
warm	The index becomes read-only and serves queries only.
cold	The index is no longer updated and is queried infrequently. Query speed may decrease.
delete	The index is permanently deleted.

You can apply an ILM policy to an index in two ways:

Apply a policy to an index template. The policy covers all indices under the alias. The following example uses this method.
Apply a policy to a single index. The policy affects only the current index. New indices created by rollover are not affected.

The following hot-warm scenario demonstrates this lifecycle:

Write data in real time. When the index reaches a threshold, a rollover creates a new index.
After rollover, the old index stays in the hot phase for 30 minutes, then enters warm.
After merge and shrink complete, the index enters cold 1 hour after rollover.
Data moves to warm nodes. The index is deleted 2 hours after rollover.

Procedure

Step 1: Create a hot-warm cluster and check attributes

Set hot and warm node attributes during cluster creation.
Step 2: Configure an ILM policy

Define an ILM policy and apply it to indices under an alias.
Step 3: Verify data distribution

Verify that cold-phase index shards reside on warm nodes.
Step 4: Update an ILM policy

Update an existing policy.
Step 5: Switch an ILM policy

Switch between different policies for a rollover.

Step 1: Create a hot-warm cluster and check attributes

A hot-warm cluster contains hot nodes for real-time writes and warm nodes for historical data.

Node type	Data requirements	Read/write performance	Specifications	Storage requirements
Hot node (hot)	Recent data, such as logs from the last 2 days.	High	High (for example, 32-core, 64 GB).	SSD cloud disks are recommended.
Warm node (warm)	Historical data, such as logs older than 2 days.	Low	Low (for example, 8-core, 32 GB).	Ultra disks recommended. Use OpenStore for serverless storage of large cold datasets.

In Alibaba Cloud Elasticsearch, warm nodes have a box_type value of warm (not cold), matching the warm tier in native Elasticsearch.

When you create an Alibaba Cloud Elasticsearch cluster, enable warm nodes to create a hot-warm cluster.
After warm nodes are enabled, the system adds the -Enode.attr.box_type parameter to node startup arguments:
- Hot nodes: -Enode.attr.box_type=hot
- Warm nodes: -Enode.attr.box_type=warm
Data nodes are designated as hot nodes only after you enable warm nodes.
Log on to the Kibana console of the cluster. Connect to an Elasticsearch cluster using Kibana.
In the left-side navigation pane, click Dev Tools.
In the Console, run the following command to check the node attributes.
```
GET _cat/nodeattrs?v&h=host,attr,value
```
If the response lists both hot and warm nodes, the cluster supports hot-warm architecture.

Step 2: Configure an ILM policy

In the Kibana console, run the following command to define an ILM policy.

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Parameter	Description
hot	A rollover triggers when any condition is met: index reaches 1 GB (`max_size`), exceeds one day (`max_age`), or contains 1,000 documents (`max_docs`). The old index waits 30 minutes before entering warm.
warm	The index shrinks to one shard and force-merges to one segment. It enters cold 1 hour after rollover.
cold	The index moves to a warm node. It enters delete 2 hours after rollover.
delete	The index is deleted.

Policy names cannot be changed after creation. You can also create policies in the Kibana console, but the UI limits time units to hours. Use the API for finer-grained units such as seconds.

Create an index template that routes new indices to hot nodes.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-policy",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

Parameter	Description
index.routing.allocation.require.box_type	Node type for index allocation.
index.lifecycle.name	Name of the ILM policy to apply.
index.lifecycle.rollover_alias	Alias used for rollover.

Create the initial index with a sequence number.

PUT gamestabes-000001
{
"aliases": {
    "gamestabes":{
       "is_write_index": true
        }
      }
}

You can also create a time-based index with date math.

Write data through the alias. The index rolls over when it meets the policy conditions and the next ILM check runs.

PUT gamestabes/_doc/1
{
    "EU_Sales" : 3.58,
    "Genre" : "Platform",
    "Global_Sales" : 40.24,
    "JP_Sales" : 6.81,
    "Name" : "Super Mario Bros.",
    "Other_Sales" : 0.77,
    "Platform" : "NES",
    "Publisher" : "Nintendo",
    "Year_of_Release" : "1985",
    "na_Sales" : 29.08
}

By default, ILM checks indices that match the policy criteria every 10 minutes. You can modify the check interval using the indices.lifecycle.poll_interval parameter.

(Optional) View the index lifecycle status in Kibana.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch section, click Index Management.
3. Click the Lifecycle phase drop-down list and select a lifecycle phase to filter indices.
4. Click the name of a filtered index to view its detailed configuration.

Step 3: Verify data distribution

After an index enters the cold phase, verify that its shards reside on warm nodes.

In the Kibana console, find the index that entered the cold phase.
Run the following command to check the shard distribution. Replace shrink-gamestabes-000012 with your index name.
```
GET _cat/shards/shrink-gamestabes-000012
```
If the shard's node has box_type set to warm, the index resides on a warm node.

Step 4: Update an ILM policy

Run the following command to update the game-policy. This example changes the delete phase duration:

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "3h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

View the updated policy version.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch section, click Index Lifecycle Policies.
3. Check the policy version number. It increments with each update. Active indices keep the old version until the next rollover.

Step 5: Switch an ILM policy

Create a new policy.

PUT /_ilm/policy/game-new
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "3GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply the new policy to the template.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-new",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

FAQ

How do I adjust the ILM check interval?

By default, ILM checks indices against the policy every 10 minutes. Data may exceed the threshold between checks. For example, an index may contain more than 1,000 documents before rollover triggers even if max_docs is 1,000.

Adjust the check frequency with the indices.lifecycle.poll_interval parameter:

Important

A shorter interval increases node load. Set this value based on your workload.

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}