Configure ILM for a hot-warm architecture - Elasticsearch

Elasticsearch version 6.6.0 and later provides the index lifecycle management (ILM) feature. ILM divides the index lifecycle into four phases: hot, warm, cold, and delete. You can use ILM with a hot-warm architecture to automate data transitions and optimize storage costs.

Phase	Description
hot	Hot phase. The index handles real-time writes for time-series data. You can use the rollover API to create a new index when the current index reaches a specified document count, size, or age.
warm	Warm phase. The index is no longer written to but can still be queried.
cold	Cold phase. The index is no longer updated and is queried infrequently. Query performance is slower.
delete	Delete phase. The index is permanently deleted.

You can add a lifecycle policy to an index in two ways:

Add a policy to an index template: The policy applies to all indices covered by the alias. This topic uses this method as an example.
Add a policy to a single index: The policy applies only to the current index. New indices created by a rollover are not affected.

This topic uses a hot-warm architecture scenario as an example to demonstrate the following workflow:

Index data is written to Elasticsearch in real time. When the index reaches a specified threshold, subsequent data is automatically written to a new index.
The previous index remains in the hot phase for 30 minutes before transitioning to the warm phase.
After the merge and shrink operations in the warm phase complete, the index transitions to the cold phase one hour after the rollover.
In the cold phase, data is migrated to warm nodes. The index is deleted two hours after the rollover.

Procedure

Step 1: Create a hot-warm cluster and check attributes
Set the hot and warm attributes for nodes when you create the cluster.
Step 2: Configure an ILM policy
Define an ILM policy and apply it to all indices covered by the alias.
Step 3: Verify data distribution
Verify that the shards of the index in the cold phase are distributed on warm nodes.
Step 4: Update an ILM policy
Update an existing policy.
Step 5: Switch ILM policies
Switch between different policies for new indices.

Step 1: Create a hot-warm cluster and check attributes

A hot-warm cluster contains nodes with hot and warm attributes. Hot nodes handle real-time writes, and warm nodes store historical data. The following table describes the differences.

Node type	Data storage	Performance	Specifications	Storage
Hot node (hot)	Recent data, such as log data from the last two days.	High	High, for example, 32-core 64 GB.	SSD cloud disks are recommended.
Warm node (warm)	Historical data, such as log data older than two days.	Low	Low, for example, 8-core 32 GB.	Ultra Disks are recommended. Alternatively, use OpenStore for serverless storage of large volumes of cold data.

In Alibaba Cloud Elasticsearch, the box_type value for a warm node is warm, not cold. This is because this node type corresponds to the warm tier in the native Elasticsearch architecture.

When you create an Alibaba Cloud Elasticsearch instance, enable warm nodes to create a hot-warm cluster.
After you enable and purchase warm nodes, the system adds the -Enode.attr.box_type parameter to the node startup configuration:
- Hot nodes: -Enode.attr.box_type=hot
- Warm nodes: -Enode.attr.box_type=warm
When you enable warm nodes, your existing data nodes are automatically configured as hot nodes.
Log on to the Kibana console of the cluster.
In the left-side navigation pane, click Dev Tools.
In the Console, run the following command to view the attributes of the hot and warm nodes.
```
GET _cat/nodeattrs?v&h=host,attr,value
```
If the response includes both hot and warm nodes, the cluster supports the hot-warm architecture.

Step 2: Configure an ILM policy

In the Kibana console, run the following command to define an ILM policy.

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Parameter	Description
hot	A rollover occurs when the index reaches 1 GB, exceeds one day in age, or contains 1,000 documents. The previous index then waits 30 minutes before entering the warm phase.
warm	The system shrinks the index to one shard and force-merges it into a single segment. After these actions complete, the index transitions to the cold phase one hour after the rollover.
cold	The system migrates the index from hot nodes to warm nodes. The index then transitions to the delete phase two hours after the rollover.
delete	The system permanently deletes the index.

Policy names cannot be changed after creation. You can also create policies in the Kibana console, but the minimum unit for max_age in Kibana is hours, whereas the minimum unit when using the API is seconds.

Create an index template and specify the box_type attribute in the settings to ensure that newly indexed data is stored on hot nodes.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-policy",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

Parameter	Description
index.routing.allocation.require.box_type	Specifies the node type to which the index is allocated when it is created.
index.lifecycle.name	Specifies the name of the lifecycle policy.
index.lifecycle.rollover_alias	Specifies the rollover alias.

Create the initial index with a sequence number.
```
PUT gamestabes-000001
{
"aliases": {
    "gamestabes":{
       "is_write_index": true
        }
      }
}
```
You can also create an index based on time. For more information, see using date math.

Write data to the alias. The index rolls over when the rollover conditions are met and the ILM check is triggered.

PUT gamestabes/_doc/1
{
    "EU_Sales" : 3.58,
    "Genre" : "Platform",
    "Global_Sales" : 40.24,
    "JP_Sales" : 6.81,
    "Name" : "Super Mario Bros.",
    "Other_Sales" : 0.77,
    "Platform" : "NES",
    "Publisher" : "Nintendo",
    "Year_of_Release" : "1985",
    "na_Sales" : 29.08
}

By default, ILM checks for indices that meet the policy criteria every 10 minutes. You can use the indices.lifecycle.poll_interval parameter to modify the polling interval.

(Optional) View the index lifecycle status in Kibana.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch area, click Index Management.
3. Click the Lifecycle phase drop-down list and select a lifecycle phase to filter indices.
4. Click the name of a filtered index to view its detailed configuration.

Step 3: Verify data distribution

After an index enters the cold phase, verify that its shards are distributed on warm nodes.

In the Kibana console, identify the name of an index that has entered the cold phase.
Run the following command to query the shard distribution of the index. Replace shrink-gamestables-000012 with the actual index name.
```
GET _cat/shards/shrink-gamestables-000012
```
In the returned result, if the box_type of the node that holds the shard is warm, it indicates that the index data for the cold phase has been distributed to the warm nodes.

Step 4: Update an ILM policy

Run the following command to update an existing ILM policy. The following example modifies the min_age for the delete phase of the game-policy:

PUT /_ilm/policy/game-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "1GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "3h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

View the version of the updated policy.
1. In the left-side navigation pane, click Management.
2. In the Elasticsearch area, click Index Lifecycle Policies.
3. View the policy version number. After the update, the version number is incremented by 1. The current write index continues to use the old policy version. The new policy takes effect at the next rollover.

Step 5: Switch ILM policies

Create a new policy.

PUT /_ilm/policy/game-new
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "3GB",
            "max_age": "1d",
            "max_docs": 1000
          }
        }
      },
      "warm": {
        "min_age": "30m",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "1h",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "2h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Bind the new policy to the template.

PUT _template/gamestabes_template
{
  "index_patterns" : ["gamestabes-*"],
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.routing.allocation.require.box_type":"hot",
    "index.lifecycle.name": "game-new",
    "index.lifecycle.rollover_alias": "gamestabes"
  }
}

FAQ

How do I adjust the ILM check interval?

By default, ILM checks for indices that meet the policy criteria every 10 minutes. During this interval, an index may grow beyond the specified threshold. For example, if you set max_docs to 1,000, the rollover might be triggered only after the actual number of documents has already surpassed 1,000.

You can control the polling interval by modifying the indices.lifecycle.poll_interval parameter:

Important

Setting the check interval too low (a high frequency) can increase the node load. Set this parameter carefully based on your business requirements.

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}