Time series data increase over time. You can use the index lifecycle management (ILM) feature to periodically roll over the data to new indexes. This ensures high query efficiency and reduces query costs. As indexes age and fewer queries are required, you can migrate the indexes to a less expensive disk and reduce the numbers of primary and replica shards. This topic describes how to use ILM to manage Heartbeat indexes.
Background information
| Phase | Description |
|---|---|
| hot | Time series data is written in real time. You can call the rollover operation to roll over the data in existing indexes to new ones based on the number of documents, volume of data, and duration of these indexes. |
| warm | Data is no longer written to indexes. You can only query data from them. |
| cold | Indexes are no longer updated. Few queries are performed on these indexes, and the query process slows down. |
| delete | Data is deleted. |
Test scenario:
A large number of time series indexes whose names start with heartbeat- exist in your Elasticsearch cluster, and the size of a single index is about 4 MB each day. The number of shards increases with the data volume. This may cause cluster overload. In this case, you must configure different rollover policies for the four phases. In the hot phase, data in historical monitoring indexes whose names start with heartbeat- is rolled over to new indexes. In the warm phase, indexes are shrunk, and segments in each index are merged. In the cold phase, data is migrated from hot nodes to warm nodes. In the delete phase, data is deleted on a regular basis.
Procedure
Preparations
Step 1: Enable and configure the ILM feature in the heartbeat.yml file
To manage Heartbeat indexes by using the ILM feature of Elasticsearch, you can configure the feature in the heartbeat.yml file. For more information, see Set up index lifecycle management.
Step 2: Create an ILM policy
./heartbeat setup --ilm-policy command to load the default policy and write it to Elasticsearch. You can run the
./heartbeat export ilm-policy command to export the default policy to stdout. Then, modify the default policy to manually create an ILM policy.
Log on to the Kibana console of your Elasticsearch cluster and run the following command:
PUT /_ilm/policy/hearbeat-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "5mb",
"max_age": "1d",
"max_docs": 100
}
}
},
"warm": {
"min_age": "60s",
"actions": {
"forcemerge": {
"max_num_segments":1
},
"shrink": {
"number_of_shards":1
}
}
},
"cold": {
"min_age": "3m",
"actions": {
"allocate": {
"require": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "1h",
"actions": {
"delete": {}
}
}
}
}
}| Parameter | Description |
|---|---|
| hot | A rollover is triggered if an index associated with the ILM policy meets one of the following conditions:
Notice If the value of max_docs, max_size, or max_age is reached during a
rollover, Elasticsearch archives the index.
|
| warm | After the index enters the warm phase, the system shrinks it down to a new index that has only one primary shard and merges segments in the index into one segment. The index enters the cold phase three minutes after the rollover starts. |
| cold | The system migrates the index from a hot node to a warm node. The index enters the delete phase one hour later. |
| delete | The index is deleted one hour later. |
Step 3: Associate the ILM policy with an index template
After you start Heartbeat, the system automatically creates a Heartbeat index template in your Elasticsearch cluster. You must associate the ILM policy created in Step 2: Create an ILM policy with this index template.
- In the Index lifecycle policies section, find the ILM policy you created, click Actions, and select Add policy to index template.
- In the dialog box that appears, select heartbeat from the Index template drop-down list and specify Alias for rollover index.
Step 4: Associate the ILM policy with an index
After you start Heartbeat, the system automatically creates Heartbeat indexes in your Elasticsearch cluster. You must associate the first index with the ILM policy that is associated with the index template you created. For more information, see Step 3: Associate the ILM policy with an index template.
- On the Summary tab of the pane that appears, click Manage and select Remove lifecycle policy to remove the default policy of Heartbeat.
- In the dialog box that appears, select the ILM policy you created (heartbeat-policy) from the Lifecycle policy drop-down list and set Index rollover alias to heartbeat. Then, click Add policy.
If the association is successful, the information shown in the following figure appears.
Step 5: View indexes in different phases
You can use this method to view indexes in other phases.
Appendix: Set the interval to execute an ILM policy
indices.lifecycle.poll_interval parameter to control the check interval. This ensures that data in indexes is rolled over in a timely manner.
1m.
PUT _cluster/settings
{
"transient": {
"indices.lifecycle.poll_interval":"1m"
}
}Additional information
- An ILM policy can be added for an index only after an index template and an alias are configured for the index.
- You can use the methods described in the following table to add an ILM policy.
Method Description Add an ILM policy to an index template The added ILM policy applies to all the indexes that are created by using the index template. Add an ILM policy to a single index The added ILM policy applies only to the index. - If you modify an ILM policy during a rollover, the new policy takes effect from the next rollover.