This topic describes how to use the index lifecycle management (ILM) feature to separate hot data from cold data in an Alibaba Cloud Elasticsearch cluster. The separation enables you to implement the hot-warm architecture. This architecture improves the read/write performance of the cluster, automates the maintenance of hot and cold data, and reduces your production costs.
Background information
- Write data to the indexes of an Elasticsearch cluster in real time. When the data volume in the cluster reaches a specific level, the system automatically rolls over data to new indexes.
- The new indexes stay in the hot phase for 30 minutes and enter the warm phase.
- In the warm phase, the system shrinks the new indexes and merges the segments in the indexes. The indexes stay in the warm phase for 30 minutes and enter the cold phase.
- In the cold phase, data is migrated from hot nodes to warm nodes to separate hot data from cold data. The indexes are deleted one hour later.
Recommended configurations
- You must configure ILM policies based on your business model. For example, we recommend that you configure different aliases and ILM policies for indexes with different structures. This facilitates index management.
- The name of an initial index must end with an auto-increment six-digit number, such as -000001. Otherwise, ILM policies cannot take effect. For example, an initial index is named myindex-000001. After a rollover, a new index named myindex-000002 is generated. If the names of your indexes do not meet the preceding requirements, we recommend that you reindex your data.
- In the hot phase, the system writes data. To ensure that data is written in chronological
order, we recommend that you do not write data to indexes in the warm or cold phase.
For example, for the warm phase, set
actions
toshrink
orread only
. This way, indexes are read only after they enter the warm phase.Note For more information about each lifecycle phase, see Use ILM to manage Heartbeat indexes. - You configure more vCPUs and use disks with higher I/O performance for hot nodes to process hot data. You configure more disk space for warm nodes to store cold data. Warm nodes can still provide services even if you configure fewer vCPUs and use disks with lower I/O performance for them.
Configure an ILM policy for indexes
Verify data distribution
Update the ILM policy
Switch the ILM policy
Summary
This topic provides instructions on how to separate hot data from cold data by using
ILM.
- Configure an ILM policy for indexes.
Procedure:
- Configure hot and warm attributes.
- Configure an index template based on your needs.
- Configure an ILM policy based on your needs and associate the policy with the index template.
- Create an initial index whose name ends with -000001. The name of the index generated after a rollover is automatically incremented by one.
- Verify data distribution.
Check whether the shards of indexes in the cold phase are distributed on warm nodes.
- Update the ILM policy.
Update the ILM policy.
- Switch the ILM policy.
Switch the ILM policy.