Elasticsearch:Manage Heartbeat data with ILM

Procedure

1. Step 1: Create and configure a hot-warm cluster

   Create a hot-warm cluster, enable automatic index creation, and configure a public access IP whitelist.

2. Step 2: Configure ILM in Heartbeat

   Enable ILM and configure its parameters in the heartbeat.yml file. Heartbeat automatically generates an index template in Elasticsearch on startup.

3. Step 3: Create an ILM policy

   Create a lifecycle policy that defines index rollover and archiving conditions.

4. Step 4: Associate the ILM policy with an index template

   Associate the ILM policy with the Heartbeat index template.

5. Step 5: Apply the ILM policy to an index

   Apply the ILM policy to the initial Heartbeat index so all subsequent indices inherit it.

6. Step 6: View indices in each phase

   View indices in each lifecycle phase: hot, warm, cold, and delete.

Step 1: Create and configure a hot-warm cluster

1. Create a hot-warm cluster and verify its node attributes. The following table compares the node types.

   Node type	Data storage requirements	Read/write performance	Specifications	Storage requirements
   hot node	Recent data, such as log data from the last two days.	High	High (for example, 32-core 64 GB).	SSD cloud disks recommended.
   warm node	Historical data, such as log data older than two days.	Low	Low (for example, 8-core 32 GB).	Ultra disks recommended. OpenStore is also available for serverless cold data storage.

   In Alibaba Cloud Elasticsearch, the box_type for warm nodes is warm (not cold), matching the warm tier in native Elasticsearch.

   1. When you create an Alibaba Cloud Elasticsearch instance, enable warm nodes to create a hot-warm cluster.

      After you enable and purchase warm nodes, the system adds the -Enode.attr.box_type parameter to the node startup parameters:

      • Hot data nodes: -Enode.attr.box_type=hot

      • Warm data nodes: -Enode.attr.box_type=warm

      Data nodes function as hot nodes only after warm nodes are enabled.

   2. Log on to the Kibana console of the cluster. Connect to a cluster by using Kibana.

   3. In the left navigation pane, click Dev Tools.

   4. In the Console, run the following command to view the cluster's node attributes.

      GET _cat/nodeattrs?v&h=host,attr,value

      If the response includes both hot and warm nodes, the cluster supports the hot-warm architecture.

2. Enable automatic index creation for the target cluster. Enable automatic index creation by configuring YML parameters.

3. Add the IP address of the Heartbeat server to the cluster's public endpoint IP whitelist. Configure an IP whitelist.

Step 2: Configure ILM in Heartbeat

The Elastic guide Set up index lifecycle management covers detailed ILM configuration.

1. Download and decompress the Heartbeat installation package.

2. Edit the heartbeat.yml file to define heartbeat.monitors, setup.template.settings, setup.kibana, and output.elasticsearch.

   Sample configuration:

   heartbeat.monitors:
   - type: icmp
     schedule: '*/5 * * * * * *'
     hosts: ["47.111.xx.xx"]
   
   setup.template.settings:
     index.number_of_shards: 3
     index.codec: best_compression
     index.routing.allocation.require.box_type: "hot"
   
   setup.template.overwrite: true
   
   setup.kibana:
     host: "https://es-cn-4591jumei00xxxxxx.kibana.elasticsearch.aliyuncs.com:5601"
   
   output.elasticsearch:
     hosts: ["es-cn-4591jumei00xxxxxx.elasticsearch.aliyuncs.com:9200"]
     ilm.enabled: true
     ilm.rollover_alias: "heartbeat"
     ilm.pattern: "{now/d}-000001"
     username: "elastic"
     password: "<your_password>"

   Key parameters are listed below. The Heartbeat configuration documentation covers all available parameters.

   Parameter	Description
   index.number_of_shards	Number of primary shards. Default: 1.
   index.routing.allocation.require.box_type	Routes index data to hot nodes.
   setup.template.overwrite	Whether to overwrite the existing index template. Set to true if this version of the template is already loaded. This is a top-level parameter at the same level as setup.template.settings.
   host	Public endpoint of your Kibana service, available on the Kibana configuration page.
   hosts	Public or internal endpoint of your Elasticsearch cluster, available on the Basic Information page. View instance basic information. Public endpoints require the client IP in the cluster's IP whitelist. Configure an IP whitelist. Internal endpoints require the cluster and Heartbeat server to be in the same VPC.
   ilm.enabled	Set to true to enable ILM.
   ilm.rollover_alias	Alias for the rolled-over index. Default: heartbeat-{beat.version}.
   ilm.pattern	Pattern for the rolled-over index name. Supports date math. Default: {now/d}-000001. On rollover, the trailing number increments (for example, heartbeat-2020.04.29-000001 becomes heartbeat-2020.04.29-000002).
   username	Default: elastic.
   password	Password for the elastic user, set during instance creation. Reset the access password of an instance.

   Important

   If you modify ilm.rollover_alias or ilm.pattern after the index template is loaded, you must set setup.template.overwrite to true to rewrite the index template.

3. Start the Heartbeat service.

   sudo ./heartbeat -e

Step 3: Create an ILM policy

You can create ILM policies through the API or the Kibana console. This example uses the API to create a policy named heartbeat-policy.

Heartbeat loads a default policy when you run ./heartbeat setup --ilm-policy. Export it with ./heartbeat export ilm-policy to use as a starting point for custom policies.

Run the following command in the Kibana console to create the ILM policy:

PUT /_ilm/policy/heartbeat-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5mb",
            "max_age": "1d",
            "max_docs": 100
          }
        }
      },
      "warm": {
        "min_age": "60s",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Policy names are immutable. The Kibana console also supports policy creation, but its minimum max_age unit is hours. The API supports smaller units such as seconds.

Step 4: Associate the ILM policy with an index template

Heartbeat automatically creates an index template in Elasticsearch on startup. Associate the heartbeat-policy from Step 3: Create an ILM policy with this template.

1. Log on to the Kibana console of your Elasticsearch instance. Connect to a cluster by using Kibana.

2. In the left navigation pane, click Management.

3. In the Elasticsearch section, click Index Lifecycle Policies.

4. In the Index lifecycle policies list, find heartbeat-policy and click Actions > Add policy to index template .

5. In the dialog box, select the index template from the Index template list and enter the rollover alias in the Alias for rollover index field.

6. Click Add policy.

Step 5: Apply the ILM policy to an index

Heartbeat creates an initial index on startup. Apply the ILM policy to this index. Subsequent indices inherit the policy from the template configured in Step 4: Associate the ILM policy with an index template.

1. On the Management page, go to the Elasticsearch section and click Index Management.

2. In the Index management list, find the target index and click its name.

3. On the Summary page, click Manage > Remove lifecycle policy to remove the default policy included with Heartbeat.

4. In the confirmation dialog box, click Remove policy.

5. Then, click Manage > Add lifecycle policy.

6. In the dialog box, select heartbeat-policy from Step 3: Create an ILM policy in the Lifecycle policy list. Enter the same rollover alias from Step 4: Associate the ILM policy with an index template in the Index rollover alias field, and click Add policy.

Step 6: View indices in each phase

On the Index management page, select a phase from the Lifecycle phase drop-down list to filter indices.

FAQ

How do I adjust the ILM policy check frequency?

ILM checks for eligible indices every 10 minutes by default. During this interval, data volume may exceed the threshold. For example, in Step 3: Create an ILM policy, max_docs is set to 100, but the actual count may exceed 100 before rollover triggers.

Adjust the check frequency with the indices.lifecycle.poll_interval parameter:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}