Elasticsearch:Manage Heartbeat data with ILM

Procedure

1. Step 1: Create and configure a hot-warm cluster

   Create a hot-warm cluster, enable automatic index creation, and configure an access whitelist for the public endpoint.

2. Step 2: Configure ILM in Heartbeat

   Enable and configure ILM parameters in the heartbeat.yml file. After you start Heartbeat, a Heartbeat index template is automatically generated in Elasticsearch.

3. Step 3: Create an ILM policy

   Use the ILM policy API to create a lifecycle management policy that defines the conditions for index rollover and archiving.

4. Step 4: Link ILM policy to an index template

   Associate the ILM policy with the Heartbeat index template.

5. Step 5: Link an index to the ILM policy

   Associate the ILM policy with the first Heartbeat index to apply the policy to all indices covered by the index template.

6. Step 6: View indices in each phase

   View indices archived in each phase: hot, warm, cold, and delete.

Step 1: Create and configure a hot-warm cluster

1. Create a hot-warm cluster and view the hot and warm attributes of its nodes. The differences between node types are as follows:

   Node type	Data storage requirements	Read/write performance	Specifications	Storage requirements
   hot node	Recent data, such as log data from the last 2 days.	High	High, such as 32-core 64 GB	SSD cloud disk is recommended.
   warm node	Historical data, such as log data older than 2 days.	Low	Low, such as 8-core 32 GB	Efficient cloud disks are recommended. You can also use OpenStore for serverless storage of large volumes of data.

   In Alibaba Cloud Elasticsearch, the box_type value for a warm node is warm, not cold. This is because this node type corresponds to the warm tier in the native Elasticsearch architecture.

   1. When you create an Alibaba Cloud Elasticsearch instance, enable warm nodes to create a hot-warm cluster.

      After you enable and purchase warm nodes, the system adds the -Enode.attr.box_type parameter to the node startup parameters:

      • Hot data node: -Enode.attr.box_type=hot

      • Warm data node: -Enode.attr.box_type=warm

      Data nodes become hot nodes only after you enable warm nodes.

   2. Log on to the Kibana console for the cluster. For more information, see Connect to a cluster by using Kibana.

   3. In the left-side navigation pane, click Dev Tools.

   4. In the Console, run the following command to view the hot and warm attributes of the nodes.

      GET _cat/nodeattrs?v&h=host,attr,value

      If the result contains both hot and warm nodes, the cluster supports a hot-warm architecture.

2. Enable automatic index creation for the cluster. For more information, see Enable automatic index creation by configuring YML parameters.

3. Configure an access whitelist for the public endpoint of the cluster, and add the IP address of the Heartbeat server to the whitelist. For more information, see Configure an IP address whitelist.

Step 2: Configure ILM in Heartbeat

For detailed ILM configuration instructions, see Set up index lifecycle management.

1. Download and decompress the Heartbeat installation package.

2. In the heartbeat.yml file, define the heartbeat.monitors, setup.template.settings, setup.kibana, and output.elasticsearch settings.

   The following example shows a sample configuration.

   heartbeat.monitors:
   - type: icmp
     schedule: '*/5 * * * * * *'
     hosts: ["47.111.xx.xx"]
   
   setup.template.settings:
     index.number_of_shards: 3
     index.codec: best_compression
     index.routing.allocation.require.box_type: "hot"
   
   setup.template.overwrite: true
   
   setup.kibana:
     host: "https://es-cn-4591jumei00xxxxxx.kibana.elasticsearch.aliyuncs.com:5601"
   
   output.elasticsearch:
     hosts: ["es-cn-4591jumei00xxxxxx.elasticsearch.aliyuncs.com:9200"]
     ilm.enabled: true
     ilm.rollover_alias: "heartbeat"
     ilm.pattern: "{now/d}-000001"
     username: "elastic"
     password: "<your_password>"

   The following table describes some of the parameters. For more information, see the Heartbeat configuration documentation.

   Parameter	Description
   index.number_of_shards	The number of primary shards. The default value is 1.
   index.routing.allocation.require.box_type	Specifies that index data is written to hot nodes.
   setup.template.overwrite	Specifies whether to overwrite the existing index template. If you have already loaded an index template of this version into Elasticsearch, you must set this parameter to true to overwrite the template. This is a top-level configuration item at the same level as setup.template.settings.
   host	Replace the value with your Kibana service's public endpoint, which is available on the Kibana configuration page.
   hosts	Replace the value with your Elasticsearch cluster's public or private endpoint, available on the cluster's basic information page. For more information, see View the basic information of an instance. If you use a public endpoint, you must configure an access whitelist for the cluster. For more information, see Configure an IP address whitelist. If you use a private endpoint, make sure that the cluster and the Heartbeat server are in the same VPC.
   ilm.enabled	Set this parameter to true to enable index lifecycle management (ILM).
   ilm.rollover_alias	The alias for the index that is generated during a rollover. The default value is heartbeat-\{beat.version\}.
   ilm.pattern	The pattern for the index that is generated during a rollover. This parameter supports date math. The default value is {now/d}-000001. When a rollover is triggered, the last digit of the new index name is incremented by 1. For example, if the first rollover creates an index named heartbeat-2020.04.29-000001, the next rollover creates an index named heartbeat-2020.04.29-000002.
   username	The default username is elastic.
   password	You set the password for the elastic user when creating the instance. If you forget the password, reset it by following the instructions in Reset the access password of an instance.

   Important

   If you modify ilm.rollover_alias or ilm.pattern after the index template is loaded, you must set setup.template.overwrite to true to overwrite the index template.

3. Start the Heartbeat service.

   sudo ./heartbeat -e

Step 3: Create an ILM policy

You can create an ILM policy by using the API or in the Kibana console. This section explains how to create the heartbeat-policy policy with the API.

Heartbeat supports loading a default policy into Elasticsearch with the ./heartbeat setup --ilm-policy command. The default policy can be exported with the ./heartbeat export ilm-policy command. You can modify the default policy to manually create a policy.

In the Kibana console, run the following command to create the ILM policy.

PUT /_ilm/policy/heartbeat-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5mb",
            "max_age": "1d",
            "max_docs": 100
          }
        }
      },
      "warm": {
        "min_age": "60s",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The policy name cannot be changed after creation. You can also create a policy in the Kibana console. However, Kibana sets max_age in hours, while the API supports seconds.

Step 4: Link ILM policy to an index template

After you start Heartbeat, a Heartbeat index template is automatically created in Elasticsearch. Associate the heartbeat-policy policy created in Step 3: Create an ILM policy with this index template.

1. Log on to the Kibana console for your Alibaba Cloud Elasticsearch instance. For more information, see Connect to a cluster by using the Kibana console.

2. In the left-side navigation pane, click Management.

3. In the Elasticsearch section, click Index Lifecycle Policies.

4. In the Index lifecycle policies list, find the heartbeat-policy policy, and then click Actions > Add policy to index template.

5. In the dialog box that appears, select the index template from the Index template list and enter an index alias in the Alias for rollover index text box.

6. Click Add policy.

Step 5: Link an index to the ILM policy

After starting Heartbeat, an index is automatically created in Elasticsearch. Associate the first index with the ILM policy you associated with the index template in Step 4: Link ILM policy to an index template.

1. On the Management page, in the Elasticsearch section, click Index Management.

2. In the Index management list, find the target index and click the index name.

3. On the Summary page, select Manage > Remove lifecycle policy to remove the default Heartbeat policy.

4. In the dialog box that appears, click Remove policy.

5. Then, select Manage > Add lifecycle policy.

6. In the dialog box that appears, select the heartbeat-policy policy that you created in Step 3: Create an ILM policy from the Lifecycle policy list. In the Index rollover alias text box, enter the index alias that you defined in Step 4: Link ILM policy to an index template, and then click Add policy.

Step 6: View indices in each phase

On the Index management page, click the Lifecycle phase drop-down list and select a phase, such as Hot, Warm, or Cold, to filter and view the indices in that phase.

FAQ

How do I adjust the ILM policy check frequency?

By default, ILM checks for policy-matching indices every 10 minutes. This means the data volume can exceed the specified threshold before a check occurs. For example, in Step 3: Create an ILM policy, max_docs is set to 100, but a rollover might be triggered only after the document count exceeds 100.

You can control the check frequency by modifying the indices.lifecycle.poll_interval parameter:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}