Elasticsearch:Manage Heartbeat data using ILM

Procedure

1. Step 1: Create and configure a hot-warm cluster

   Create a cluster that contains hot and warm nodes, enable automatic index creation, and configure an IP whitelist for public access.

2. Step 2: Configure ILM in Heartbeat

   Enable the ILM feature and configure its parameters in the heartbeat.yml file. After you start Heartbeat, Elasticsearch automatically generates a Heartbeat index template.

3. Step 3: Create an ILM policy

   Use the ILM policy API to create a policy that defines conditions for index rollover and archiving.

4. Step 4: Associate the ILM policy with an index template

   Associate the ILM policy with the Heartbeat index template.

5. Step 5: Associate an index with the ILM policy

   Associate the ILM policy with the first Heartbeat index to apply the policy to all indices that are covered by the index template.

6. Step 6: View indices in each phase

   View the indices archived in each phase: hot, warm, cold, and delete.

Step 1: Create and configure a hot-warm cluster

1. Create a hot-warm cluster, which contains both hot and warm nodes, and view their attributes. The following table describes the differences.

   Node type	Data requirements	Read/write performance	Specifications	Storage
   hot node	Recent data, such as log data from the last two days.	High	High, such as 32-core 64 GB.	SSD cloud disks are recommended.
   warm node	Historical data, such as log data older than two days.	Low	Low, such as 8-core 32 GB.	Ultra disks are recommended. You can also use OpenStore to implement serverless storage for large amounts of cold data.

   In Alibaba Cloud Elasticsearch, the box_type value for a warm node is warm, not cold. This is because warm nodes in Alibaba Cloud Elasticsearch correspond to the warm tier in the native Elasticsearch architecture.

   1. When you create an Alibaba Cloud Elasticsearch instance, enable Warm nodes to create a hot-warm cluster.

      After you enable and purchase warm nodes, the system adds the -Enode.attr.box_type parameter to the node startup parameters:

      • Hot node: -Enode.attr.box_type=hot

      • Warm node: -Enode.attr.box_type=warm

      Enabling warm nodes designates your existing data nodes as hot nodes.

   2. Log on to the Kibana console of the cluster. For more information, see Connect to a cluster by using Kibana.

   3. In the left-side navigation pane, click Dev Tools.

   4. In the Console, run the following command to view the attributes of the hot and warm nodes in the cluster.

      GET _cat/nodeattrs?v&h=host,attr,value

      If the output contains hot and warm nodes, the cluster supports the hot-warm architecture.

2. Enable automatic index creation for the cluster. For more information, see Enable automatic index creation by configuring YML parameters.

3. Configure an IP whitelist for public access to the cluster and add the IP address of the server where Heartbeat is installed to the whitelist. For more information, see Configure an IP whitelist.

Step 2: Configure ILM in Heartbeat

For more information about how to configure ILM, see Set up index lifecycle management.

1. Download and decompress the Heartbeat installation package.

2. Edit the heartbeat.yml file to define the heartbeat.monitors, setup.template.settings, setup.kibana, and output.elasticsearch sections.

   The following example shows a sample configuration.

   heartbeat.monitors:
   - type: icmp
     schedule: '*/5 * * * * * *'
     hosts: ["47.111.xx.xx"]
   
   setup.template.settings:
     index.number_of_shards: 3
     index.codec: best_compression
     index.routing.allocation.require.box_type: "hot"
   
   setup.template.overwrite: true
   
   setup.kibana:
     host: "https://es-cn-4591jumei00xxxxxx.kibana.elasticsearch.aliyuncs.com:5601"
   
   output.elasticsearch:
     hosts: ["es-cn-4591jumei00xxxxxx.elasticsearch.aliyuncs.com:9200"]
     ilm.enabled: true
     ilm.rollover_alias: "heartbeat"
     ilm.pattern: "{now/d}-000001"
     username: "elastic"
     password: "<your_password>"

   The following table describes some of the parameters. For more information, see the Heartbeat configuration documentation.

   Parameter	Description
   index.number_of_shards	The number of primary shards. The default is 1.
   index.routing.allocation.require.box_type	Specifies that index data is written to hot nodes.
   setup.template.overwrite	Specifies whether to overwrite the original index template. If you have already loaded this version of the index template into Elasticsearch, you must set this parameter to true to overwrite the original template. This is a top-level configuration item at the same level as setup.template.settings.
   host	Replace the value with the public endpoint of your Kibana service. You can obtain the endpoint from the Kibana configuration page.
   hosts	Replace the value with the public or private endpoint of your Elasticsearch cluster. You can obtain the endpoint from the Basic Information page of the cluster. For more information, see View the basic information of an instance. If you set this parameter to the public endpoint, you must configure an IP whitelist for public access to the cluster. For more information, see Configure an IP whitelist. If you set this parameter to the private endpoint, ensure the cluster and the server hosting Heartbeat are in the same VPC.
   ilm.enabled	Set this parameter to true to enable ILM.
   ilm.rollover_alias	The alias for the index that is generated during a rollover. The default value is heartbeat-{beat.version}.
   ilm.pattern	The pattern for the index that is generated during a rollover. This parameter supports date math. The default value is {now/d}-000001. When a rollover is triggered, the numeric counter in the index name increments by 1. For example, if the first rollover generates an index named heartbeat-2020.04.29-000001, the next rollover generates a new index named heartbeat-2020.04.29-000002.
   username	The default username is elastic.
   password	The password for the elastic user. The password is set when you create the instance. If you forget the password, you can reset it. For more information, see Reset the access password of an instance.

   Important

   If you modify ilm.rollover_alias or ilm.pattern after you load the index template, you must set setup.template.overwrite to true to overwrite the template.

3. Start the Heartbeat service.

   sudo ./heartbeat -e

Step 3: Create an ILM policy

You can create an ILM policy in Elasticsearch by using the API or the Kibana console. The following example shows how to create the heartbeat-policy policy by using the API.

Heartbeat can load and write a default policy to Elasticsearch with the ./heartbeat setup --ilm-policy command. To create a custom policy, export the default policy with ./heartbeat export ilm-policy and then modify it.

In the Kibana console, run the following command to create an ILM policy.

PUT /_ilm/policy/heartbeat-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5mb",
            "max_age": "1d",
            "max_docs": 100
          }
        }
      },
      "warm": {
        "min_age": "60s",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

The policy name cannot be changed after the policy is created. You can also create a policy in the Kibana console. However, the minimum unit for max_age in Kibana is hours. If you use the API, you can specify the minimum unit in seconds.

Step 4: Associate the ILM policy to an index template

After you start Heartbeat, a Heartbeat index template is automatically created in Elasticsearch. Associate the heartbeat-policy policy that you created in Step 3: Create an ILM policy with this index template.

1. Log on to the Kibana console of your Alibaba Cloud Elasticsearch instance. For more information, see Connect to a cluster by using Kibana.

2. In the left-side navigation pane, click Management.

3. In the Elasticsearch area, click Index Lifecycle Policies.

4. In the Index lifecycle policies list, find the heartbeat-policy policy and choose Actions > Add policy to index template in the policy row.

5. In the dialog box that appears, select the index template from the Index template list and enter an index alias in the Alias for rollover index text box.

6. Click Add policy.

Step 5: Associate an index with the ILM policy

After you start Heartbeat, Elasticsearch automatically creates an index. You must manually associate the ILM policy with this first index to override its default policy. This is necessary even though the policy is already associated with the index template as described in Step 4: Associate the ILM policy to an index template.

1. On the Management page, in the Elasticsearch area, click Index Management.

2. In the Index management list, find the target index and click the index name.

3. On the Summary page, choose Manage > Remove lifecycle policy to remove the default policy that comes with Heartbeat.

4. In the dialog box that appears, click Remove policy.

5. Then, choose Manage > Add lifecycle policy.

6. In the dialog box that appears, select the heartbeat-policy policy that you created in Step 3: Create an ILM policy from the Lifecycle policy list. In the Index rollover alias text box, enter the index alias that you defined in Step 4: Associate the ILM policy to an index template. Then, click Add policy.

Step 6: View indices in each phase

On the Index management page, click the Lifecycle phase drop-down list and select a lifecycle phase, such as Hot, Warm, or Cold, to filter and view the indices in that phase.

FAQ

How do I adjust the ILM policy check frequency?

By default, ILM checks for indices that match a policy every 10 minutes. During this interval, the amount of data can exceed the specified threshold. For example, in Step 3: Create an ILM policy, max_docs is set to 100, but the rollover might not be triggered until the number of documents already exceeds 100.

You can control the check frequency by modifying the indices.lifecycle.poll_interval parameter:

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval":"1m"
  }
}