Elasticsearch:Manage Heartbeat data using index lifecycle management

Step 2: Configure ILM in Heartbeat

To seamlessly integrate Heartbeat with Elasticsearch ILM, you must define the ILM configuration in the heartbeat.yml file. For details, see Set up index lifecycle management.

1. Download the Heartbeat installation package and decompress it.

2. Edit the heartbeat.yml file to define heartbeat.monitors, setup.template.settings, setup.kibana, and output.elasticsearch.

   The following configuration is used in this topic.

   heartbeat.monitors:
   - type: icmp
     schedule: '*/5 * * * * * *'
     hosts: ["47.111.xx.xx"]
   
   setup.template.settings:
     index.number_of_shards: 3
     index.codec: best_compression
     index.routing.allocation.require.box_type: "hot"
   
   setup.kibana:
   
     # Kibana Host
     # Scheme and port can be left out and will be set to the default (http and 5601)
     # In case you specify and additional path, the scheme is required: http://localhost:5601/path
     # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
     host: "https://es-cn-4591jumei00xxxxxx.kibana.elasticsearch.aliyuncs.com:5601"
   
   output.elasticsearch:
     # Array of hosts to connect to.
     hosts: ["es-cn-4591jumei00xxxxxx.elasticsearch.aliyuncs.com:9200"]
     ilm.enabled: true
     setup.template.overwrite: true
     ilm.rollover_alias: "heartbeat"
     ilm.pattern: "{now/d}-000001"
   
     # Enabled ilm (beta) to use index lifecycle management instead daily indices.
     #ilm.enabled: false
   
     # Optional protocol and basic auth credentials.
     #protocol: "https"
     username: "elastic"
     password: "<your_password>"

   Parameters:

   • index.number_of_shards: The number of primary shards. The default value is 1.

   • index.routing.allocation.require.box_type: Writes index data to hot nodes.

   • host: Replace this with the public endpoint of your Kibana service. Obtain the endpoint from the Kibana configuration page.

   • hosts: Replace this with the public or internal endpoint of your Elasticsearch cluster. Get the endpoint from the Basic Information page of the cluster. For more information, see View the basic information of an instance.

     Note

     If you set this to the public endpoint, configure a public network access whitelist for the cluster. For more information, see Configure a public or private access whitelist for an instance. If you set this to the internal endpoint, make sure the cluster and the server where Heartbeat is installed are in the same VPC.

   • ilm.enabled: Set to true to enable ILM.

   • setup.template.overwrite: Specifies whether to overwrite the original index template. If you have already loaded this version of the index template into Elasticsearch, set this parameter to true to overwrite the original template.

   • ilm.rollover_alias: The alias for the index generated during a rollover. The default is heartbeat-\{beat.version\}.

   • ilm.pattern: The pattern for the index generated during a rollover. Supports date math. The default is {now/d}-000001. When a rollover is triggered, the last digit of the new index name is incremented by 1. For example, the first rollover creates an index named heartbeat-2020.04.29-000001. When the rollover condition is met again, Elasticsearch creates a new index named heartbeat-2020.04.29-000002.

   • username: The default username is elastic.

   • password: The password for the elastic user is set when you create the cluster.

   For more information about other parameters, see the official Heartbeat configuration documentation.

   Important

   If you modify ilm.rollover_alias or ilm.pattern after loading the index template, set setup.template.overwrite to true to rewrite the index template.

3. Start Heartbeat.

   sudo ./heartbeat -e

Step 3: Create an ILM policy

In the Kibana console, run the following command to create an ILM policy.

PUT /_ilm/policy/heartbeat-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5mb",
            "max_age": "1d",
            "max_docs": 100
          }
        }
      },
      "warm": {
        "min_age": "60s",
        "actions": {
          "forcemerge": {
                "max_num_segments":1
              },
          "shrink": {
                "number_of_shards":1
              }
        }
      },
      "cold": {
        "min_age": "3m",
        "actions": {
          "allocate": {
            "require": {
              "box_type": "warm"
            }
          }
        }
      },
      "delete": {
        "min_age": "1h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Step 4: Apply the ILM policy to an index template

After starting Heartbeat, a Heartbeat index template is automatically created in your Elasticsearch cluster. Apply the ILM policy created in Step 3: Create an ILM policy with this template.

1. Log on to the Kibana console.

2. In the left navigation menu, click Management.

3. In the Elasticsearch area, click Index Lifecycle Policies.

4. In the Index lifecycle policies list, select Actions > Add policy to index template.

   

5. In the dialog box, select the index template from the Index template dropdown list, and enter an index alias in the Alias for rollover index text box.

   

6. Click Add policy.

Step 5: Associate an index with the ILM policy

After starting Heartbeat, an index is automatically created in your Elasticsearch cluster. You must manually link the corresponding ILM policy with an index for initialization.

1. On the Management page, in the Elasticsearch area, choose Index Management.

2. In the Index management list, find the target index and click its name.

3. On the Summary page, click Manage > Remove lifecycle policy to remove the default policy that comes with Heartbeat.

   

4. In the dialog box, click Remove policy.

5. Then, select Manage > Add lifecycle policy.

6. In the dialog box, select the lifecycle policy created in Step 3: Create an ILM policy from Lifecycle policy. In the Index rollover alias text box, enter the index alias defined in Step 4: Apply the ILM policy to an index template, and click Add policy.

   

   After the ILM policy is associated with an initial index, the result is shown in the following figure.