Elasticsearch:Manage indexes with Curator

Install Curator

Prerequisites

• An Alibaba Cloud Elasticsearch cluster is created.

• An Alibaba Cloud Elastic Compute Service (ECS) instance is created.

  Ensure it resides in the same VPC as your Elasticsearch cluster.

• A Python environment is prepared.

Procedure

1. Connect to the ECS instance.

   Note

   In this example, a common user is used.

2. Run the following command to install Curator:

   sudo pip install elasticsearch-curator

   Note

   • The following table shows the version compatibility between Curator and Alibaba Cloud Elasticsearch:

     Curator version	Supported Elasticsearch versions
     5.6.x	5.x, 6.x
     5.7+	5.x, 6.x, 7.x

     Curator does not support Elasticsearch 8.x. For Elasticsearch 8.x clusters, use the built-in Index Lifecycle Management (ILM) feature instead. For more information, see Index lifecycle management.

     For more information, see Version Compatibility.

   • For more information about Curator, see Curator Reference.

3. After Curator is installed, run the following command to check its version:

   sudo curator --version

   If the command is successfully run, the following result is returned:

   curator, version 5.6.0

Use the singleton CLI

The singleton CLI (curator_cli) performs a single action without requiring configuration files. This is useful for quick, one-off operations.

The following code shows the basic syntax:

curator_cli [OPTIONS] COMMAND [ARGS]

The following examples show common operations:

• Show all indexes:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password show-indices --verbose

• Delete indexes older than 30 days:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password delete-indices --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":30}]'

• Close indexes matching a pattern:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password close --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"logstash-"}]'

Schedule a task by using the cron expression

Use cron expressions and curator commands to schedule recurring index management tasks.

The following code shows the curator command syntax:

curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

Before running the curator command, create two configuration files:

• curator.yml: Defines Elasticsearch cluster connection settings.

• action.yml: Defines the actions to perform and the filters to apply.

The following table shows common cron expression examples:

Migrate indexes from hot nodes to warm nodes

In the hot-warm architecture, data is stored on different node types based on access frequency:

• Hot nodes: Store frequently accessed, recently indexed data.

• Warm nodes: Store less frequently accessed, older data.

Curator automates the migration of indexes from hot nodes to warm nodes based on index age. This helps optimize storage costs while maintaining query performance for recent data. For more information about the hot-warm architecture, see "Hot-Warm" Architecture in Elasticsearch 5.x.

1. Create a .curator.yml file under the /usr/curator/ directory. Example:

   client:
     hosts:
       - http://es-sg-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
     port: 9200
     url_prefix:
     use_ssl: False
     certificate:
     client_cert:
     client_key:
     ssl_no_validate: False
     username: elastic
     password: password
     timeout: 30
     master_only: False
   logging:
     loglevel: INFO
     logfile:
     logformat: default
     blacklist: ['elasticsearch', 'urllib3']

   • hosts: Set the value to the internal or public endpoint of the Elasticsearch cluster. The internal endpoint is used in this example.

   • username and password: Set to the credentials used to access the Elasticsearch cluster.

2. Create an action.yml file under the /usr/curator/ directory. Example:

   actions:
     1:
       action: allocation
       description: "Apply shard allocation filtering rules to the specified indices"
       options:
         key: box_type
         value: warm
         allocation_type: require
         wait_for_completion: true
         timeout_override:
         continue_if_exception: false
         disable_action: false
       filters:
       - filtertype: pattern
         kind: prefix
         value: logstash-
       - filtertype: age
         source: creation_date
         direction: older
         timestring: '%Y-%m-%dT%H:%M:%S'
         unit: minutes
         unit_count: 30

   In this example, indexes that are created on hot nodes 30 minutes ago and whose names start with logstash- are migrated to warm nodes.

   The following table describes the key parameters in the action.yml file:

   Parameter	Description
   action	The action type. Set to allocation to migrate indexes between node types.
   key	The node attribute key. Set to box_type for hot-warm migration.
   value	The target node type. Set to warm to migrate indexes to warm nodes.
   filtertype: pattern	Filters indexes by name pattern. The prefix kind matches indexes starting with the specified value.
   filtertype: age	Filters indexes by age. The unit and unit_count parameters define the time threshold.

3. Run the following command to check whether the curator command runs normally:

   sudo curator --config /usr/curator/.curator.yml /usr/curator/action.yml

   If the curator command runs normally, information similar to the following code is returned:

   2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
   2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
   2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
   2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
   2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
   2019-02-12 20:12:57,925 INFO      Job completed.

4. Run the following command to enable the curator command to run at 15-minute intervals:

   crontab -e
   */15 * * * * curator --config /usr/curator/.curator.yml /usr/curator/action.yml