Elasticsearch:Manage indexes with Curator

Install Curator

Prerequisites

• An Alibaba Cloud Elasticsearch cluster is created.

• An Alibaba Cloud Elastic Compute Service (ECS) instance is created.

  Ensure it resides in the same VPC as your Elasticsearch cluster.

• A Python environment is prepared.

Procedure

1. Connect to the ECS instance.

   Note

   This example uses a regular user.

2. Install Curator:

   sudo pip install elasticsearch-curator

   Note

   • Version compatibility between Curator and Alibaba Cloud Elasticsearch:

     Curator version	Supported Elasticsearch versions
     5.6.x	5.x, 6.x
     5.7+	5.x, 6.x, 7.x

     Curator does not support Elasticsearch 8.x. For 8.x clusters, use the built-in Index Lifecycle Management (ILM) feature instead.

     Version Compatibility.

   • Curator Reference.

3. Verify the installation:

   sudo curator --version

   Expected output:

   curator, version 5.6.0

Use the singleton CLI

The singleton CLI (curator_cli) runs a single action without configuration files, suited for one-off operations.

Basic syntax:

curator_cli [OPTIONS] COMMAND [ARGS]

Common operations:

• Show all indexes:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password show-indices --verbose

• Delete indexes older than 30 days:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password delete-indices --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":30}]'

• Close indexes matching a pattern:

  curator_cli --host es-sg-xxxxx.elasticsearch.aliyuncs.com --port 9200 --username elastic --password password close --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"logstash-"}]'

Schedule tasks with cron

Schedule recurring index management tasks with cron expressions.

Syntax:

curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

Create two configuration files before running the curator command:

• curator.yml: Defines Elasticsearch cluster connection settings.

• action.yml: Defines the actions to perform and the filters to apply.

Common cron expressions:

Migrate indexes from hot nodes to warm nodes

In the hot-warm architecture, data is stored on different node types based on access frequency:

• Hot nodes: Store frequently accessed, recently indexed data.

• Warm nodes: Store less frequently accessed, older data.

Curator automates index migration from hot to warm nodes based on index age, optimizing storage costs while maintaining query performance for recent data. Learn more about the "Hot-Warm" Architecture in Elasticsearch 5.x.

1. Create a .curator.yml file under the /usr/curator/ directory. Example:

   client:
     hosts:
       - http://es-sg-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
     port: 9200
     url_prefix:
     use_ssl: False
     certificate:
     client_cert:
     client_key:
     ssl_no_validate: False
     username: elastic
     password: password
     timeout: 30
     master_only: False
   logging:
     loglevel: INFO
     logfile:
     logformat: default
     blacklist: ['elasticsearch', 'urllib3']

   • hosts: The internal or public endpoint of your Elasticsearch cluster. This example uses the internal endpoint.

   • username and password: The credentials for your Elasticsearch cluster.

2. Create an action.yml file under the /usr/curator/ directory. Example:

   actions:
     1:
       action: allocation
       description: "Apply shard allocation filtering rules to the specified indices"
       options:
         key: box_type
         value: warm
         allocation_type: require
         wait_for_completion: true
         timeout_override:
         continue_if_exception: false
         disable_action: false
       filters:
       - filtertype: pattern
         kind: prefix
         value: logstash-
       - filtertype: age
         source: creation_date
         direction: older
         timestring: '%Y-%m-%dT%H:%M:%S'
         unit: minutes
         unit_count: 30

   This example migrates indexes prefixed with logstash- that were created on hot nodes more than 30 minutes ago to warm nodes.

   Key parameters:

   Parameter	Description
   action	The action type. Set to allocation to migrate indexes between node types.
   key	The node attribute key. Set to box_type for hot-warm migration.
   value	The target node type. Set to warm to migrate indexes to warm nodes.
   filtertype: pattern	Filters indexes by name pattern. The prefix kind matches indexes starting with the specified value.
   filtertype: age	Filters indexes by age. The unit and unit_count parameters define the time threshold.

3. Test the curator configuration:

   sudo curator --config /usr/curator/.curator.yml /usr/curator/action.yml

   Expected output:

   2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
   2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
   2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
   2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
   2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
   2019-02-12 20:12:57,925 INFO      Job completed.

4. Schedule the command to run every 15 minutes:

   crontab -e
   */15 * * * * curator --config /usr/curator/.curator.yml /usr/curator/action.yml