Curator is an index management tool provided by open-source Elasticsearch. This tool allows you to create, delete, and disable indexes. It also allows you to merge index segments. This topic describes how to install Curator, use the singleton command line interface (CLI), schedule a task by using crontab, separate hot and cold data, and migrate indexes from hot nodes to warm nodes.

Install Curator

Before you install Curator, make sure that you have completed the following preparations:
Connect to the ECS instance and run the following command to install Curator: 
pip install elasticsearch-curator
Note We recommend that you install Curator 5.6.0 that is compatible with Alibaba Cloud Elasticsearch V5.5.3 and V6.3.2. For more information about the compatibility between Curator and Alibaba Cloud Elasticsearch, see Version Compatibility.
Run the following command to check the version of Curator: 
curator --version
If the command is successfully executed, the following result is returned: 
curator, version 5.6.0
Note For more information about Curator, see Curator Index Management.

Use the singleton CLI

You can run the curator_cli command to perform a single action. For more information, see Singleton Command Line Interface.
Note
  • The curator_cli command allows you to perform only one action at a time.
  • Some actions such as Alias and Restore cannot be performed on the singleton CLI.

Schedule a task by using crontab

You can use crontab and curator commands to schedule the actions in a task.

The following code provides an example of the curator command: 
curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

When you run the curator command, you must specify the config.yml and action.yml files.

Separate hot and cold data

For more information, see "Hot-Warm" Architecture in Elasticsearch 5.x.

Migrate indexes from hot nodes to warm nodes

  1. Create a config.yml file in the /usr/curator/ directory. Example: 
    client:
  hosts:
    - http://es-cn-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: user:password
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']
    • hosts: Set the value to the internal or public endpoint of the Elasticsearch cluster. The internal endpoint is used in this example.
    • http_auth: Set the value to the username and password that are used to access the Elasticsearch cluster.
  2. Create an action.yml file in the /usr/curator/ directory. Example: 
    actions:
  1:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y-%m-%dT%H:%M:%S'
      unit: minutes
      unit_count: 30

    In this example, indexes that are created on hot nodes 30 minutes ago and start with logstash- are migrated to warm nodes. You can also configure an action.yml file as required.

  3. Check whether the curator command runs normally. 
    curator --config /usr/curator/config.yml /usr/curator/action.yml
    If the curator command runs normally, information similar to the following code is returned: 
    2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
2019-02-12 20:12:57,925 INFO      Job completed.
  4. Enable the curator command to run at 15-minute intervals. 
    */15 * * * * curator --config /usr/curator/config.yml /usr/curator/action.yml