Curator is the official index management tool provided by Elasticsearch. This tool enables you to create, delete, and disable indexes, as well as merge index segments.

Overview

This topic describes how to: install Elasticsearch Curator(Curator), use the singleton command line interface, use the curator and crontab commands to schedule tasks, and use Curator to migrate indexes from hot nodes to warm nodes.

Preparations

Before installing Curator, you must purchase an Alibaba Cloud ECS instance that is in the same VPC as your Alibaba Cloud Elasticsearch cluster. This topic uses an ECS instance running 64-bit CentOS 7.3 as an example.

Install Elasticsearch Curator

To install Curator, run the following command on the CLI of the ECS instance you purchased: 
pip install elasticsearch-curator
Note
  • We recommend that you install Elasticsearch Curator 5.6.0. This version supports Alibaba Cloud Elasticsearch 5.5.3 and 6.3.2.
  • You can check the compatibility between Curator and Elasticsearch versions from the Elasticsearch official documentation.
To verify that Curator is installed, run the following command: 
curator --version
If Curator is installed, a similar result is displayed: 
curator, version 5.6.0

Singleton command line interface

You can use the curator_cli command to perform an operation. For more information, see the Elasticsearch official documentation.
Note
  • The curator_cli command allows you to perform only one operation at a time.
  • Some operations such as Alias and Restore cannot be performed on the singleton command line interface.

Schedule a task by using crontab

You can use crontab and curator commands to schedule the operations in a task.

curator command: 
curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

When you run the curator command, you must specify the config.yml and action.yml files.

Hot-warm architecture

For more information, click here.

Migrate indexes from hot nodes to warm nodes

  1. Create a config.yml file in the /usr/curator/ path. Example: 
    client:
  hosts:
    - http://es-cn-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: user:password
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']
    • hosts: Set this parameter to the endpoint of the Alibaba Cloud Elasticsearch cluster you want to access. In this example, the internal endpoint of the Elasticsearch cluster is used.
    • http_auth: Set this parameter to the username and password used to log on to the Alibaba Cloud Elasticsearch cluster you want to access.
  2. Create an action.yml file in the /usr/curator/ path. Example: 
    actions:
  1:
    action: allocation
    description: "Apply shard allocation filtering rules to the specified indices"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y-%m-%dT%H:%M:%S'
      unit: minutes
      unit_count: 30

    In this example, indexes that are created on hot nodes 30 minutes ago and start with logstash- are migrated to warm nodes. You can also customize an action.yml file based on your business needs.

  3. Check whether the curator command runs normally. 
    curator --config /usr/curator/config.yml /usr/curator/action.yml
    If the curator command runs normally, information similar to the following is returned: 
    2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
2019-02-12 20:12:57,925 INFO      Job completed.
  4. Enable the curator command to run at 15-minute intervals. 
    */15 * * * * curator --config /usr/curator/config.yml /usr/curator/action.yml