Use Curator

Install Curator

pip install elasticsearch-curator

curator --version

curator, version 5.6.0

Use the singleton CLI

Schedule a task by using crontab

You can use the crontab and curator commands to schedule the operations of a task.

curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

Before you run the curator command, you must specify the config.yml and action.yml files.

Separate hot and cold data

For more information, see "Hot-Warm" Architecture in Elasticsearch 5.x.

Migrate indexes from hot nodes to warm nodes

1. Create a config.yml file under the /usr/curator/ directory. Example:

   client:
     hosts:
       - http://es-cn-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
     port: 9200
     url_prefix:
     use_ssl: False
     certificate:
     client_cert:
     client_key:
     ssl_no_validate: False
     http_auth: user:password
     timeout: 30
     master_only: False
   logging:
     loglevel: INFO
     logfile:
     logformat: default
     blacklist: ['elasticsearch', 'urllib3']

   • hosts: Set the value to the internal or public endpoint of the Elasticsearch cluster. The internal endpoint is used in this example.
   • http_auth: Set the value to the username and password that are used to access the Elasticsearch cluster.
2. Create an action.yml file under the /usr/curator/ directory. You can reference the following example to configure the file:

   actions:
     1:
       action: allocation
       description: "Apply shard allocation filtering rules to the specified indices"
       options:
         key: box_type
         value: warm
         allocation_type: require
         wait_for_completion: true
         timeout_override:
         continue_if_exception: false
         disable_action: false
       filters:
       - filtertype: pattern
         kind: prefix
         value: logstash-
       - filtertype: age
         source: creation_date
         direction: older
         timestring: '%Y-%m-%dT%H:%M:%S'
         unit: minutes
         unit_count: 30

   In this example, indexes that are created on hot nodes 30 minutes ago and whose names start with logstash- are migrated to warm nodes. You can also configure the action.yml file as required.

3. Run the following command to check whether the curator command runs normally:

   curator --config /usr/curator/config.yml /usr/curator/action.yml

   If the curator command runs normally, information similar to the following code is returned:

   2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
   2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
   2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
   2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
   2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
   2019-02-12 20:12:57,925 INFO      Job completed.

4. Run the following command to enable the curator command to run at 15-minute intervals:

   */15 * * * * curator --config /usr/curator/config.yml /usr/curator/action.yml