Install Elasticsearch Curator

  1. Purchase an Alibaba Cloud ECS instance in the same VPC network as your Alibaba Cloud Elasticsearch instance. This example uses an ECS instance that runs CentOS 7.3 64-bit.

  2. Run the following command:

    1. Install Elasticsearch Curator:
      pip install elasticsearch-curator
    2. View the version of the Curator:
      curator --version
      Returned version information:
      curator, version 5.6.0

Singleton command line interface

  • You can use curator_cli to perform an action.

  • Singleton command line interface.

    • You can perform only one action each time.
    • Not all of the actions can be performed by using Curator, for example, Alias and Restore.

Schedule tasks using Crontab

You can use the crontab and curator commands to schedule a task to perform multiple actions.

Curator command:

  --config PATH  Path to configuration file.  Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes. 
  --version      Show the version and exit. 
  --help         Show this message and exit. 

Hot-warm architecture practice

Use Curator to migrate indexes from hot nodes to warm nodes (official reference).

Migrate indexes from hot nodes to warm nodes

  1. Create the config.yml file in the /usr/curator/ path as follows:
    • hosts: Replace hosts with the address of the Alibaba Cloud Elasticsearch instance that you need to access. In this example, the private address of the Elasticsearch instance is used.
    • http_auth: Replace http_auth with the username and password that are used to log on to the Alibaba Cloud Elasticsearch instance.
      port: 9200
      use_ssl: False
      ssl_no_validate: False
      http_auth: user:password
      timeout: 30
      master_only: False
      loglevel: INFO
      logformat: default
      blacklist: ['elasticsearch', 'urllib3']
  2. Create the action.yml file in the /usr/curator/ path as follows:
    • The following content migrates indexes created 30 minutes ago and starting with logstash- from hot nodes to warm nodes.
    • You can customize the following content based on your business needs.
        action: allocation
        description: "Apply shard allocation filtering rules to the specified indices"
          key: box_type
          value: warm
          allocation_type: require
          wait_for_completion: true
          continue_if_exception: false
          disable_action: false
        - filtertype: pattern
          kind: prefix
          value: logstash-
        - filtertype: age
          source: creation_date
          direction: older
          timestring: '%Y-%m-%dT%H:%M:%S'
          unit: minutes
          unit_count: 30
  3. Check whether the curator command can run normally:
    curator --config /usr/curator/config.yml /usr/curator/action.yml
    The following information is returned when the command runs normally:
    2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation" 
    2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices 
    2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'} 
    2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed. 
    2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed. 
    2019-02-12 20:12:57,925 INFO      Job completed.
  4. Run the crontab command to run the curator command at an interval of 15 minutes:
    */15 * * * * curator --config /usr/curator/config.yml /usr/curator/action.yml