Elasticsearch:Downgrade a cluster

Notes on downgrading

• Service stability

  • The following rules apply to service stability during a cluster configuration change:

    Cluster	Service status	Countermeasures
    High payload + No replicas High payload: High concurrency for write or query operations during a downgrade, with CPU usage > 60% and heap memory usage > 50%.	Occasional access timeouts	Enable a retry mechanism on the client. Before downgrading, increase the number of index replicas to at least 1.
    Number of data nodes ≤ 2 after the downgrade.	May cause data loss.

  • Perform this operation during off-peak hours.

• Configuration constraints

  • Downgrading the storage space of data nodes is supported only for clusters that use the v3 deployment architecture.

  • You cannot upgrade the cluster version during a downgrade.

  • You can change only one type of node in a single downgrade operation.

  • You cannot downgrade elastic data nodes.

  • The interval between two consecutive downgrade operations on the same cluster must be 30 minutes or longer.

• CPU specification constraints after a downgrade

  • Basic rule: The CPU and memory of the target specifications must be at least half of the current specifications.

  • You cannot downgrade to the following specifications: 1-core 2 GiB, 2-core 2 GiB, 2-core 4 GiB, or 4-core 4 GiB. Kibana nodes can be downgraded to 2-core 2 GiB.

    Special case: To downgrade to a disallowed specification, you must create a new cluster and then perform data migration.

• Cost impact

  After you submit the downgrade order, the system bills you based on the new configuration. For more information, see Pay-as-you-go and Subscription.

Pre-downgrade checks

• Cluster health

  Run GET _cluster/health to make sure the cluster status is GREEN.

• Payload security

  The cluster can be downgraded only if it meets the following conditions:

  Node type	CPU usage	JVM heap memory usage
  Dedicated master node	Peak usage of a single node in the last 24 hours < 30%	Peak usage of a single node in the last 24 hours < 25%
  Nodes of other roles	Both conditions must be met: • Peak usage of a single node in the last 24 hours < 50% • Average usage of all nodes in the last 24 hours < 30%	Both conditions must be met: • Peak usage of a single node in the last 24 hours < 50% • Average usage of all nodes in the last 24 hours < 30%

• Index readiness

  • Run GET /_cat/indices?v to check for indexes in the CLOSE state. If you find any, run POST /<index_name>/_open to temporarily open them. If you do not open these indexes, the configuration change may fail for the following reasons:

    • If an index is in the CLOSE state, the cluster status cannot change to GREEN. Elasticsearch requires the cluster status to be GREEN before it performs certain sensitive configuration changes, such as adjusting shard allocation rules.

    • During the configuration change, the cluster reallocates shards:

      • Shards of a closed index cannot be reallocated.

      • This causes operations that depend on the GREEN status to fail.

      • This prevents the cluster status from changing to GREEN. The highest status it can reach is YELLOW.

  • Run GET _cat/indices?v to check that the number of replicas for each index is at least 1.

    For a multi-zone instance, ensure that the number of replicas for each index is less than the number of zones available for the instance. We recommend that you set the number of replicas to 1. After the downgrade is complete, you can manually increase the number of replicas.

Method 2: Downgrade a cluster by calling an API

For more information about the API for downgrading a cluster, see UpdateInstance.

Data migration and rollback

To ensure data security, the data nodes that you want to scale in must be empty. If the selected data nodes contain data, the system prompts you to migrate it. After the migration, the selected nodes no longer contain any index data, and no new index data is written to them.

Data migration

1. In the Remove Data Nodes section, click Data Migration Tool in the prompt.

   

   The Data Migration Tool uses the Elasticsearch shard allocation filtering feature to perform a smooth data migration. The data migration process is transparent to your services.

2. In the Migrate Data dialog box, select a node migration method.

   

   Parameter	Description
   Smart Migration	The system automatically selects the data nodes to be migrated.
   Custom	Manually select the data nodes to be migrated.

3. Accept the data migration agreement and click OK.

Data rollback

Data migration can be a long process. During this time, changes in the cluster status or data may cause the migration to fail. You can view the task details in the Task List. If the data migration fails, or after it is complete, you can roll back the migrated nodes by performing the following steps:

1. Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.

   For more information about how to log on to the Kibana console, see Log on to the Kibana console.

   Note

   In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.

2. In the left-side navigation pane of the page that appears, click Dev Tools.

3. In the Console, run the following command to obtain the IP addresses of the migrated nodes.

   GET _cluster/settings

   If the command is run successfully, the following result is returned.

   {
     "transient": {
       "cluster": {
         "routing": {
           "allocation": {
             "exclude": {
               "_ip": "192.168.xx.xx,192.168.xx.xx,192.168.xx.xx"
             }
           }
         }
       }
     }
   }                        

4. Run the following command to roll back the data on the migrated nodes.

   • Roll back the data on specific nodes. In the configuration, remove the IP addresses of the nodes that you want to roll back, but keep the IP addresses of the nodes that you do not want to roll back.

     PUT _cluster/settings
     {
       "transient": {
         "cluster": {
           "routing": {
             "allocation": {
               "exclude": {
                 "_ip": "192.168.xx.xx,192.168.xx.xx"
               }
             }
           }
         }
       }
     }

   • Roll back the data on all migrated nodes.

     PUT _cluster/settings
     {
       "transient": {
         "cluster": {
           "routing": {
             "allocation": {
               "exclude": {
                 "_ip": null
               }
             }
           }
         }
       }
     }                            

5. Run the following command to verify that the data rollback is complete.

   GET _cluster/settings

   If the command runs successfully and the result does not contain the IP addresses of the migrated nodes, the rollback is complete. You can also check whether shards are reallocated to the nodes.

   Note

   During data migration or rollback, you can run the GET _cat/shards?v command to view the task status.

FAQ

• The cluster configuration cannot be changed. What should I do?

• Does an ES instance support changing the disk type?

• Does changing the cluster configuration affect the ES service?

• After changing the number of nodes, does the cluster automatically rebalance the shards?

• What do I do if I select an incorrect configuration when purchasing an ES instance?

• After upgrading the instance specifications, can I downgrade the configuration, and how?

• Why can't cold data nodes be downgraded?