downgrade cluster,scale in cluster,downgrade cluster configuration,change cluster configuration,reduce cluster configuration,scale in data nodes,scale in nodes - Elasticsearch

When business traffic fluctuates between peak and off-peak hours, the resource utilization of an Alibaba Cloud Elasticsearch cluster can stay low over a long period. Downgrade the cluster to reduce node specifications, decrease the number of nodes, or change the storage type so that resources match business demand and costs are optimized.

Precautions

A downgrade can cause service delays, configuration conflicts, and billing changes. Read the following precautions before you proceed.

Service stability

The following table describes the service stability rules that apply during a cluster configuration change.

Condition	Impact	Mitigation
The cluster is under high load and has no index replicas. High load means writes or queries under high concurrency during the downgrade, with CPU usage higher than 60% or JVM heap memory usage higher than 50%.	Occasional access timeouts	Enable a retry mechanism on the client. Increase the number of index replicas to at least 1 before the downgrade.
The number of data nodes is 2 or fewer after the downgrade.	Potential data loss	Make sure that each index has at least 1 replica before the downgrade. If the downgrade removes data nodes, migrate the data off those nodes first.

The high-load values in the preceding table describe when access timeouts can occur during a configuration change. They are not the validation thresholds that the system checks when you submit a downgrade order. For the validation thresholds, see Load safety in Pre-downgrade checks.

Maintenance window — Perform the downgrade during off-peak business hours.
Node IP addresses — After the change is complete, the IP addresses of the nodes in the cluster change.

Warning

If you enable weak validation for downgrade, the system relaxes the resource validation performed before the downgrade. A downgrade can then cause insufficient resources and affect cluster stability and services. Service unavailability caused by a downgrade performed with this option enabled is not covered by the Service-Level Agreement (SLA). Weak validation is disabled by default and is not recommended. Fully evaluate the risks before you enable it.

Configuration constraints

Storage space of data nodes — Only clusters in the cloud-native control architecture (v3) support downgrading the storage space of data nodes. A cluster in the v2 control architecture does not support reducing the disk capacity of a single node, and you cannot downgrade the storage space of a v2 cluster through the console or an API. To downgrade the storage space of a v2 cluster, contact technical support. To reduce the total disk capacity without technical support, scale in data nodes. This operation changes the number of nodes in the cluster.
Cluster version — You cannot upgrade the cluster version while you downgrade the cluster.
Node types per operation — A single downgrade operation can change only one type of node.
Elastic data nodes — Elastic data nodes cannot be downgraded.
Change interval — Two consecutive downgrade operations on the same cluster must be at least 30 minutes apart.
Number of nodes after a downgrade — The impact of a downgrade on the number of nodes depends on the update method:
In-place update — The number of nodes stays the same. Only the node specifications or the storage space changes.
Blue-green update — The system first adds the same number of new nodes as the current node count and migrates data to them. After the migration is complete, the old nodes are removed. The final number of nodes equals the number of nodes required by the target downgrade specifications. A temporary increase in the number of nodes during this process is normal.

CPU and memory specification constraints

Basic rule — The target CPU and memory must be at least half of the current CPU and memory.
Disallowed specifications — The following specifications cannot be used as downgrade targets: 1-core 2 GiB, 2-core 2 GiB, 2-core 4 GiB, and 4-core 4 GiB.
Kibana nodes — 2-core 2 GiB is a valid downgrade target for a Kibana node, even though this specification appears in the disallowed list for other node types.
Special case — If you must downgrade to a disallowed specification, create a new cluster and then migrate your data.

Cost impact

After you submit a downgrade order, the system bills you based on the updated configuration. For the billing rules, see pay-as-you-go and subscription.

Pre-downgrade checks

A downgrade performed without the following checks can crash the cluster, cause data loss, or make the service unavailable. Verify each item before you submit the downgrade order.

Health of the cluster

Run GET _cluster/health to make sure that the cluster status is GREEN.

Load safety

A cluster supports a downgrade if it meets the conditions in the following table.

Node type	CPU usage	JVM heap usage
Dedicated master node	Peak usage of a single node in the last 24 hours is lower than 30%.	Peak usage of a single node in the last 24 hours is lower than 25%.
Other node types	Both conditions must be met: peak usage of a single node in the last 24 hours is lower than 50%, and average usage of all nodes in the last 24 hours is lower than 30%.	Both conditions must be met: peak usage of a single node in the last 24 hours is lower than 50%, and average usage of all nodes in the last 24 hours is lower than 30%.

The thresholds in the preceding table are the validation thresholds that the system checks when you submit a downgrade order. They are evaluated over a statistical period of the last 1 day (the last 24 hours), not against the current real-time values shown on the monitoring page. Peak usage is the highest usage reached by a single node in the last 1 day. Average usage is the mean usage of all nodes in the last 1 day. Therefore, even if the current real-time value is low, the validation still blocks the order if the peak usage of a single node or the average usage of all nodes exceeded the threshold at any point in the last 1 day when you click Downgrade Configuration on the Instances page and submit the order. These validation thresholds are different from the high-load values that describe the risk of access timeouts during a configuration change. For the high-load values, see Service stability in Precautions.

If a message indicates that the CPU usage is higher than the validation threshold when you submit the order, check the peak usage of a single node and the average usage of all nodes in the last 1 day against the thresholds listed for each node type in the preceding table. Submit the downgrade order again after both metrics fall below the thresholds.

Index readiness

Run GET _cat/indices?v to check whether any index has a CLOSE status. If such an index exists, run POST /<index_name>/_open to temporarily open those indices. Because the indices are opened only temporarily, close them again after the change is complete if your business does not require them to stay open. A closed index can cause the configuration change to fail in the following ways:
- The cluster status cannot reach GREEN and can only reach YELLOW at best. Elasticsearch requires a GREEN cluster status before it performs certain sensitive configuration changes, such as adjusting shard allocation rules.
- The cluster reallocates shards during the configuration change. Shards of a closed index cannot participate in the reallocation, so operations that depend on a GREEN status then fail.
Run GET _cat/indices?v to check that the number of replicas of each index is at least 1.
For a multi-zone instance, make sure that the number of replicas of any index in the cluster is smaller than the number of zones during the change. (Recommended) Set the number of replicas to 1 and manually increase it after the change is complete.

If indexes contain a large number of deleted document fragments, the fragments consume substantial disk space. A disk downgrade can fail validation or report an error when the remaining disk space is insufficient. During off-peak business hours, perform a force merge to remove deleted document fragments and release disk space before you downgrade the disk.

Downgrade in the console

You can downgrade an Alibaba Cloud Elasticsearch cluster in the console or by calling an API operation. For the API method, see UpdateInstance. The following procedure describes the console method.

Choose a tab based on the change that you want to make. Use the Downgrade specifications, disk type, and space tab to downgrade node specifications, disk type, or disk space. Scaling in data nodes supports two entries: the Remove Data Nodes page, and Number of data nodes on the Downgrade Configuration page. A cluster in the cloud-native control architecture (v3) can be scaled in by decreasing Number of data nodes directly on the Downgrade Configuration page, so use the Downgrade specifications, disk type, and space tab for that operation. Use the Scale in data nodes tab if the downgrade page displays The current page does not support decreasing the number of nodes of this type. For data nodes, go to the cluster data node scale-in page.

Downgrade specifications, disk type, and space

In the Instances list, click the more icon on the right side of the target instance and select Downgrade Configuration from the drop-down menu.
Another entry: On the Basic Information page, click Configuration Update and then Downgrade.
On the Downgrade Configuration page, adjust the configuration parameters based on your business needs.
The parameters that you can adjust vary with the cluster type and version. The Downgrade Configuration page prevails.
- Node storage type — Downgrades follow this order, from the highest to the lowest performance. For the constraints on CPU and memory specifications, see CPU and memory specification constraints in Precautions.
  - Local disks: from local SSD (NVMe SSD local disk) to local SATA (SATA HDD local disk).
    Local disks are local hard disk devices on the physical machine where the ECS instance resides. They provide local storage access for the ECS instance and suit business scenarios that have very high requirements for storage I/O performance and cost-effective mass storage.
  - ESSD: An ESSD (Enterprise SSD) combines 25 GE networking and RDMA technology to deliver up to one million random read/write operations per disk and low single-path latency. An ESSD PL0 disk cannot be downgraded to a standard SSD.
  - Previous-generation disks: from standard SSD to ultra disk to basic disk. These disks are being phased out in some regions and zones. (Recommended) Select an ESSD when you choose a disk.
- Downgrade of data node storage space (only for clusters in the cloud-native control architecture (v3)) — To keep the cluster stable, the disk space usage after the downgrade must be lower than 60%. Before the downgrade, make sure that: current disk usage < disk space after the downgrade × 0.6.
  On the Basic Information page of the instance, check Control Architecture and confirm that its value is cloud-native control architecture (v3). For the options available to a cluster in the v2 control architecture, see Configuration constraints in Precautions.
- Downgrade of the number of data nodes (scale in data nodes) — In the Select Change Items section, click data node and decrease Number of data nodes to scale in data nodes.
  Note
  The downgrade page of some clusters does not support decreasing the number of nodes. If, after you decrease Number of data nodes, the page displays The current page does not support decreasing the number of nodes of this type. For data nodes, go to the cluster data node scale-in page., switch to the Scale in data nodes tab and complete the scale-in through the Remove Data Nodes entry.
- Weak validation for downgrade (disabled by default, not recommended) — After you enable this option, the system relaxes the resource validation performed before the downgrade. For the stability and SLA risks of this option, see Service stability in Precautions.
  You do not need to specify the update method. The system automatically selects the optimal update method (in-place update or blue-green update) based on the configuration items that you change.
Read and confirm the Terms of Service and the Service Level Agreement. If you have no objections, click Buy Now. The system charges fees based on your billing method.

Scale in data nodes

If the Downgrade Configuration page displays The current page does not support decreasing the number of nodes of this type. For data nodes, go to the cluster data node scale-in page., scale in data nodes on the Remove Data Nodes page.

On the Basic Information page of the instance, click Configuration Update and then Remove Data Nodes.
Select the node type and the number of nodes to remove based on your business needs.
The parameters that you can adjust vary with the cluster type and version. The console page prevails. In this example, the cluster runs Enhanced Edition 8.17.0.
For Select Node Type, you can choose data node or cold data node. The available options depend on the actual configuration of the cluster. Current Number of Nodes is the existing number of nodes of that type. In Number of Nodes to Remove, enter the number of nodes that you want to remove this time.
Click OK. The system performs the scale-in operation and calculates fees based on the cluster configuration and your billing method.
Alibaba Cloud Elasticsearch automatically performs a node safety check before the scale-in. If the check fails, resolve the error based on the error message and try the scale-in again.
If the data nodes that you want to remove contain data, the system prompts you to migrate the data. For more information, see Data migration and rollback.

During the change, the cluster status changes to Initializing. Performance of the cluster may fluctuate briefly and transient request failures may occur. After the change is complete, the cluster status updates to Normal. To confirm that the new configuration has taken effect, see Monitor progress and verify the downgrade.

Data migration and rollback

This section applies only when you scale in data nodes. To keep your data secure, the data nodes that you scale in must not contain data. If the selected data nodes contain data, the system prompts you to migrate the data. After the migration, no index data remains on the selected nodes and no new index data is written to them.

Data migration

In the Remove Data Nodes section, click Data Migration Tool in the prompt bar.
The Data Migration Tool uses the Elasticsearch shard allocation filter to achieve a smooth data migration. Your business is unaware of the migration process.
In the Migrate Data dialog box, select a node migration method.
Set Migration Node Type to data node and enter the Number of Nodes to Migrate. The system intelligently recommends the IP addresses of the nodes to migrate based on the data distribution and the cluster health. Confirm the data security a second time. Select the checkbox to agree to the Alibaba Cloud Elasticsearch Data Migration Service Agreement, and then click OK.
Select the data migration agreement and click OK.
The following table describes the node migration methods.

Parameter	Description
Smart Migration	The system automatically selects the data nodes to migrate based on its suggestions.
Custom	Manually select the data nodes to migrate.

Data rollback

Data migration is a long process. During this period, changes to the cluster status or to the data may cause the migration to fail. You can view the details in the Task List. After the data migration fails or completes, roll back the migrated nodes by performing the following steps:

Log on to the Kibana console of your Elasticsearch cluster.

For instructions, see Log on to the Kibana console.

Note
Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.
In the left navigation menu, choose Management > Dev Tools.

In the Console, run the following command to obtain the IP addresses of the migrated nodes.

GET _cluster/settings

After the command runs successfully, the following result is returned.

{
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "exclude": {
            "_ip": "192.168.xx.xx,192.168.xx.xx,192.168.xx.xx"
          }
        }
      }
    }
  }
}

Run the following command to roll back the data on the migrated nodes.

Roll back the data on some of the nodes. Remove the nodes to roll back from the configuration, but keep the nodes that are not rolled back.

PUT _cluster/settings
{
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "exclude": {
            "_ip": "192.168.xx.xx,192.168.xx.xx"
          }
        }
      }
    }
  }
}

Roll back the data on all of the nodes.

PUT _cluster/settings
{
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "exclude": {
            "_ip": null
          }
        }
      }
    }
  }
}

Run the following command to verify whether the data rollback is complete.
```
GET _cluster/settings
```
After the command runs successfully, if the returned result does not contain the IP addresses of the migrated nodes, the migration rollback task for those nodes is complete. You can also determine this by observing whether shards are reallocated to the corresponding nodes.

Note

During data migration or rollback, you can run the GET _cat/shards?v command to view the task status.

Monitor progress and verify the downgrade

Monitor the change progress

In the upper-right corner of the Basic Information page of the instance, click the More icon (three horizontal lines) to open the Task List panel. On the Change Progress tab, you can view information such as the change start time, change type, change status, and completion percentage.

Click Show Details to view the change details.

Verify the downgrade result

After the downgrade is complete, confirm on the cluster Basic Information page whether the configuration has taken effect:

The cluster status is restored to Normal.
Node count and storage specifications — Confirm that the number of nodes, the storage specifications, and the storage space match your expectations.
Shard balancing — Run GET _cat/allocation?v to check the shard distribution. If the shards are unbalanced, resolve the issue by following Solutions for uneven cluster load.

If the configuration change fails

View the change status and the change details on the Change Progress tab of the Task List panel, and then follow the troubleshooting steps in What do I do if a cluster configuration change fails? in the FAQ section of this topic.

FAQ

What do I do if a cluster configuration change fails?
Can I change the cloud disk type for an Elasticsearch instance?
Will changing the cluster configuration affect the Elasticsearch service?
Does the cluster automatically rebalance shards after the number of nodes changes?
What do I do if I chose the wrong configuration when purchasing an Elasticsearch instance?
Can I downgrade an instance specification after upgrading it? If so, how?
Why can't I downgrade cold data nodes?
Why does the number of nodes increase after a disk downgrade?
Elasticsearch adjusts the number of nodes to reach the target disk capacity. Therefore, the number of nodes may increase after a disk downgrade.
Do Logstash instances support downgrades?
Logstash instances do not support downgrade operations. To adjust the configuration of a Logstash instance, create a new instance and migrate your data. The downgrade operations described in this topic apply only to Elasticsearch instances.
Why did I not receive a refund after a downgrade? How is the downgrade refund calculated?
Downgrade refund amount = remaining fees of the original specification − purchase fees of the new specification. If the result calculated by this rule is less than 0, the remaining fees of the original specification are not enough to cover the purchase fees of the new specification, and no refund is issued. Not receiving a refund after a downgrade is a normal result in this case. You can verify the fees of this downgrade based on the preceding rule. Details such as the refund arrival time and the available refund channels are subject to description of downgrade refund rules.
What do I do if the message "The order configuration parameters do not meet the validation conditions" appears when I switch from subscription to pay-as-you-go?
When this error appears after you click Switch to Pay-as-you-go, it is usually caused by the compatibility of the instance architecture version. (Recommended) Upgrade the instance architecture to V3 first and then submit the switch to pay-as-you-go again. If the message about the order configuration parameters still appears after the upgrade, or if the architecture version of the instance cannot be upgraded, record the RequestID returned with the error and submit a ticket so that technical support can investigate and handle the issue.
What do I do if a downgrade is unavailable or a warning about insufficient resources appears?
In the Alibaba Cloud Console, go to the bottom of the Downgrade Configuration page and enable Weak validation for downgrade to skip some resource checks, including warnings about insufficient resources during a scale-in. After the scale-in is complete, perform shard balancing again. Enabling weak validation or performing a forced downgrade poses cluster stability risks and can cause service unavailability or data loss. Proceed only after you understand and accept these risks.