Alibaba Cloud Elasticsearch allows you to remove data nodes from an Elasticsearch cluster to scale in it.
Precautions
After you remove data nodes from an Elasticsearch cluster, the system restarts the cluster. Therefore, before the removal, make sure that the restart does not affect your business.
Remove data nodes
Roll back data migration
Data migration is time-consuming. Cluster status changes or data modifications may result in a data migration failure. You can view detailed information in the Tasks dialog box. To roll back data migration, perform the following steps:
FAQ
- What do I do if the "This operation may cause a shard distribution error or insufficient
storage, CPU, or memory resources." message appears?
Cause
- Insufficient resources
After data nodes are removed, the cluster does not have sufficient disk capacity, memory, or CPU resources to store system data or handle workloads.
- Shard allocation errors
Based on Lucene principles, Elasticsearch does not migrate two or more replica shards of the same index on a data node to the same data node. In this case, after data nodes are removed, the number of replica shards in a cluster may be greater than or equal to the number of data nodes. This results in shard allocation errors.
Solution- Insufficient resources
Run the GET _cat/indices?v
command to check whether the resource usage, such as the disk usage, is greater than the threshold. You must make sure that the cluster has sufficient resources to store data or process requests. If these requirements are not met, upgrade the configuration of the cluster. For more information, see Upgrade the configuration of a cluster. - Shard allocation errors
Run the
GET _cat/indices?v
command to check whether the number of replica shards in the cluster is less than the number of data nodes after data nodes are removed. If this requirement is not met, change the number of replica shards. For more information, see Index Templates. The following sample code demonstrates how to modify the index template to set the number of replica shards to 2:PUT _template/template_1 { "template": "*", "settings": { "number_of_replicas": 2 } }
- Insufficient resources
- What do I do if the "The cluster is running tasks or in an error status. Try again
later." message appears?
Solution: Run the
GET _cluster/health
command to check the status of the cluster or go to the pages under Intelligent Maintenance to view the cause. - What do I do if the "The nodes in the cluster contain data. You must migrate the data
first." message appears?
Solution: Migrate data. For more information, see Remove data nodes.
- What do I do if the "The number of nodes that you reserve must be more than two and
more than half of the existing nodes." message appears?
Cause: To ensure the reliability and stability of the cluster, the number of data nodes you reserve during data node removal or data migration must be greater than two and greater than half of the existing data nodes.
Solution: If the preceding requirements are not met, re-select the data nodes or upgrade the configuration of the cluster. For more information about how to upgrade the configuration of a cluster, see Upgrade the configuration of a cluster.
- What do I do if the "The current Elasticsearch cluster configuration does not support
this operation. Check the Elasticsearch cluster configuration first." message appears?
Solution: Run the
GET _cluster/settings
command to view the cluster configuration. Then, check whether the cluster configuration contains the settings that do not allow data allocation. - What do I do if data nodes fail to be removed or data fails to be migrated due to
the
auto_expand_replicas
index setting?Cause: Some users may use the access control feature provided by the X-Pack plug-in. In earlier Elasticsearch versions, this feature applies the
"index.auto_expand_replicas" : "0-all"
setting to the .security index by default. This causes errors when you migrate data or remove data nodes.Solution:- Query index settings.
GET .security/_settings
The following result is returned:{ ".security-6" : { "settings" : { "index" : { "number_of_shards" : "1", "auto_expand_replicas" : "0-all", "provided_name" : ".security-6", "format" : "6", "creation_date" : "1555142250367", "priority" : "1000", "number_of_replicas" : "9", "uuid" : "9t2hotc7S5OpPuKEIJ****", "version" : { "created" : "6070099" } } } } }
- Use one of the following methods to modify the auto_expand_replicas index setting:
- Method 1:
PUT .security/_settings { "index" : { "auto_expand_replicas" : "0-1" } }
- Method 2:
PUT .security/_settings { "index" : { "auto_expand_replicas" : "false", "number_of_replicas" : "1" } }
Notice Setnumber_of_replicas
based on your business requirements. Make sure that the number of replica shards enabled for each index is greater than or equal to one but no more than the number of available data nodes.
- Method 1:
- Query index settings.