×
Community Blog FAQ about Alibaba Cloud Elasticsearch clusters 02

FAQ about Alibaba Cloud Elasticsearch clusters 02

This topic provides answers to some frequently asked questions about Alibaba Cloud Elasticsearch clusters.

FAQ about cluster configurations and configuration modifications

How do I plan resources, such as cluster specifications, the number of shards, and the size of each shard, before I use Elasticsearch?

You can evaluate the total amount of the resources that you need to purchase based on your business requirements. For more information, see Evaluate specifications and storage capacity.

How do I view the configuration of an Elasticsearch cluster?

You can view the configuration of the Elasticsearch cluster on the Basic Information page of the cluster. For more information, see View the basic information of a cluster.

When you use Transport Client to access an Elasticsearch cluster, set the cluster.name parameter to the ID of your cluster. For more information, see Transport Client (5.x).

Are services affected when I modify the configuration of an Elasticsearch cluster?

The system restarts the cluster after you modify the configuration of the cluster. The system uses the rolling restart method to restart a cluster. Before the restart, make sure that the cluster is in the Active state (indicated by the color green), each index has at least one replica shard for each primary shard, and resource usage is not high. For example, the value of NodeCPUUtilization(%) is about 80%, that of NodeHeapMemoryUtilization is about 50%, and that of NodeLoad_1m is less than the number of vCPUs of the current node. If all the conditions are met, the cluster can still provide services during the restart. You can view the resource usage on the Cluster Monitoring page. However, we recommend that you modify the configuration of your cluster during off-peak hours.

Does the system reallocate shards for nodes in an Elasticsearch cluster after the number of nodes in the cluster is changed?

Yes, the system reallocates shards for nodes in an Elasticsearch cluster after the number of nodes in the cluster is changed. However, the system cannot ensure even shard allocation. Due to factors such as the index size, number of shards, and number of nodes, data may still be unevenly distributed among shards after the reallocation. For information about how to troubleshoot and resolve uneven shard allocation, see Unbalanced loads on a cluster.

Can I change the cloud disk type of an Elasticsearch cluster?

Yes, you can change the cloud disk type of an Elasticsearch cluster. The following types of cloud disks are supported: ultra disks, standard SSDs, and enhanced SSDs (ESSDs). These types of cloud disks are listed in ascending order of their storage performance.

Can I convert other types of nodes in an Elasticsearch cluster to warm nodes?

No, you cannot convert other types of nodes in an Elasticsearch cluster to warm nodes. The conversion can cause your cluster to be unstable. For more information, see "Hot-Warm" Architecture in Elasticsearch 5.x.

Can I downgrade the configuration of an Elasticsearch cluster? If yes, how do I do?

Yes, you can downgrade the configuration of an Elasticsearch cluster. For more information, see Scale in a cluster or Downgrade the configuration of a cluster.

How do I modify the configuration of an Elasticsearch cluster to ensure that services run as expected when a temporary business surge occurs?

We recommend that you add nodes to the cluster when the temporary business surge occurs and remove the nodes after the business surge. For more information, see Upgrade the configuration of a cluster and Scale in a cluster. For the changes to take effect, the system restarts the cluster. Before the restart, take note of the following items:

  • The cluster is in the Active state (indicated by the color green).
  • Each index of the cluster has at least one replica shard for each primary shard, and the resource usage of the cluster is not high. For example, the value of NodeCPUUtilization(%) is about 80%, that of NodeHeapMemoryUtilization is about 50%, and that of NodeLoad_1m is less than the number of vCPUs of the current node. You can view the resource usage on the Cluster Monitoring page of the cluster.

When I upgrade the configuration of an Elasticsearch cluster, the "UpgradeVersionMustFromConsole" error message is reported. What do I do?

The error message is reported because the version change does not meet requirements. You can upgrade the versions of clusters only from V5.5.3 to V5.6.16, from V5.6.16 to V6.3.2, or from V6.3.2 to V6.7.0.

How long is required to upgrade the version of an Elasticsearch cluster?

The required time is determined by the data volume, data structure, and specifications of your cluster. The version upgrade requires about 1 hour.

Are services affected when I upgrade the version of an Elasticsearch cluster?

When you upgrade the version of an Elasticsearch cluster, you can still read data from or write data to the cluster but cannot make other changes. We recommend that you perform a version upgrade during off-peak hours. For more information about the precautions and procedure for a version upgrade, see Upgrade the version of a cluster.

What do I do if an error is reported when I upgrade the configuration of an Elasticsearch cluster or when a configuration upgrade for an Elasticsearch cluster times out?

In most cases, this issue occurs because the cluster is in an abnormal state. In this case, we recommend that you stop the query and write operations, and troubleshoot the issue by following the instructions described in What do I do if an Elasticsearch cluster is in a state indicated by the color red due to heavy loads? After the cluster recovers to a normal state, upgrade the configuration of the cluster again. You can also ignore the health status of the cluster and perform a forced update when you upgrade the configuration of the cluster. However, the forced update may affect the services provided by the cluster. Proceed with caution.

If this issue occurs due to other causes, resolve the issue based on the error message that is reported.

What do I do if I fail to update the configuration of my Elasticsearch cluster?

We recommend that you refer to the following instructions to troubleshoot the issue:

1)Check whether local disks are configured for the nodes in the cluster. The specifications of a local disk cannot be changed. If you want to use disks with higher specifications for the cluster, you must change the disk type for the cluster.

2)If the result of the check performed at the frontend shows that resources in the selected zone are insufficient, we recommend that you change the zone of the cluster and update the configuration of the cluster or wait until the clusters of other users in the selected zone are released.

3)If the result of the check performed at the frontend shows that the cluster is unhealthy, you must check whether the cluster stores indexes in the close state. If the cluster stores indexes in this state, you must temporarily open the indexes. If the cluster is in a state indicated by the color red, you must check whether some nodes in the cluster stop providing services or whether shards cannot be allocated to nodes in the cluster. If such issues occur, resolve the issues first.

4)If you want to downgrade the configuration of the cluster, make sure that the following conditions are met:

The selected vCPU and memory specifications are greater than or equal to half of the current specifications and are not the following specifications: 1 vCPU and 2 GiB of memory, 2 vCPUs and 2 GiB of memory, 2 vCPUs and 4 GiB of memory, and 4 vCPUs and 4 GiB of memory.

  • Note

If you want to downgrade the configuration of a cluster to 2 vCPUs and 4 GiB of memory or to 4 vCPUs and 4 GiB of memory, you need to create a cluster with such configuration and migrate data from the original cluster to the new cluster. You can use Logstash to migrate data.

  • The load of the cluster meets requirements. For more information about configuration downgrade, see Downgrade the configuration of a cluster.
  • The disk capacity does not need to be decreased. Configuration downgrade does not support the decrease of disk capacity.

Can I use the YML configuration file of an Elasticsearch cluster to configure the http.max_content_length and discovery.zen.ping_timeout parameters?

No, you cannot configure the two parameters. You can configure only parameters provided by Alibaba Cloud Elasticsearch. For more information, see Configure the YML file. Parameters that are not provided cannot be configured.

Note

In most cases, you do not need to change the settings of the following parameters: discovery.zen.ping_timeout, discovery.zen.fd.ping_timeout, discovery.zen.fd.ping_interval, and discovery.zen.fd.ping_retries.

Can I switch the VPC of an Elasticsearch cluster?

No, you cannot switch the VPC of an Elasticsearch cluster. You can purchase an Elasticsearch cluster in the desired VPC and migrate data from the original cluster to the new cluster. Then, cancel the subscription of or release the original cluster.

Will the existing data in an Elasticsearch cluster be lost if I change the cloud disk type of the cluster?

No, the existing data in an Elasticsearch cluster will not be lost if you change the cloud disk type of the cluster. However, new data that is continuously written to the cluster may be lost. We recommend that you change the cloud disk type during off-peak hours or after you stop the data write operations. For more information about how to change the cloud disk type of an Elasticsearch cluster, see Upgrade the configuration of a cluster.

When I upgrade the configuration of an Elasticsearch cluster, the system displays a prompt message indicating that the cluster is in an unhealthy state, but the cluster is in a state indicated by the color green. What do I do?

Some indexes in the cluster may be in the close state. You can run the POST /<index_name>/_open command to temporarily open the indexes. For more information, see Upgrade the configuration of a cluster.

Can I upgrade the vCPU configurations of an Elasticsearch cluster without migrating data?

No, you cannot upgrade the vCPU configurations of an Elasticsearch cluster without migrating data. If you upgrade or downgrade the vCPU configurations of an Elasticsearch cluster, the system performs a blue-green update for the cluster. After the blue-green update, the IP addresses of nodes in the cluster are changed, and data is migrated from the original nodes to the new nodes.

Why am I unable to downgrade the configuration of warm nodes in my Elasticsearch cluster?

Specific conditions must be met before you can downgrade the configuration of a cluster. For example, the selected vCPU and memory specifications must be greater than or equal to half of the current specifications and cannot be the following specifications: 1 vCPU and 2 GiB of memory, 2 vCPUs and 2 GiB of memory, 2 vCPUs and 4 GiB of memory, and 4 vCPUs and 4 GiB of memory. For more information, see Limits.

If your Elasticsearch cluster does not meet the conditions required for configuration downgrade, you can create another cluster that meets your business requirements, migrate data from the original cluster to the new cluster, and then cancel the subscription of or release the original cluster. For information about data migration, see Select a data migration solution.

When I remove data nodes from my Elasticsearch cluster, the following error message is reported: "This operation may cause a shard allocation error or insufficient storage, CPU, or memory resources." What do I do?

Possible cause Solution
The resources of the cluster are insufficient.After data nodes are removed, the cluster does not have sufficient resources to store system data or handle workloads. The resources include disks, memory, and vCPUs. Run the GET _cat/indices?v command to check whether the resource usage of your cluster, such as disk usage, is greater than the related threshold. Make sure that the cluster has sufficient resources to store data and process requests. If these requirements are not met, upgrade the configuration of the cluster. For more information, see Upgrade the configuration of a cluster.
Errors occur on shard allocation.Elasticsearch is based on Lucene principles. This indicates that Elasticsearch does not migrate two or more replica shards of an index on a data node to the same data node. In this case, after data nodes are removed, the number of replica shards in a cluster may be greater than or equal to the number of data nodes. This results in shard allocation errors. Run the GET _cat/indices?v command to check whether the number of replica shards in the cluster is less than the number of data nodes after specific data nodes are removed. If this requirement is not met, change the number of replica shards. For more information, see Index Templates. The following code provides an example on how to change the number of replica shards to 2 in the index template:PUT _template/template_1 { "template": "*", "settings": { "number_of_replicas": 2 } }

When I remove data nodes from my Elasticsearch cluster, the error message "The cluster is in an abnormal state or has ongoing tasks." is reported. What do I do?

Use the Cluster Diagnosis feature to diagnose the cluster and troubleshoot the issue based on the diagnostic results and suggestions. For more information, see Perform a diagnostic on an Elasticsearch cluster.

When I remove data nodes from my Elasticsearch cluster, the error message "The number of nodes that you reserve must be more than two." is reported. What do I do?

To ensure cluster reliability and stability, at least two data nodes must be reserved after data node removal. For a multi-zone cluster, the number of data nodes in each zone must be greater than or equal to two, and the numbers of remaining data nodes in all zones must be the same. If the requirements are not met, adjust the data nodes to remove, or upgrade the configuration of the cluster. For more information about how to upgrade the configuration of an Elasticsearch cluster, see Upgrade the configuration of a cluster.

When I remove data nodes from my Elasticsearch cluster, the error message "The operation is not supported." is reported. What do I do?

Run the GET _cluster/settings command to query the configurations of the cluster and check whether the cluster contains the configuration "cluster.routing.allocation.enable" : "none". This configuration does not allow data distribution. If the cluster contains the configuration, you can temporarily change the configuration to "cluster.routing.allocation.enable" : "all". If the configuration affects your other operations, you can change the configuration to the original setting after data node removal.

What do I do if data nodes fail to be removed or data fails to be migrated due to the auto_expand_replicas index setting?

  • Cause

You may use the access control feature provided by the X-Pack plug-in. In earlier Elasticsearch versions, this feature applies the "index.auto_expand_replicas" : "0-all" setting to .security indexes by default. This causes errors when you migrate data or remove data nodes.

  • Solution
  1. Run the following command to query index settings:
GET .security/_settings

The following result is returned:

{
  ".security-6" : {
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "provided_name" : ".security-6",
        "format" : "6",
        "creation_date" : "1555142250367",
        "priority" : "1000",
        "number_of_replicas" : "9",
        "uuid" : "9t2hotc7S5OpPuKEIJ****",
        "version" : {
          "created" : "6070099"
        }
      }
    }
  }
}
  1. Use one of the following methods to modify the auto_expand_replicas index setting:
  • Method 1
PUT .security/_settings
{
  "index" : {
    "auto_expand_replicas" : "0-1"
  }
}
  • Method 2
PUT .security/_settings
{
  "index" : {
    "auto_expand_replicas" : "false",
    "number_of_replicas" : "1"
  }
}

Important

The number_of_replicas parameter specifies the number of replica shards for each primary shard in an index. You can configure this parameter based on your business requirements. Make sure that the value of this parameter is greater than or equal to 1 but no more than the number of available data nodes.

How do I clear the cache for an Elasticsearch cluster?

Log on to the Kibana console of the cluster and run one of the following commands:

  • Clear the cache of a specific index
POST /<Index name>/_cache/clear?fielddata=true
  • Clear all cache
POST /_cache/clear

How do I migrate nodes in an Elasticsearch cluster from one zone to another?

Perform the steps described in Migrate nodes in a zone to migrate nodes from one zone to another.

Can I update only the disk configuration of an Elasticsearch cluster during a configuration upgrade of the cluster?

Yes, you can update only the disk configuration of an Elasticsearch cluster during a configuration upgrade of the cluster. For more information, see Upgrade the configuration of a cluster.

Important

When you update the disk configuration of an Elasticsearch cluster, the system performs a rolling restart for the cluster. We recommend that you update the disk configuration of an Elasticsearch cluster during off-peak hours.

Can I change the JVM parameter settings of an Elasticsearch cluster?

Alibaba Cloud Elasticsearch clusters use the JVM parameter settings that are recommended by open source Elasticsearch. The settings cannot be changed. By default, the JVM heap memory of an Elasticsearch cluster is half of the memory of the cluster. A maximum of 32 GB of JVM heap memory can be allocated to an Elasticsearch cluster. For more information, see Heap size settings.

FAQ about plug-ins, tokens, and synonyms

How do I update dictionaries when I use the IK analysis plug-in?

You can use the standard update or rolling update feature of the IK analysis plug-in to update dictionaries. For more information, see Use the analysis-ik plug-in.

When I use the IK analysis plug-in, the "ik startOffset" error message is reported. What do I do?

The error message is returned because of an Elasticsearch V6.7 bug. You must restart your cluster. For more information, see Restart a cluster or node.

The IK dictionary files on my on-premises machine are lost. Can I retrieve them on the cluster management page?

No, you cannot retrieve them on the cluster management page. You can only delete or update dictionary files on the cluster management page. We recommend that you download the official main and stopword dictionary files. Then, change the tokens in the files to those in your system dictionary file and upload the files to your cluster.

How do I apply updated IK dictionaries to existing data?

You must perform a reindex operation. If indexes are configured with IK tokens, the updated dictionaries apply only to new data in these indexes. If you want to apply the updated dictionaries to all the data in these indexes, you must perform a reindex operation. For more information, see Configure a remote reindex whitelist.

Is a threshold specified for full GC?

Full garbage collection (GC) is used to clean the entire heap memory. Whether full GC is correctly performed needs to be analyzed based on the service latency, heap memory size before full GC, and heap memory size after full GC. The CMS collector starts to collect garbage when the memory usage reaches 75%. This is because some space is reserved for burst traffic.

Can I remove built-in plug-ins that are not used?

You can remove only some plug-ins. On the Built-in Plug-ins tab of the Plug-ins page for your Elasticsearch cluster, you can view the plug-ins that can be removed. If Remove is displayed in the Actions column of a plug-in, the plug-in can be removed. For more information about how to remove a plug-in, see Install and remove a built-in plug-in.

Are the dictionaries provided by the IK analysis plug-in of Alibaba Cloud Elasticsearch the same as the dictionaries provided by the IK analysis plug-in of open source Elasticsearch?

Yes, the dictionaries provided by the IK analysis plug-in of Alibaba Cloud Elasticsearch are the same as the dictionaries provided by the IK analysis plug-in of open source Elasticsearch. For more information, see IK Analysis for Elasticsearch.

Can a custom plug-in access an external network, such as reading dictionary files on GitHub?

No, custom plug-ins cannot access external networks. If you want your Elasticsearch cluster to access external files, upload the files to Object Storage Service (OSS) and connect your Elasticsearch cluster to OSS.

Does a custom plug-in support the rolling update method?

No, custom plug-ins do not support the rolling update method. If you want a custom plug-in to support this method, configure the plug-in based on the rolling update method of the IK analysis plug-in. For more information, see IK Analysis for Elasticsearch.

How do I configure the analysis-aliws plug-in? What is the format of the dictionary file for this plug-in?

For more information about how to configure the plug-in, see Use the analysis-aliws plug-in.

The dictionary file must meet the following requirements:

  • Name: aliws_ext_dict.txt.
  • Encoding format: UTF-8.
  • Content: Each row contains one word and ends with n (line feed in UNIX or Linux). No whitespace characters are used before or after this word. If the dictionary file is generated in Windows, you must use the dos2unix tool to convert the file before you upload it.

What are the differences among Elasticsearch synonyms, IK tokens, and AliNLP tokens?

Token type Usage Description Supported file type Tokenizer and analyzer
Synonym You can upload a synonym dictionary file on the Cluster Configuration page of your cluster to enable the cluster to use it. After you write several synonyms to the file, the system displays all the synonyms when you query one of them. The synonym dictionary file must be a TXT file encoded in UTF-8. Custom tokenizer and analyzer
IK token The IK tokens are used based on the analysis-ik plug-in. The system splits a paragraph based on the main.dic file. If you send a query request that contains one or more words split from the paragraph, the system returns the entire paragraph in the query result. The analysis-ik plug-in also provides a stopword file named stop.dic. The query result does not include the stopwords in the stop.dic file. You can view the dictionary file from the official documentation. The main and stopword dictionary files must be DIC files encoded in UTF-8. Tokenizer:ik_smartik_max_word
AliNLP token The AliNLP tokens are used based on the analysis-aliws plug-in. The analysis-aliws plug-in works in a similar way as the analysis-ik plug-in, but the analysis-aliws plug-in does not provide a separate stopword dictionary file. Stopwords are integrated into the main dictionary file aliws_ext_dict.txt. The file is invisible to you. In addition, you are not allowed to customize stopwords. The dictionary file name must be aliws_ext_dict.txt. The file must be encoded in UTF-8. Analyzer: aliws, which does not return function words, function phrases, or symbolsTokenizer: aliws_tokenizer

How do I install the analysis-ik plug-in?

analysis-ik is an IK analysis plug-in provided by Alibaba Cloud Elasticsearch. This plug-in is a built-in plug-in and cannot be removed. You can use the standard or rolling update method to update the built-in IK main dictionary and stopword list of the analysis-ik plug-in. Then, you can use the updated dictionary and stopword list when you configure mappings for an index. For more information about how to use the analysis-ik plug-in, see Use the analysis-ik plug-in.

Which built-in Chinese tokenizers are supported by Alibaba Cloud Elasticsearch?

Alibaba Cloud Elasticsearch supports the following built-in Chinese tokenizers: analysis-ik and analysis-aliws. You can use these plug-ins after you configure the related dictionaries.

If I use the rolling update method to update dictionaries that are dynamically loaded from OSS and the dictionaries stored in OSS are updated, will the dictionaries on all nodes in my Elasticsearch cluster be automatically updated?

No, the dictionaries on all nodes in your Elasticsearch cluster will not be automatically updated. Alibaba Cloud Elasticsearch does not support the automatic update of dictionaries on nodes in an Elasticsearch cluster after a rolling update of the dictionaries stored in OSS. After the dictionaries stored in OSS are changed, you must manually upload the dictionary file for the updated dictionary file to take effect. For indexes that are configured with IK tokens, synonyms, or AliNLP tokens, new dictionaries take effect only for data that is inserted after a standard or rolling update. If you also want the new dictionaries to take effect for existing data, you must reindex the existing data.

Does the analysis-ik plug-in provided by Alibaba Cloud Elasticsearch support a remote dictionary?

No, the analysis-ik plug-in provided by Alibaba Cloud Elasticsearch does not support a remote dictionary. The analysis-ik plug-in allows you to upload or update dictionaries. For more information about this plug-in, see Use the analysis-ik plug-in. The analysis-ik plug-in does not support a remote dictionary or configurations related to a remote dictionary. For example, the IKAnalyzer.cfg.xml file cannot contain configurations related to a remote dictionary.

How do I install the aliyun-knn plug-in for an Elasticsearch V7.10 cluster?

The aliyun-knn plug-in for Alibaba Cloud Elasticsearch V7.10 clusters is integrated into the built-in apack plug-in. If you want to remove or reinstall the aliyun-knn plug-in, you must perform operations on the apack plug-in. For information about the apack plug-in, see Use the physical replication feature of the apack plug-in. For information about how to install the aliyun-knn plug-in for Elasticsearch clusters of other versions, see Use the aliyun-knn plug-in.

Note

If the kernel version of your cluster is V1.4.0 or later, the apack plug-in is of the latest version. You can run the GET _cat/plugins?v command to obtain the version of the apack plug-in.

Are cluster services affected when an Elasticsearch cluster is restarted after a plug-in is installed for the cluster?

In most cases, if the load of a cluster is not high and the indexes in the cluster have replica shards, the cluster can still provide services during a restart. However, access timeouts may occur during a restart in the following cases: Some nodes in the cluster are forced to restart at the same time, the cluster is heavily loaded and is not accessible, the indexes in the cluster do not have replica shards, and large amounts of data are written or queried during a restart or forced restart. In these cases, we recommend that you design a retry mechanism on your client and restart the cluster during off-peak hours.

FAQ about logs

Can I specify a retention period for the .security indexes of an Elasticsearch cluster?

Yes, you can specify a retention period for the .security indexes of an Elasticsearch cluster. You can use the index lifecycle management (ILM) feature to specify the retention period. For more information, see Use ILM to manage Heartbeat indexes.

Important

The .security indexes store information about the elastic account of Elasticsearch clusters. If you enable the system to periodically delete such indexes, you may fail to log on to the Kibana console of your Elasticsearch cluster by using your elastic account.

How do I store the logs of an Elasticsearch cluster on my on-premises machine?

You can call the ListSearchLog API operation to obtain the logs of your cluster. Then, store the obtained logs on your on-premises machine. For more information, see ListSearchLog.

I am unable to view the search and update logs of an Elasticsearch cluster. What do I do?

You can configure slow logs and reduce the timestamp precision of log entries. For more information, see References.

How do I configure slow log collection for and view the slow logs of an Elasticsearch cluster?

By default, Elasticsearch logs only read and write operations that require 5 seconds to 10 seconds to complete as slow logs. You can log on to the Kibana console of the cluster and run the related command to reduce the timestamp precision of log entries. This helps capture more logs. For more information, see References.

Note

You cannot change the format of slow logs.

How do I obtain the slow logs of an Elasticsearch cluster on a regular basis?

You can call the ListSearchLog API operation to obtain the slow logs of your cluster on a regular basis. For more information, see ListSearchLog.

How do I query the clients that are used to access an Elasticsearch cluster?

You can view the access logs or audit logs of an Elasticsearch cluster to obtain the required information about the cluster.

  • If you want to view the information about the operations that are performed on an Elasticsearch cluster, such as add, delete, modify, and query operations, you must enable audit log collection for the cluster.
  • If you want to view the details of all query requests that are received by an Elasticsearch cluster, such as the names of nodes that are requested, IP addresses of the nodes, sizes of request bodies, request content, time when the requests are initiated, client IP addresses that are used to send requests, and URIs, you must log on to the Elasticsearch console and view the access logs of the cluster on the Access Log tab of the Logs page.

For information about the limits and precautions for access logs and audit logs and how to enable audit log collection for an Elasticsearch cluster, see Query logs.

FAQ about data backup and restoration

Can I restore data from the snapshots of an Elasticsearch cluster to an Elasticsearch cluster of a different version?

For automatic snapshots, you can restore data from the snapshots to the original cluster or use a shared OSS repository to restore data from the snapshots of an Elasticsearch cluster to other Elasticsearch clusters. For more information, see Create automatic snapshots and restore data from automatic snapshots and Configure a shared OSS repository.

For manual snapshots, you can directly restore data from the snapshots to other clusters. We recommend that you use a destination cluster whose version is the same as the version of the original cluster. If the versions are different, compatibility issues may occur. For more information, see Create manual snapshots and restore data from manual snapshots.

What do I do if a message indicating that the Elasticsearch cluster is unhealthy appears when I back up data for the cluster?

When an Elasticsearch cluster is unhealthy, snapshots for data backup cannot be created for the cluster. We recommend that you recover the cluster to a normal state indicated by the color green before data backup.

I enable the Auto Snapshot feature but do not specify shared OSS repositories for an Elasticsearch cluster. Are snapshots created?

Elasticsearch provides an OSS bucket for your cluster by default. You can log on to the Kibana console of your cluster and run the GET _snapshot/aliyun_auto_snapshot/_all command to obtain automatic snapshots. For more information about how to log on to the Kibana console, see Log on to the Kibana console.

When I restore data from snapshots, the destination Elasticsearch cluster displays a message indicating that shards are abnormal. After I run the **POST /_cluster/reroute?retry_failed=true** command to reroute the shards, the issue persists. What do I do?

The following figure shows the issue.

img

Delete the problematic index and call the _restore API to restore it. You must add the max_restore_bytes_per_sec parameter to the command that is used to restore data. This parameter is used to limit the restoration rate. The default value of this parameter is 40mb. This value indicates that the index is restored at a speed of 40 MB per second.

POST /_snapshot/aliyun_snapshot_from_instanceId/es-cn-instanceId_datetime/_restore
{
    "indices": "myIndex",
    "settings": {
    "max_restore_bytes_per_sec" : "150mb" 
    }
}

Note

You can also add the following parameters:

  • compress: specifies whether to enable data compression. Default value: true.
  • max_snapshot_bytes_per_sec: specifies the rate at which snapshots are created for each node. Default value: 40mb.

Can I export data from an Elasticsearch cluster to my on-premises machine?

Yes, you can export data from an Elasticsearch cluster to your on-premises machine. You can use the data backup feature provided by Elasticsearch to export data. For more information, see Data backup overview. You can create snapshots, store them in OSS, and then download objects from OSS. For more information, see Download objects.

How do I restore data from snapshots of an Elasticsearch cluster to another Elasticsearch cluster?

Use a shared OSS repository to restore the data. For information about the detailed operations, limits, and precautions, see Configure a shared OSS repository. If you want to migrate data between two Elasticsearch clusters that belong to the same Alibaba Cloud account but reside in different regions, you can run the commands that are used to create manual snapshots for index data and restore data from the snapshots. For information about available data migration solutions, see Select a data migration solution.

What data backup features does Alibaba Cloud Elasticsearch provide?

For information about the data backup features that are provided by Alibaba Cloud Elasticsearch and the use scenarios and limits of the features, see Data backup overview.

FAQ about cluster monitoring and alerting

How do I configure a DingTalk chatbot or WeCom chatbot to receive alert notifications from X-Pack Watcher?

X-Pack Watcher is a monitoring and alerting service based on Elasticsearch. For information about how to configure a DingTalk chatbot or a WeCom chatbot to receive alert notifications from X-Pack Watcher, see Configure a DingTalk chatbot to receive alert notifications from X-Pack Watcher and Configure a WeCom chatbot to receive alert notifications from X-Pack Watcher.

Note

If you configure X-Pack Watcher for your Elasticsearch cluster, X-Pack Watcher can trigger actions when specific conditions are met. For example, if the logs index contains errors, X-Pack Watcher triggers the system to send alert notifications by DingTalk message. X-Pack Watcher is a monitoring and alerting service based on Elasticsearch.

What do I do if the system reports an alert indicating that memory cannot be allocated to the garbage collector?

The possible causes of this issue include heavy loads, high query QPS, and large amounts of data to write. Refer to the following instructions to resolve the issue:

High query QPS or large amounts of data to write: We recommend that you install the aliyun-qos plug-in on your Elasticsearch cluster to implement read/write throttling. For more information, see Use the aliyun-qos plug-in.

  • Note

For image retrieval, we recommend that you install the aliyun-knn plug-in on your Elasticsearch cluster and plan your cluster and indexes. For more information, see Use the aliyun-knn plug-in.

What do the values of the ClusterStatus(value) metric mean?

The ClusterStatus(value) metric is used to evaluate the health status of a cluster. The value 0.00 indicates that the cluster is normal. The following table describes the values of the ClusterStatus(value) metric. For more information, see Metrics and exception handling suggestions.

Value Description
0.00 The Elasticsearch cluster is in a normal state.
1.00 The Elasticsearch cluster is in a sub-healthy state. One or more indexes have unassigned replica shards. The Elasticsearch cluster can continue to provide services.
2.00 The Elasticsearch cluster is in an abnormal state. One or more indexes have unassigned primary shards. The Elasticsearch cluster cannot continue to provide services. You must recover the Elasticsearch cluster to a normal state at the earliest opportunity.

How do I view the disk usage of each node in an Elasticsearch cluster?

You can use one of the following methods to view the disk usage of each node in an Elasticsearch cluster: View the monitoring data of the Elasticsearch cluster on the Cluster Monitoring page of the Elasticsearch console, or view the monitoring log data that is generated after you configure monitoring indexes for the Elasticsearch cluster in the Kibana console of the cluster. For more information, see Metrics and exception handling suggestions and Configure monitoring indexes.

The promotion failed error is reported when the CMS garbage collector performs GC. What do I do?

This error is reported because the space in the old generation is insufficient and objects fail to be promoted to the old generation. You can resolve the issue based on the following instructions:

  • Analysis of monitoring data and logs
  • View GC logs to obtain detailed GC information, check whether frequent CMS GC operations or full GC operations are performed, and check whether the promotion failure is caused by insufficient space in the old generation.
  • Log on to the Elasticsearch console and go to the Logs page of your Elasticsearch cluster. On the Logs page, search for logs that contain the promotion failed keyword. Then, analyze the logs to obtain the failure cause.
  • Adjustment of the heap memory size and garbage collector configuration
  • If your Elasticsearch cluster is of V6.7.0 or later and the memory size of each data node in the cluster is greater than or equal to 32 GiB, we recommend that you use the G1 garbage collector instead to optimize GC performance.
  • Check whether you need to increase the memory size of data nodes based on your business requirements and cluster resource usage.
  • Optimization suggestion

If your cluster frequently encounters memory-related issues, you may need to evaluate the data volume of the cluster and check cluster loads and resource configurations. If necessary, contact Alibaba Cloud technical support for professional optimization guidance.

FAQ about access to clusters

How do I use a client to access an Alibaba Cloud Elasticsearch cluster? What is the difference between access to an Alibaba Cloud Elasticsearch cluster and access to an open source Elasticsearch cluster?

You can access an Alibaba Cloud Elasticsearch cluster by using its internal or public endpoint. You can access an open source Elasticsearch cluster by using its address. For more information, see Use a client to access an Alibaba Cloud Elasticsearch cluster.

Can I disable the basic authentication feature when I use a client to access an Elasticsearch cluster?

No, you cannot disable the basic authentication feature. The basic authentication feature is a Kibana authentication mechanism provided by the built-in X-Pack plug-in of Elasticsearch. You cannot disable the feature.

I purchased an ECS instance that resides in the same VPC as but a different zone from an Elasticsearch cluster. Can I use the ECS instance to access the Elasticsearch cluster over an internal network?

Yes, you can use the ECS instance to access the Elasticsearch cluster over an internal network. You can use an ECS instance to access an Elasticsearch cluster over an internal network if they reside in the same VPC.

How do I configure a public or private IP address whitelist for an Elasticsearch cluster?

If you want to access an Alibaba Cloud Elasticsearch cluster over the Internet or a VPC, you must add the IP address of your device to a public or private IP address whitelist of the cluster. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster. Before you configure an IP address whitelist, take note of the following items:

  • By default, Public Network Access is turned off. You must turn on Public Network Access before you can configure a public IP address whitelist.
  • An IP address whitelist can contain a maximum of 50 IP addresses or CIDR blocks.
  • If you want to specify CIDR blocks, make sure that the IP address that precedes the forward slash (/) in each CIDR block is the first IP address obtained based on subnet mask calculation.
  • You are not allowed to add 0.0.0.0/0 and one or more other IP addresses or CIDR blocks to the same IP address whitelist. Otherwise, the system displays an error message. If you need to add 0.0.0.0/0 to an IP address whitelist for a test, add only 0.0.0.0/0 to the whitelist.

How do I access an Elasticsearch cluster over the Internet?

You can access an Elasticsearch cluster over the Internet by using its public endpoint. However, you must configure a public IP address whitelist before you access the cluster. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster. When you access the cluster, you must configure parameters such as the domain name, username, and password. For more information, see Use a client to access an Alibaba Cloud Elasticsearch cluster.

I fail to access an Elasticsearch cluster and the system reports the following error message: Failed to establish a new connection: [Errno 61] Connection refused. What do I do?

The following table describes the possible causes of this issue and the related solutions.

Possible cause Solution
The Elasticsearch cluster cannot be accessed over the Internet. If you access the Elasticsearch cluster over its public endpoint, refer to the following instructions to troubleshoot the issue:Make sure that the IP address of your device is added to a public IP address whitelist of the cluster. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster.Check the network connectivity of the Elasticsearch cluster by running a ping or telnet command. If the network connection of the Elasticsearch cluster is normal, run a curl command to access the Elasticsearch cluster. For more information, see Use curl commands and API operations to manage an Alibaba Cloud Elasticsearch cluster.NotePing command: ping <Public endpoint of the Elasticsearch cluster>Telnet command: telnet <Public endpoint of the Elasticsearch cluster> <Port number>
The Elasticsearch cluster cannot be accessed over an internal network. If you access the Elasticsearch cluster over its internal endpoint, refer to the following instructions to troubleshoot the issue:Make sure that the client that you use to access the Elasticsearch cluster resides in the same VPC as the Elasticsearch cluster. You can run the ping <Internal endpoint of the Elasticsearch cluster> command to test the network connectivity of the Elasticsearch cluster over its internal endpoint.Make sure that the curl command that you use to access the Elasticsearch cluster is correct. For more information, see Use curl commands and API operations to manage an Alibaba Cloud Elasticsearch cluster.
The Elasticsearch cluster is unhealthy. If the network connection of the Elasticsearch cluster is normal but access to the cluster fails, refer to the following instructions to check the status of the cluster and resolve the issue based on the actual situation:Run the GET _cat/health?v command to query the health status of the Elasticsearch cluster and check whether situations such as node disconnection and unassigned shards exist.View the monitoring data of the Elasticsearch cluster to check whether the resource usage of the cluster, such as the CPU utilization, JVM heap memory usage, and disk usage, is normal. For more information, see Metrics and exception handling suggestions.View the logs of the Elasticsearch cluster to check whether situations such as circuit breaking, node disconnection, and node removal exist. For more information, see Query logs.

Is access to an Elasticsearch cluster affected if I reset the password of the elastic account for the cluster?

If you reset the password of the elastic account for an Elasticsearch cluster in the Elasticsearch console, only access to the cluster by using the elastic account is affected. Access to the Elasticsearch cluster by using other accounts is not affected. We recommend that you use a custom account to access the Elasticsearch cluster. The custom account must be assigned a role with the required permissions. For more information, see Use the RBAC mechanism provided by Elasticsearch X-Pack to implement access control.

Note

After you reset the password of the elastic account for the Elasticsearch cluster, the system does not automatically restart the Elasticsearch cluster for the new password to take effect.

What do I do if I fail to use ElasticSearch Head 5.0.0 to access all versions of Alibaba Cloud Elasticsearch clusters?

This issue occurs because Google Chrome does not allow cross-origin resource sharing (CORS). You can perform the following steps to resolve this issue for devices that run macOS. For devices that run other operating systems, refer to the CORS-related configurations provided by Google Chrome.

1)Create a folder.

2)Start Terminal and run the following command:

open-n/Applications/Google\Chrome.app/--args--disable-web-security--user-data-dir=Path of the folder

Important

ElasticSearch Head is no longer maintained in versions later than Elasticsearch 5.x. We recommend that you use Cerebro to access your Alibaba Cloud Elasticsearch cluster. For more information, see Use Cerebro to access an Elasticsearch cluster.

0 1 0
Share on

Data Geek

100 posts | 4 followers

You may also like

Comments

Data Geek

100 posts | 4 followers

Related Products

  • Alibaba Cloud Elasticsearch

    Alibaba Cloud Elasticsearch helps users easy to build AI-powered search applications seamlessly integrated with large language models, and featuring for the enterprise: robust access control, security monitoring, and automatic updates.

    Learn More
  • CloudBox

    Fully managed, locally deployed Alibaba Cloud infrastructure and services with consistent user experience and management APIs with Alibaba Cloud public cloud.

    Learn More
  • Alibaba Cloud Flow

    An enterprise-level continuous delivery tool.

    Learn More