In the YML Configuration section of the Cluster Configuration page of your Alibaba Cloud Elasticsearch cluster, you can enable the Auto Indexing, Audit Log Indexing, or Watcher feature. You can also specify Index Deletion and Other Configurations. This topic describes how to configure the following items: parameters in the YML file of the cluster, cross-origin resource sharing (CORS), a remote reindex whitelist, the Audit Log Indexing feature, and queue sizes.
Due to the adjustment made to the Alibaba Cloud Elasticsearch network architecture, clusters created after October 2020 do not support the X-Pack Watcher and LDAP authentication features. You cannot reindex, search for, and replicate data between a cluster created before October 2020 and a cluster created after October 2020. You can perform the operations only between clusters created before October 2020 or between clusters created after October 2020. The features will be available soon.
Configure the parameters in the YML file
- Log on to the Elasticsearch console.
- In the left-side navigation pane, click Elasticsearch Clusters.
- Navigate to the desired cluster.
- In the top navigation bar, select a resource group and a region.
- In the left-side navigation pane, click Elasticsearch Clusters. On the Elasticsearch Clusters page, find the desired cluster and click its ID.
- In the left-side navigation pane of the page that appears, click Cluster Configuration.
- On the Cluster Configuration page, click Modify Configuration on the right side of YML Configuration.
- In the YML File Configuration panel, configure the following parameters.
Parameter Description Auto Indexing Specifies whether to automatically create an index when a new document is uploaded to your Elasticsearch cluster but no index exists. We recommend that you disable Auto Indexing because indexes created by this feature may not meet your business requirements.
This parameter corresponds to the action.auto_create_index configuration item in the YML file. The default value of this configuration item is false.
Index Deletion Specifies whether to specify the index name when you delete an index. If you set this parameter to Allow Wildcards, you can use wildcards to delete multiple indexes at a time. Deleted indexes cannot be recovered. Exercise caution when you configure this parameter.
This parameter corresponds to the action.destructive_requires_name configuration item in the YML file. The default value of this configuration item is true.
Audit Log Indexing If you enable Audit Log Indexing, the system generates audit logs when you create, delete, modify, or search for an index in your Elasticsearch cluster. These logs consume disk space and affect cluster performance. Therefore, we recommend that you disable Audit Log Indexing and exercise caution when you configure this parameter. For more information, see Configure the Audit Log Indexing feature.Notice This parameter is unavailable for Elasticsearch clusters of V7.0 or later.
This parameter corresponds to the xpack.security.audit.enabled configuration item in the YML file. The default value of this configuration item is false.
Watcher If you enable Watcher, you can use the X-Pack Watcher feature. You must clear the .watcher-history* index on a regular basis to free up disk space.
This parameter corresponds to the xpack.watcher.enabled configuration item in the YML file. The default value of this configuration item is false.
Other Configurations The following description provides some of the supported configuration items. Unless otherwise specified, these items are available for Elasticsearch V5.X, V6.X, and V7.X.
- Configure CORS
- Configure a remote reindex whitelist
- Configure the Audit Log Indexing feature
Notice The Audit Log Indexing feature is not supported in Elasticsearch V7.0 and later.
- Configure queue sizes
- thread_pool.bulk.queue_size (available for Elasticsearch V5.X)
- thread_pool.write.queue_size (available for Elasticsearch V6.X and V7.X)
- Configure a custom SQL plug-in
By default, Elasticsearch uses the built-in SQL plug-in provided by X-Pack. To upload a custom SQL plug-in, set xpack.sql.enabled to false.
- Before you configure the YML file, you must make sure that the cluster is in a normal state. After you configure the YML file, the system restarts the cluster. The time required for the restart depends on the size, data volume, and load of the cluster. We recommend that you configure the YML file during off-peak hours.
In most cases, if the indexes of your cluster have replica shards and the load of your cluster is normal, your cluster can still provide services during a cluster modification. The following items indicate that the load of a cluster is normal: The CPU utilization of the cluster is about 60%, the heap memory usage of the cluster is about 50%, and the value of NodeLoad_1m is less than the number of vCPUs for the cluster.
If the indexes of your cluster do not have replica shards, the load of the cluster is excessively high, and large amounts of data are written to or queried in your cluster, access timeouts may occur during a cluster modification. We recommend that you configure an access retry mechanism for your client before you perform a cluster modification. This reduces the impact on your business.
- Configure CORS
- Scroll down to the lower part of the panel, select This operation will restart the cluster. Continue?, and then click OK.
Then, the system restarts the Elasticsearch cluster. You can view the restart progress in the Tasks dialog box. After the cluster is restarted, the configurations in the YML file are updated.
|http.cors.enabled||false||Specifies whether to enable CORS. CORS allows browsers on other origins to access your Elasticsearch cluster. Valid values:
|http.cors.allow-origin||""||Specifies the origins from which requests are allowed. By default, no origin is allowed. You can set this parameter to a regular expression, such as /https?:\/\/localhost(:[0-9]+)?/. This regular expression indicates that the system responds to requests that match the regular expression.
Notice Asterisks ( *) are valid characters but may cause security risks. If an asterisk is used, your cluster is open to all origins. We recommend that you do not use asterisks.
|http.cors.max-age||1728000 (20 days)||Browsers can send OPTIONS requests to query CORS configurations. This parameter specifies the cache duration of the retrieved CORS configurations. Unit: seconds.|
|http.cors.allow-methods||OPTIONS, HEAD, GET, POST, PUT, DELETE||Specifies the request methods.|
|http.cors.allow-headers||X-Requested-With, Content-Type, Content-Length||Specifies the request headers.|
|http.cors.allow-credentials||false||The credential configuration item. This item specifies whether Access-Control-Allow-Credentials can be contained in the response header. Valid values:
Configure a remote reindex whitelist
|reindex.remote.whitelist||||Specifies the remote cluster of the current cluster.
Specify a remote cluster in the format of Host address:Port number. If you want to specify multiple remote clusters, separate them with commas (,), such as otherhost:9200,another:9200,127.0.10.**:9200,localhost:**. The whitelist does not identify protocols.
- Single-zone cluster
- Multi-zone cluster
reindex.remote.whitelist: ["10.0.xx.xx:9200","10.0.xx.xx:9200","10.0.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200"]Note After the remote reindex whitelist is configured, you can call the reindex API to reindex data. For more information, see Use the reindex operation to migrate data.
Configure the Audit Log Indexing feature
xpack.security.audit.index.bulk_size: 5000 xpack.security.audit.index.events.emit_request_body: false xpack.security.audit.index.events.exclude: run_as_denied,anonymous_access_denied,realm_authentication_failed,access_denied,connection_denied xpack.security.audit.index.events.include: authentication_failed,access_granted,tampered_request,connection_granted,run_as_granted xpack.security.audit.index.flush_interval: 180s xpack.security.audit.index.rollover: hourly xpack.security.audit.index.settings.index.number_of_replicas: 1 xpack.security.audit.index.settings.index.number_of_shards: 10
|Configuration item||Default value||Description|
|xpack.security.audit.index.bulk_size||1000||You can write audit events to audit log indexes in batches. This parameter specifies the maximum number of audit events that can be written in each batch.|
|xpack.security.audit.index.flush_interval||1s||Specifies the frequency at which buffered audit events are flushed to audit log indexes.|
|xpack.security.audit.index.rollover||daily||Specifies the frequency at which audit events are rolled over to a new audit log index. Valid values: hourly, daily, weekly, and monthly.|
|xpack.security.audit.index.events.include||access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted||Specifies the types of audit events that can be written to audit log indexes. For more information about audit event types, see Audit Event Types.|
|xpack.security.audit.index.events.exclude||null, which indicates that the system does not process audit events||Specifies the types of audit events that cannot be written to audit log indexes.|
|xpack.security.audit.index.events.emit_request_body||false||Specifies whether to ignore or include REST request bodies when a specific audit event is triggered, such as authentication_failed.|
- If an audit event contains the request bodies, sensitive data may be exposed.
- After the Audit Log Indexing feature is enabled, audit events are stored in the audit log indexes of your cluster. The names of the indexes start with .security_audit_log-. These indexes consume the storage of your cluster. Elasticsearch does not automatically clear expired indexes. You must manually clear expired audit log indexes.
xpack.security.audit.index.settings: index: number_of_shards: 1 number_of_replicas: 1
For more information, see Auditing Security Settings.
Configure queue sizes
thread_pool.bulk.queue_size: 500 thread_pool.write.queue_size: 500 thread_pool.search.queue_size: 1200
|thread_pool.bulk.queue_size||200||The size of the document write queue. This parameter is available for Elasticsearch V5.X.|
|thread_pool.write.queue_size||200||The size of the document write queue. This parameter is available for Elasticsearch V6.X and V7.X.|
|thread_pool.search.queue_size||1000||The size of the document search queue.|