Configure the YML file - Elasticsearch - Alibaba Cloud Documentation Center

This topic describes how to configure various settings for your Alibaba Cloud Elasticsearch cluster by modifying its YML configuration. You can enable features like Auto Indexing, Audit Log Indexing, and Watcher, and adjust parameters such as CORS, remote reindex whitelists, and queue sizes.

Usage notes

Cluster restart required:
Most YML file changes (especially static configurations) require a rolling restart of your cluster to take effect.s
- Impact: If your indices have replica shards and the cluster load is normal (CPU ~60%, Heap ~50%, NodeLoad_1m < vCPU count), services can remain available during the restart. However, if your cluster lacks replicas, has high load, or is experiencing heavy data writes/queries, timeouts may occur.
- Recommendation: Perform YML configurations during off-peak hours. Implement an access retry mechanism on your client applications to minimize service impact during restarts.
Monitoring: Monitor restart progress in the Tasks dialog box.
Cross-cluster reindex: Alibaba Cloud Elasticsearch adjusted its control architecture in October 2020. This may affect data migration between clusters (especially older vs. newer ones) using the reindex API.
- Action: If you need to reindex data between a cluster created before October 2020 and one created in October 2020 or later in certain regions (e.g., China (Zhangjiakou) and regions outside China), submit a ticket to Alibaba Cloud technical support to verify network compatibility.
- Reference: See the "Limitations" section in Migrate data from a self-managed Elasticsearch cluster to Alibaba Cloud Elasticsearch over a private connection.

Access YML configuration

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation menu, choose Elasticsearch Clusters.
Navigate to the target cluster.
1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
2. On the Elasticsearch Clusters page, find the cluster and click its ID.
In the left navigation pane for your cluster, choose Configuration and Management > Cluster Configuration.
On the Cluster Configuration page, click Modify Configuration in the YML File Configuration section.
In the YML File Configuration panel, configure the parameters.
See Core YML configuration parameters and Other advanced YML configurations.
After making changes, select This operation will restart the cluster. Continue? and click OK.
The cluster will restart to apply the new configurations.

Core YML configuration parameters

Auto Indexing

This parameter controls whether Elasticsearch automatically creates an index when a document is uploaded for an index that does not yet exist. Corresponds to action.auto_create_index in the YML file. Default: false (prohibits auto-creation). Here are some methods to enable auto indexing:

Important

Automatically created indexes may not align with your business requirements (e.g., shard count, mappings). Evaluate risks before enabling.

Elasticsearch console

Set Auto Indexing to Enable in the YML File Configuration panel. This is a static configuration and triggers a cluster restart.

Kibana

Enable/Disable the parameter via Kibana (Dev Tools) without a restart.

Enable for all indices:

json

PUT /_cluster/settings
{  "persistent": {    "action": {      "auto_create_index": "true"    }  }}

Disable for all indices: (Set auto_create_index to false)

json

PUT /_cluster/settings
{  "persistent": {    "action": {      "auto_create_index": "false"    }  }}

Enable for specific indices (e.g., system indices):

json

PUT /_cluster/settings
{  "persistent": {    "action": {      "auto_create_index": "+.*,-*"    }  }}

Index Deletion

Specifies whether you can use wildcards (*) to delete multiple indices at once without explicitly naming each index. Corresponds to action.destructive_requires_name in the YML file. Default: true (requires explicit name).

In Console: Set Index Deletion to Allow Wildcards.
Deleted indexes cannot be recovered. Proceed with extreme caution when allowing wildcard deletions.

Audit Log Indexing

Enabling this feature generates audit logs for create, delete, modify, and search operations, providing a security trail. Corresponds to xpack.security.audit.enabled in the YML file. Default: false.

In Console: Set Audit Log Indexing to Enable in the YML File Configuration panel.
Impact: Audit logs consume disk space and can affect cluster performance. We recommend disabling Audit Log Indexing when not actively troubleshooting.
Viewing Logs:
- V7.X or later: In some regions, you can view audit logs directly on the Logs page of the Elasticsearch console. See Query logs and Limits for region specifics.
- V5.X and V6.X: Audit logs are stored in indices named .security_audit_log-*. You can query these indices directly in Kibana Dev Tools.

For details, see Detailed audit log indexing configuration.

Watcher

Enables the X-Pack Watcher feature, allowing you to create alerts and notifications based on events in your cluster. Corresponds to xpack.watcher.enabled in the YML file. Default: false.

In Console: Set Watcher to Enable in the YML File Configuration panel.
Management: You must regularly clear the .watcher-history* index to manage disk space if Watcher is enabled.

Other advanced YML configurations

Cross-origin resource sharing (CORS)

Configure CORS to allow web browsers from different origins to access your Elasticsearch cluster. These are static configurations and require a cluster restart. They depend on your cluster's network settings.

Parameter	Default value	Description
http.cors.enabled	false	Specifies whether to enable CORS. Valid values: `true`: Allow processing of `OPTIONS` CORS requests and return `Access-Control-Allow-Origin` if the origin is allowed. `false`: Disable CORS, ignoring origins and not returning the `Access-Control-Allow-Origin` header.
http.cors.allow-origin	""	The origins (URLs) from which requests are allowed. Can be a regular expression (e.g., `/https?:\/\/localhost(:[0-9]+)?/`). Warning Using asterisks (``) is valid but creates significant security risks by opening your cluster to all origins. It is strongly recommended to avoid ``.
http.cors.max-age	1728000 (20 days)	The duration (in seconds) for which browsers can cache retrieved CORS configurations from `OPTIONS` requests. (20 days by default).
http.cors.allow-methods	OPTIONS, HEAD, GET, POST, PUT, DELETE	The HTTP request methods allowed for cross-origin requests.
http.cors.allow-headers	X-Requested-With, Content-Type, Content-Length	The HTTP request headers allowed for cross-origin requests.
http.cors.allow-credentials	false	Specifies whether `Access-Control-Allow-Credentials` can be included in the response header. Set to `true` to allow it.

Remote reindex whitelist

Before using the reindex API to migrate data from a remote Elasticsearch cluster, you must configure a whitelist of allowed remote cluster addresses.

Parameter	Default value	Description
reindex.remote.whitelist	[]	A list of remote cluster addresses in `Host_IP_address:Port_number` format. Separate multiple addresses with commas (e.g., `otherhost:9200,another:9200,127.0.10.:9200,localhost:*`). This whitelist does not identify protocols (HTTP/HTTPS).

Configuration examples:

Single-zone cluster

reindex.remote.whitelist: ["es-cn-09k1rgid9000g****.elasticsearch.aliyuncs.com:9200"]

Multi-zone cluster: Specify IP addresses of all data nodes.
```
reindex.remote.whitelist: ["10.0.xx.xx:9200","10.0.xx.xx:9200","10.0.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200"]
```
For more information on data migration using the reindex API, see Use the reindex API to migrate data between Alibaba Cloud Elasticsearch clusters.

Detailed audit log indexing configuration

For Elasticsearch V5.X and V6.X clusters, you can customize various audit log indexing parameters in Other Configurations of the YML File Configuration panel. For V7.X+, only xpack.security.audit.logfile.events.include is supported via console YML.

Note

For more information about the Audit Log Indexing feature, see Auditing Security Settings.

V7.X or later

Go to the YML File Configuration panel.
For more information, see Access YML configuration.
Set Audit Log Indexing to Enable.

In the Other Configurations section, configure the audit log indexing feature.

Example:

xpack:
  security:
    audit:
      logfile:
        events:
          include: >-
            access_denied,anonymous_access_denied,authentication_failed,connection_denied,tampered_request,run_as_denied,run_as_granted

Note

To obtain the audit logs of successful requests, add the access_granted event for the cluster.

View the audit logs of the cluster.

V5.X and V6.X clusters

Go to the YML File Configuration panel.

Set Audit Log Indexing to Enable.

Example default configuration:

xpack.security.audit.index.bulk_size: 5000
xpack.security.audit.index.events.emit_request_body: false
xpack.security.audit.index.events.exclude: run_as_denied,anonymous_access_denied,realm_authentication_failed,access_denied,connection_denied
xpack.security.audit.index.events.include: authentication_failed,access_granted,tampered_request,connection_granted,run_as_granted
xpack.security.audit.index.flush_interval: 180s
xpack.security.audit.index.rollover: hourly
xpack.security.audit.index.settings.index.number_of_replicas: 1
xpack.security.audit.index.settings.index.number_of_shards: 10

Parameter	Default value	Description
xpack.security.audit.index.bulk_size	1000	Maximum number of audit events to write in each batch to the audit log index.
xpack.security.audit.index.flush_interval	1s	Frequency at which buffered audit events are flushed to the audit log index.
xpack.security.audit.index.rollover	daily	Frequency at which audit events are rolled over to a new audit log index. Valid values: `hourly`, `daily`, `weekly`, `monthly`.
xpack.security.audit.logfile.events.include	access_denied,anonymous_access_denied,authentication_failed, connection_denied,tampered_request,run_as_denied,run_as_granted	Types of audit events to collect and write to audit log indices. For supported types, see Audit event types (7.x).
xpack.security.audit.index.events.include	access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted	Types of audit events to collect and write to audit log indices. For supported types, see Audit event types (6.x).
xpack.security.audit.index.events.exclude	null, which indicates that the system does not process audit events	Types of audit events to exclude from writing to audit log indices.
xpack.security.audit.index.events.emit_request_body	false	Specifies whether to include REST request bodies when specific audit events are triggered (e.g., `authentication_failed`).

Warning

If an audit event contains a request body, sensitive data may be exposed.

View the audit logs of the cluster.
Audit events are stored in the audit log indexes of the cluster. The names of the indexes start with .security_audit_log-. You can view the audit logs of the cluster by accessing these indexes in the Kibana console of the cluster.
Note
Audit log indexes consume storage and are not automatically cleared by Elasticsearch. You must manually clear expired audit log indices.
(Optional) Configure shards for audit log indexes.
You can configure the number of primary and replica shards for audit log indices using xpack.security.audit.index.settings. Example:
```
xpack.security.audit.index.settings:
  index:
    number_of_shards: 1
    number_of_replicas: 1
```
Note
If not customized, default settings (number_of_shards: 5, number_of_replicas: 1) are used.

Queue sizes

Customize the sizes of the thread pool queues for write (bulk) and search operations to optimize cluster performance under different loads.

Parameters:

Parameter	Default value	Description
thread_pool.bulk.queue_size	200	Size of the document write queue (for bulk operations). Available for Elasticsearch V5.X and V6.X clusters.
thread_pool.write.queue_size	200	Size of the document write queue. Available for Elasticsearch V6.X, V7.X, and V8.X clusters.
thread_pool.search.queue_size	1000	Size of the document search queue. Note The maximum effective value for this parameter is 1000. If set higher in the YML file, 1000 will be used. For special use cases, contact technical support.

Configuration examples:

Elasticsearch V5.X and V6.X clusters

thread_pool.bulk.queue_size: 500
thread_pool.search.queue_size: 1000

Elasticsearch V6.X, V7.X, and V8.X clusters

thread_pool.write.queue_size: 500
thread_pool.search.queue_size: 1000

LDAP authentication configuration

For Elasticsearch clusters (V5.X excluded), you can configure Lightweight Directory Access Protocol (LDAP) authentication and other realms. Parameters for this are set in Other Configurations.

Supported parameters (Examples):
- xpack.security.authc.realms.ldap1
- xpack.security.authc.realms.active_directory1
- xpack.security.authc.realms.pki1
- xpack.security.authc.realms.saml1
- xpack.security.authc.realms.kerberos1
- xpack.security.authc.token.enabled

Custom SQL plugin configuration

By default, Alibaba Cloud Elasticsearch uses the built-in SQL plugin provided by X-Pack. If you need to use a custom SQL plugin, you must disable the built-in one.

Parameter: xpack.sql.enabled
Action: Set xpack.sql.enabled to false in Other Configurations to use a custom SQL plugin.