All Products
Search
Document Center

Elasticsearch:Configure the YML file

Last Updated:Aug 29, 2024

In the YML Configuration section of the Cluster Configuration page of your Alibaba Cloud Elasticsearch cluster, you can enable the Auto Indexing, Audit Log Indexing, or Watcher feature. You can also configure Index Deletion and Other Configurations. This topic describes how to configure the following items: parameters in the YML file of the cluster, cross-origin resource sharing (CORS), a remote reindex whitelist, the Audit Log Indexing feature, and queue sizes.

Precautions

The network architecture of Alibaba Cloud Elasticsearch was adjusted in October 2020. Due to this adjustment, you cannot use the reindex API to migrate data between clusters in some scenarios. For more information, see the "Limits" section in Migrate data from a self-managed Elasticsearch cluster to an Alibaba Cloud Elasticsearch cluster deployed in the new network architecture.

Note

The time when the network architecture in the China (Zhangjiakou) region and the regions outside China was adjusted is uncertain. If you want to perform the preceding operations between a cluster created before October 2020 and a cluster created in October 2020 or later in such a region, submit a ticket to contact Alibaba Cloud technical support to check whether the clusters are deployed in the same network architecture.

Configure the parameters in the YML file

  1. Log on to the Alibaba Cloud Elasticsearch console.
  2. In the left-side navigation pane, click Elasticsearch Clusters.
  3. Navigate to the desired cluster.
    1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
    2. On the Elasticsearch Clusters page, find the cluster and click its ID.
  4. In the left-side navigation pane of the page that appears, choose Configuration and Management > Cluster Configuration.

  5. On the Cluster Configuration page, click Modify Configuration on the right side of YML File Configuration.

  6. In the YML File Configuration panel, configure the following parameters.

    Note

    If you want to view the content of the elasticsearch.yml file, you can log on to the Kibana console of your cluster and run the GET _cluster/settings?include_defaults command.

    Parameter

    Description

    Auto Indexing

    Specifies whether to automatically create an index if a new document is uploaded to the Elasticsearch cluster but no index exists.

    This parameter corresponds to the action.auto_create_index configuration item in the YML file. The default value of this configuration item is false.

    By default, the system is prohibited from automatically creating indexes. You can use one of the following methods to enable the system to automatically create indexes:

    Important

    The indexes that are automatically created may not meet your business requirements. Therefore, we recommend that you perform evaluation before you enable the system to automatically create indexes.

    • Set Auto Indexing to Enable in the YML File Configuration panel. This operation is a static configuration and will trigger a cluster restart.

    • Use a dynamic mode to quickly enable the system to automatically create indexes. This does not trigger a cluster restart. You can log on to the Kibana console of the cluster and run one of the following commands:

      • Enable the system to automatically create all indexes

        PUT /_cluster/settings
        {
          "persistent": {
            "action": {
              "auto_create_index": "true"
            }
          }
        }
        Important

        After you run the preceding command, the system can automatically create all indexes. If you want to prohibit the system from automatically creating indexes, set auto_create_index to false in the command.

      • Enable the system to automatically create only specific indexes (The following command provides an example on how to enable the system to automatically create only system indexes.)

        PUT /_cluster/settings
        {
          "persistent": {
            "action": {
              "auto_create_index": "+.*,-*"
            }
          }
        }

    Index Deletion

    Specifies whether to specify the index name when you delete an index. If you set this parameter to Allow Wildcards, you can use wildcards to delete multiple indexes at a time. Deleted indexes cannot be recovered. Proceed with caution.

    This parameter corresponds to the action.destructive_requires_name configuration item in the YML file. The default value of this configuration item is true.

    Auditlog

    If you enable Audit Log Indexing, the system generates audit logs for the create, delete, modify, and search operations that are performed on the Elasticsearch cluster. These logs consume disk space and affect cluster performance. Therefore, we recommend that you disable Audit Log Indexing and proceed with caution. For more information about the parameters related to the Audit Log Indexing feature, see Configure the Audit Log Indexing feature.

    Important

    In some regions, you can view the audit logs of Elasticsearch clusters of V7.X or later in the Elasticsearch console. For more information about the regions, see Limits. Before you view the audit logs of a cluster, you must make sure that the Audit Log Indexing feature is enabled for the cluster. For more information, see Query logs. For other Elasticsearch clusters, you need to run the related commands to view their audit logs. For example, to view the audit logs of such a cluster, you can log on to the Kibana console of the cluster and run the related command in the Kibana console to view the indexes whose names start with .security_audit_log-.

    This parameter corresponds to the xpack.security.audit.enabled configuration item in the YML file. The default value of this configuration item is false.

    Watcher

    After you enable Watcher, you can use the X-Pack Watcher feature. You must clear the .watcher-history* index on a regular basis to free up disk space.

    This parameter corresponds to the xpack.watcher.enabled configuration item in the YML file. The default value of this configuration item is false.

    Other Configurations

    The following descriptions provide some of the supported configuration items. Unless otherwise specified, these items are available for Elasticsearch V5.X, V6.X, and V7.X clusters.

    • Configure CORS

      • http.cors.enabled

      • http.cors.allow-origin

      • http.cors.max-age

      • http.cors.allow-methods

      • http.cors.allow-headers

      • http.cors.allow-credentials

    • Configure a remote reindex whitelist

      reindex.remote.whitelist

    • Configure the Audit Log Indexing feature

      Elasticsearch V7.X and V8.X clusters support only the xpack.security.audit.logfile.events.include parameter. Elasticsearch V5.X and V6.X clusters support the following parameters:

      • xpack.watcher.enabled

      • xpack.notification

      • xpack.security.audit.enabled

      • xpack.security.audit.index.bulk_size

      • xpack.security.audit.index.flush_interval

      • xpack.security.audit.index.rollover

      • xpack.security.audit.index.events.include

      • xpack.security.audit.index.events.exclude

      • xpack.security.audit.index.events.emit_request_body

      • xpack.security.audit.index.settings.index

    • Configure the Lightweight Directory Access Protocol (LDAP) authentication feature

      Elasticsearch clusters of a version other than V5.X support the following parameters:

      • xpack.security.authc.realms.ldap1

      • xpack.security.authc.realms.active_directory1

      • xpack.security.authc.realms.pki1

      • xpack.security.authc.realms.saml1

      • xpack.security.authc.realms.kerberos1

      • xpack.security.authc.token.enabled

    • Configure queue sizes

      • thread_pool.bulk.queue_size (available for Elasticsearch V5.X and V6.X clusters)

      • thread_pool.write.queue_size (available for Elasticsearch V6.X, V7.X, and V8.X clusters)

      • thread_pool.search.queue_size

    • Configure a custom SQL plug-in

      xpack.sql.enabled

      By default, Elasticsearch clusters use the built-in SQL plug-in provided by X-Pack. If you want to upload a custom SQL plug-in to your Elasticsearch cluster, set xpack.sql.enabled to false.

    Important
    • After you configure the YML file of your cluster, the system performs a rolling restart for the cluster. If the indexes of your cluster have replica shards and the load of your cluster is normal, your cluster can still provide services during the restart. The following items indicate that the load of a cluster is normal: The CPU utilization of each node in the cluster is about 60%, the heap memory usage of each node in the cluster is about 50%, and the value of NodeLoad_1m for each node is less than the number of vCPUs for the node. The time required for the restart depends on the size, data volume, and load of the cluster. We recommend that you configure the YML file of a cluster during off-peak hours.

    • If the indexes of your cluster do not have replica shards, the load of the cluster is excessively high, and large amounts of data are written to or queried in your cluster, access to the cluster may time out during a cluster modification. We recommend that you configure an access retry mechanism for your client before you perform a cluster modification. This helps reduce the impact on your business.

  7. Select This operation will restart the cluster. Continue? and click OK.

    Then, the system restarts the Elasticsearch cluster. You can view the restart progress in the Tasks dialog box. After the cluster is restarted, the configurations in the YML file are updated.

Configure CORS

You can configure CORS to allow browsers on other origins to access your Elasticsearch cluster. CORS can be configured in the YML File Configuration panel. The following table lists the parameters that you can configure.

Important
  • The parameters in the following table are custom parameters provided by Alibaba Cloud Elasticsearch to support HTTP.

  • The parameters in the following table support only static configurations. For the parameter settings to take effect, you must add the settings to the elasticsearch.yml file.

  • The parameters in the following table depend on the network settings of your Elasticsearch cluster.

Parameter

Default value

Description

http.cors.enabled

false

Specifies whether to enable CORS. CORS allows browsers on other origins to access your Elasticsearch cluster. Valid values:

  • true: indicates that CORS is enabled. In this case, the system can process OPTIONS CORS requests. If the origin in a request is declared in http.cors.allow-origin, the system returns a response whose header contains Access-Control-Allow-Origin.

  • false: indicates that CORS is disabled. In this case, the system ignores the origin in the request header and returns a response whose header does not contain Access-Control-Allow-Origin. If a client cannot send a pre-flight request that has origin information included in the request header or does not verify Access-Control-Allow-Origin in the response header returned from the server, cross-origin access security is compromised. If CORS is disabled, a client can send only an OPTIONS request to check whether Access-Control-Allow-Origin exists.

http.cors.allow-origin

""

The origins from which requests are allowed. By default, requests from origins are not allowed. You can set this parameter to a regular expression, such as /https?:\/\/localhost(:[0-9]+)?/. This regular expression indicates that the system responds to requests that match the regular expression.

Warning

Asterisks (*) are valid characters but may cause security risks. If an asterisk is used, your cluster is open to all origins. We recommend that you do not use asterisks.

http.cors.max-age

1728000 (20 days)

Browsers can send OPTIONS requests to query CORS configurations. This parameter specifies the cache duration of the retrieved CORS configurations. Unit: seconds.

http.cors.allow-methods

OPTIONS, HEAD, GET, POST, PUT, DELETE

The request methods.

http.cors.allow-headers

X-Requested-With, Content-Type, Content-Length

The request headers.

http.cors.allow-credentials

false

The credential configuration item. This item specifies whether Access-Control-Allow-Credentials can be contained in the response header. Valid values:

  • true: indicates that Access-Control-Allow-Credentials can be contained in the response header.

  • false: indicates that Access-Control-Allow-Credentials cannot be contained in the response header.

Configure a remote reindex whitelist

Before you call the reindex API to migrate index data from a remote Elasticsearch cluster to the current Elasticsearch cluster, you must configure a remote reindex whitelist for the current Elasticsearch cluster. You can configure the whitelist in the YML File Configuration panel. The following table describes the parameter that you can configure.

Parameter

Default value

Description

reindex.remote.whitelist

[]

The address of the remote cluster.

Specify the address in the Host IP address:Port number format. If you want to specify multiple addresses, separate them with commas (,), such as otherhost:9200,another:9200,127.0.10.**:9200,localhost:**. The whitelist does not identify protocols.

If the remote cluster is a single-zone Elasticsearch cluster, specify the address of the cluster in the <Domain name of the cluster>:9200 format. If the remote cluster is a multi-zone Elasticsearch cluster, specify the addresses of the cluster by using the IP addresses of all data nodes in the cluster and the port number of the cluster. The following descriptions provide configuration examples:

  • Single-zone cluster单可用区配置示例

    reindex.remote.whitelist: ["es-cn-09k1rgid9000g****.elasticsearch.aliyuncs.com:9200"]
  • Multi-zone cluster多可用区reindex白名单配置

    reindex.remote.whitelist: ["10.0.xx.xx:9200","10.0.xx.xx:9200","10.0.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200"]
    Note

    After the remote reindex whitelist is configured, you can call the reindex API to reindex data. For more information, see Use the reindex API to migrate data.

Configure the Audit Log Indexing feature

The Audit Log Indexing feature is disabled by default. If you want to view the audit logs of an Elasticsearch cluster, you must enable this feature for the cluster. After the feature is enabled for the cluster, the system generates audit logs for operations that are performed on the cluster, such as the create, delete, modify, and query operations. For different versions of Elasticsearch clusters, the operations for enabling and configuring the Audit Log Indexing feature and viewing audit logs vary. The following descriptions provide detailed operations.

Note

For more information about the Audit Log Indexing feature, see Auditing Security Settings.

Elasticsearch cluster of V7.X or later

  1. Go to the YML File Configuration panel.

    For more information, see Configure the parameters in the YML file.

  2. Set Audit Log Indexing to Enable to enable the Audit Log Indexing feature.

  3. Configure the Audit Log Indexing feature.

    After the Audit Log Indexing feature is enabled, the default configuration of the xpack.security.audit.logfile.events.include parameter appears in Other Configurations. You can modify the configuration based on your business requirements. The following code provides an example:

    xpack:
      security:
        audit:
          logfile:
            events:
              include: >-
                access_denied,anonymous_access_denied,authentication_failed,connection_denied,tampered_request,run_as_denied,run_as_granted
    Important
    • Clusters of V7.X or later support only the xpack.security.audit.logfile.events.include parameter.

    • If you use the default configuration of the Audit Log Indexing feature for your cluster, the system generates audit logs for only rejected or failed requests that are sent to the cluster. If you want to obtain the audit logs of successful requests, you must add the access_granted event for the cluster. After the event is added, all access information about the cluster is stored, which increases the disk usage of the cluster. Therefore, we recommend that you disable the Audit Log Indexing feature after you troubleshoot issues.

  4. View the audit logs of the cluster.

    After you enable the Audit Log Indexing feature for a cluster of V7.X or later, you can view the audit logs of the cluster on the Logs page of the Elasticsearch console. For more information, see Query logs. The Audit Log Indexing feature is available only in some regions. For more information about the regions, see Limits.

Elasticsearch V5.X and V6.X clusters

  1. Go to the YML File Configuration panel.

    For more information, see Configure the parameters in the YML file.

  2. Set Audit Log Indexing to Enable to enable the Audit Log Indexing feature.

    The following code provides the default configuration of the Audit Log Indexing feature. You can modify the configuration based on your business requirements.

    xpack.security.audit.index.bulk_size: 5000
    xpack.security.audit.index.events.emit_request_body: false
    xpack.security.audit.index.events.exclude: run_as_denied,anonymous_access_denied,realm_authentication_failed,access_denied,connection_denied
    xpack.security.audit.index.events.include: authentication_failed,access_granted,tampered_request,connection_granted,run_as_granted
    xpack.security.audit.index.flush_interval: 180s
    xpack.security.audit.index.rollover: hourly
    xpack.security.audit.index.settings.index.number_of_replicas: 1
    xpack.security.audit.index.settings.index.number_of_shards: 10

    Parameter

    Default value

    Description

    xpack.security.audit.index.bulk_size

    1000

    You can write audit events to an audit log index in batches. This parameter specifies the maximum number of audit events that can be written in each batch.

    xpack.security.audit.index.flush_interval

    1s

    The frequency at which buffered audit events are flushed to the audit log index.

    xpack.security.audit.index.rollover

    daily

    The frequency at which audit events are rolled over to a new audit log index. Valid values: hourly, daily, weekly, and monthly.

    xpack.security.audit.logfile.events.include

    access_denied,anonymous_access_denied,authentication_failed, connection_denied,tampered_request,run_as_denied,run_as_granted

    The types of audit events that can be collected to audit logs. The Audit Log Indexing feature is available only in some regions. For more information about the regions, see Limits. For more information about supported audit event types, see Audit event types (7.x).

    xpack.security.audit.index.events.include

    access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted

    The types of audit events that can be written to audit log indexes. This parameter is available only for Elasticsearch V5.X and V6.X clusters. For more information about supported audit event types, see Audit event types (6.x).

    xpack.security.audit.index.events.exclude

    null, which indicates that the system does not process audit events

    The types of audit events that cannot be written to audit log indexes.

    xpack.security.audit.index.events.emit_request_body

    false

    Specifies whether to ignore or include REST request bodies when a specific audit event is triggered, such as authentication_failed.

    Warning

    If an audit event contains a request body, sensitive data may be exposed.

  3. View the audit logs of the cluster.

    After the Audit Log Indexing feature is enabled for a V5.X or V6.X cluster, audit events are stored in the audit log indexes of the cluster. The names of the indexes start with .security_audit_log-. You can view the audit logs of the cluster by accessing these indexes in the Kibana console of the cluster.

    Important

    Audit log indexes consume the storage of a cluster, and Elasticsearch does not automatically clear expired indexes. Therefore, you must manually clear expired audit log indexes.

  4. (Optional) Configure shards for audit log indexes.

    You can use the xpack.security.audit.index.settings parameter to configure shards for audit log indexes in a V5.X or V6.X cluster. The following code provides a configuration example. In this example, the numbers of primary shards and replica shards for the indexes are set to 1.

    xpack.security.audit.index.settings:
      index:
        number_of_shards: 1
        number_of_replicas: 1
    Note

    If you want to customize the configurations for audit log indexes, add the configurations to the YML file of your cluster after you set xpack.security.audit.enabled to true to enable Audit Log Indexing. If you do not customize the configurations, the system uses the default settings number_of_shards: 5 and number_of_replicas: 1 to create the indexes.

Configure queue sizes

You can customize queue sizes to adjust the sizes of the document write queue and document search queue. You can specify the queue sizes in the YML File Configuration panel. The following code provides configuration examples. In the examples, the size of the document write queue is set to 500 and that of the document search queue is set to 1000. You can change these values based on your business requirements.

  • Elasticsearch V5.X and V6.X clusters

    thread_pool.bulk.queue_size: 500
    thread_pool.search.queue_size: 1000
  • Elasticsearch V6.X, V7.X, and V8.X clusters

    thread_pool.write.queue_size: 500
    thread_pool.search.queue_size: 1000

Parameter

Default value

Description

thread_pool.bulk.queue_size

200

The size of the document write queue. This parameter is available for Elasticsearch V5.X and V6.X clusters.

thread_pool.write.queue_size

200

The size of the document write queue. This parameter is available for Elasticsearch V6.X, V7.X, and V8.X clusters.

thread_pool.search.queue_size

1000

The size of the document search queue.

Note

The maximum value of this parameter is 1000. If you set this parameter to a value greater than 1000, the value 1000 is used.