Elasticsearch:Configure YML parameters

Precautions

Modify the configuration

1. 1. You can go to the instance details page.

      1. Log on to the Alibaba Cloud Elasticsearch console.

      2. In the left navigation pane, click Elasticsearch Clusters .

      3. In the top menu bar, select a resource group and a region.

      4. On the Elasticsearch Instances page, click the ID of the target instance to go to its details page.

2. You can go to the YML file configuration page.

   1. In the navigation pane on the left, click Configuration and Management > Cluster Configuration.

   2. On the ES cluster configuration page, next to YML File Configuration, click Modify Configuration.

3. In the YML File Configuration dialog box, configure the parameters.

   Note

   To view the contents of elasticsearch.yml, log on to the Kibana console and run the GET _cluster/settings?include_defaults command.

   Parameter	Description
   Auto Indexing	Specifies whether the system can automatically create an index if one does not exist when the Elasticsearch instance receives a new document. This corresponds to the action.auto_create_index configuration item in the YML file. The default value is false. By default, Alibaba Cloud Elasticsearch does not allow automatic index creation. You can enable it in one of the following ways: Important Automatically created indexes may not meet your expectations. We recommend that you evaluate the impact before you enable this feature. Enable it from the Cluster Configuration page in the console. This is a static YML configuration and triggers an instance restart. Enable it quickly using a dynamic method that does not require a restart. Log on to the Kibana console and use the following commands to allow automatic index creation: Allow automatic creation of all indexes PUT /_cluster/settings { "persistent": { "action": { "auto_create_index": "true" } } } Important This method enables automatic creation for all indexes. To disable it, change true to false. Allow automatic creation of only specified indexes. The following example allows automatic creation of only system indexes: PUT /_cluster/settings { "persistent": { "action": { "auto_create_index": "+.*,-*" } } }
   Index Deletion	Specifies whether you must explicitly specify an index name when deleting an index. If you select Allow Wildcards, you can use wildcard characters to delete indexes in batches. Deleted indexes cannot be recovered. Use this configuration with caution. This corresponds to the action.destructive_requires_name configuration item in the YML file. The default value is true.
   Audit Log Indexing	When enabled, the system records audit logs for operations such as create, delete, update, and query on the Elasticsearch instance. This log information consumes disk space and affects performance. We recommend that you do not enable it. Use this configuration with caution. For more information about the parameters, see Configure Auditlog (audit log). Important For Elasticsearch 7.x and later, you can view audit logs in the console. This feature is available only in some regions. For more information, see Limits. To view the logs, you must first enable Auditlog. For more information, see Query logs. For other versions, you must view audit logs in the cluster. For example, you can view audit logs by checking indexes that start with .security_audit_log-* in the Kibana console. This corresponds to the xpack.security.audit.enabled configuration item in the YML file. The default value is false.
   Watcher	When enabled, you can use the Watcher feature of X-Pack. Regularly purge the .watcher-history* indexes to avoid consuming a large amount of disk space. This corresponds to the xpack.watcher.enabled configuration item in the YML file. The default value is false.
   Other Configurations	The following are some of the supported configuration items. Unless a specific Elasticsearch version is indicated, these items are compatible with Elasticsearch 5.x, 6.x, and 7.x. Configure CORS access http.cors.enabled http.cors.allow-origin http.cors.max-age http.cors.allow-methods http.cors.allow-headers http.cors.allow-credentials Configure Reindex whitelist reindex.remote.whitelist Configure Auditlog (audit log) Elasticsearch 7.x and 8.x versions only support configuring the xpack.security.audit.logfile.events.include parameter. Versions 5.x and 6.x support the following parameters: xpack.watcher.enabled xpack.notification xpack.security.audit.enabled xpack.security.audit.index.bulk_size xpack.security.audit.index.flush_interval xpack.security.audit.index.rollover xpack.security.audit.index.events.include xpack.security.audit.index.events.exclude xpack.security.audit.index.events.emit_request_body xpack.security.audit.index.settings.index LDAP feature Supported by all versions except 5.x: xpack.security.authc.realms.ldap1 xpack.security.authc.realms.active_directory1 xpack.security.authc.realms.pki1 xpack.security.authc.realms.saml1 xpack.security.authc.realms.kerberos1 xpack.security.authc.token.enabled Configure queue size thread_pool.bulk.queue_size (for versions 5.x and 6.x) thread_pool.write.queue_size (for versions 6.x, 7.x, and 8.x) thread_pool.search.queue_size Custom SQL plugin configuration xpack.sql.enabled By default, an Elasticsearch instance enables the SQL plugin that comes with X-Pack. To upload a custom SQL plugin, set xpack.sql.enabled to false.
   Forced Update	Controls whether to forcibly apply YML configuration changes. Valid values: Close: This option does not force changes. You can select the desired change method, such as in-place or blue-green changes. The system verifies the cluster's health status, including node availability and shard allocation, to ensure the safety of the operation. Enable: Changes are forced. The system ignores the cluster's health status, such as node failures or unassigned shards. This may cause service instability during the restart phase.
   Update Mode	Controls the method used to apply YML file configuration changes. Valid values: Note You must configure the change method only when the Forced Update parameter is set to Close. In-place Update (Default): Applies changes to the nodes in the cluster one by one in a rolling manner. This method does not require data copying and the time taken is not affected by data volume, but it has a certain impact on cluster performance. Blue-green Update: Adds the same number of new nodes to the cluster, copies data to them, and then seamlessly switches the service to the new nodes. This process is relatively smooth but takes longer, and the IP addresses of the nodes will change. For more information about change methods, see Change methods.

   Important

   • Configuring the YML file triggers a rolling restart of the cluster. If the indexes in the cluster have replicas and the cluster load is normal (CPU usage is approximately 60%, heap memory usage is approximately 50%, and load_1m is less than the number of CPU cores), the service remains available during the restart. The restart duration depends on the cluster scale, data volume, and load. We recommend that you perform this operation during off-peak hours.

   • If the cluster load is high or the indexes have no replicas, or if your business involves many write or query operations, occasional access timeouts may occur during the modification process. We recommend that you configure a retry mechanism in the client access script to reduce the impact on your business.

   • If a blue-green change, such as a scale-out or scale-in, is in progress for the instance's YML configuration and the instance status is "Applying", you can only use the in-place modification method for any additional changes. You can view the change history to check if a blue-green change is in progress for the YML configuration.

4. Select This operation will restart the cluster. Continue? and click OK.

   After you confirm, the Elasticsearch instance restarts. You can view the progress in the Task List. After the instance restarts, the YML file configuration is complete.

Configure CORS access

Configure Reindex API Whitelist

To ensure the security of cross-cluster data migration, you must add the ES_2 cluster's private network connection address and communication port number to ES_1's Reindex API whitelist.

1. You can go to the Security page of ES_1. Then, click Configure Private Connection, and then click Edit. In the Configure Private Connection sidebar, click the target Endpoint ID.

   

2. In the Endpoint Connections tab of the VPC console, click the icon next to the Endpoint ID to view its corresponding domain name.

   Important

   You must remove the zone information from the domain name before configuring the Reindex API whitelist.

   For example, if the full domain name is “ep-bp1****************-cn-hangzhou-i.epsrv-bp1****************.cn-hangzhou.privatelink.aliyuncs.com”, remove the zone information, which is “-cn-hangzhou-i”, to obtain the final domain name “ep-bp1bp1****************.epsrv-bp1****************.cn-hangzhou.privatelink.aliyuncs.com”.

   

3. You can configure the Reindex API whitelist in ES_1's YML file. The whitelist must include the endpoint's domain name and communication port.

   reindex:
     remote:
       whitelist: >-
         ep-bp1bp1****************.epsrv-bp1****************.cn-hangzhou.privatelink.aliyuncs.com:9200

   

Configure Auditlog (audit log)

7.x and later

1. Go to the YML File Configuration panel.

   For more information, see Modify the configuration.

2. You can select Enable for the Audit Log Indexing parameter to enable audit logging.

3. Customize the Auditlog configuration.

   After you enable Auditlog, you can configure the xpack.security.audit.logfile.events.include parameter in Other Configurations. The following is an example:

   xpack:
     security:
       audit:
         logfile:
           events:
             include the following: >-
               access_denied,anonymous_access_denied,authentication_failed,connection_denied,tampered_request,run_as_denied,run_as_granted

   Important

   • For 7.x and later instances, you can only configure the xpack.security.audit.logfile.events.include parameter.

   • The default Auditlog configuration only prints audit logs for rejected or failed requests. To obtain audit logs for successful requests, you must add the access_granted event. After you add it, all access information is stored on the disk, which may lead to high disk usage. We recommend that you disable the audit log feature after you finish troubleshooting.

4. You can view the audit logs.

   For 7.x and later instances, you can enable Audit Log Indexing to view audit logs on the View Logs page in the console. For more information, see Query logs. The audit log feature is available only in specific regions. For more information, see Limits.

5.x and 6.x

1. Go to the YML File Configuration panel.

   For more information, see YML File Configuration.

2. To enable audit logging, select Enable for the Audit Log Indexing parameter.

   The following are the default configurations for Auditlog indexing. You can adjust them as needed.

   xpack.security.audit.index.bulk_size: 5000
   xpack.security.audit.index.events.emit_request_body: false
   xpack.security.audit.index.events.exclude: run_as_denied,anonymous_access_denied,realm_authentication_failed,access_denied,connection_denied
   xpack.security.audit.index.events.include the following: authentication_failed,access_granted,tampered_request,connection_granted,run_as_granted
   xpack.security.audit.index.flush_interval: 180s
   xpack.security.audit.index.rollover: hourly
   xpack.security.audit.index.settings.index.number_of_replicas: 1
   xpack.security.audit.index.settings.index.number_of_shards: 10

   Configuration	Default setting	Description
   xpack.security.audit.index.bulk_size	1000	When you write multiple audit events to an Auditlog index in batches, you can use this parameter to set the number of events to write.
   xpack.security.audit.index.flush_interval	1s	Controls the frequency at which buffered events are flushed to the index.
   xpack.security.audit.index.rollover	daily	Controls the frequency of rolling over to a new index. You can set it to hourly, daily, weekly, or monthly.
   xpack.security.audit.logfile.events.include	access_denied,anonymous_access_denied,authentication_failed, connection_denied,tampered_request,run_as_denied,run_as_granted	Controls which Auditlog events can be collected in the audit log. The audit log feature is available only in some regions. For more information, see Limits. For a complete list of event types, see Audit event types (7.x).
   xpack.security.audit.index.events.include	access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted	Controls which Auditlog events can be written to the index. This is supported only for 5.x and 6.x instances. For a complete list of event types, see Audit event types (6.x).
   xpack.security.audit.index.events.exclude	null (By default, no events are processed)	The Auditlog events to exclude during index building.
   xpack.security.audit.index.events.emit_request_body	false	Specifies whether to ignore or include the request body sent via REST when a specific event type, such as authentication_failed, is triggered.

   Warning

   When an Auditlog contains RequestBody information, sensitive information may be exposed in the log file.

3. View the audit logs.

   For 5.x and 6.x instances, when Auditlog is enabled, the Auditlog files are output to the Elasticsearch instance using index names that start with .security_audit_log-*. You can therefore view the audit logs by checking indexes that start with .security_audit_log-* in the Kibana console.

   Important

   Auditlog indexes consume storage space on the instance. Because Elasticsearch does not support an automatic expiration and purge policy, you must manually purge old Auditlog indexes.

4. Optional: Configure the index shards for storing audit logs.

   For 5.x and 6.x instances, you can use xpack.security.audit.index.settings to configure the index shards for storing Auditlog. The following configuration sets the number of shards and replicas for the Auditlog index to 1.

   xpack.security.audit.index.settings:
     index:
       number_of_shards: 1
       number_of_replicas: 1

   Note

   If you want to generate an Auditlog index by passing configuration parameters, include this configuration when you enable Auditlog indexing (set xpack.security.audit.enabled to true). Otherwise, the Auditlog index will use the default configurations of number_of_shards: 5 and number_of_replicas: 1.

Configure queue size