Elasticsearch:Use the aliyun-qos plug-in

Method

Code for the latest version of the plug-in installed on an Elasticsearch V7.10 cluster

Code for the plug-in installed on an Elasticsearch cluster of another version

Set the parameter related to the throttling feature to false

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.limiter.enabled": false
  }
}

PUT _cluster/settings
{
   "persistent" : {
      "apack.qos.ratelimit.enabled":"false"
   }
}

Set the parameter related to the throttling feature to null

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.limiter.enabled": null
  }
}

PUT _cluster/settings
{
   "persistent" : {
      "apack.qos.ratelimit.enabled":null
   }
}

Parameter

Description

Valid value

action

The type of operation on which throttling is performed.

• write: write operation for a document, including indexing or creating a document.

• update: update operation for a document.

• delete: deletion operation for a document.

• search: query operation.

• search_shards: query operation that is used to query the total number of primary and replica shards for an index.

limiter_type

The throttling type. The following throttling types are supported:

• Rate

• Number of concurrent threads

• Single request

• rate: the rate. If you set this parameter to rate, you can set the threshold parameter only to an integer.

• qps: the QPS. If you set this parameter to qps, you can set the threshold parameter only to an integer.

• tps: the transactions per second (TPS). If you set this parameter to tps, you can set the threshold parameter only to an integer.

• throughput: the throughput. You can set this parameter to throughput only if you set the action parameter to write, update, or delete. If you set this parameter to throughput, the units supported by the threshold parameter include GB, MB, and KB, and the maximum value of the threshold parameter is 2 GB.

• thread_count: the number of concurrent threads that can be used for a request. By default, one thread is used for one request.

• concurrent_count: the number of concurrent threads that can be used for a request. If you set this parameter to concurrent_count, the value of the threshold parameter is calculated based on the operation specified in the request. For example, if you specify search_shards.concurrent_count:20 in the limiters configuration item, a maximum of 20 concurrent threads can be used to query the number of primary or replica shards for an index.

• max_per_request: the maximum number of times that a type of operation is allowed to be performed in a single request. For example, if you specify update.max_per_request:10, a maximum of 10 update operations are allowed in a write request.

• max_size_per_request: the maximum number of operations that are allowed in a single request. You can set this parameter to max_size_per_request only if you set the action parameter to write, update, or delete.

threshold

The throttling threshold.

An integer that is greater than or equal to -1.

tagName

The tag key.

• node: the name of the current node.

• is_master: specifies whether the current node is a dedicated master node. If you set the tagName parameter to this value, the value of the tagValue parameter is true or false.

• index: the name of an index. If you want to specify multiple index names, set the tagValue parameter to an array that is composed of the index names. If you set the tagValue parameter to the alias of an index, the aliyun-qos plug-in parses the actual name of the index and performs throttling on the index based on the actual name. You can set the tagName parameter to this value only for a subrequest of the IndicesRequest type.

• shard: the name of a primary or replica shard. If you set this parameter to shard, you must set the tagValue parameter to a value in the index[id] format, such as test[0]. You can set the tagName parameter to this value only for a subrequest of the ReplicationRequest type.

• index_in_url: a string for the name of an index in a URL. If you set the tagValue parameter to the alias of an index, the aliyun-qos plug-in performs throttling on the index indicated by the alias. You can set the tagName parameter to this value only for a subrequest of the IndicesRequest type.

tagValue

The tag value.

A string or an array. If an array is specified, the throttling applies to the resources that match any element in the array. Exact match, fuzzy match, and all values are supported for the tag value. Examples:

• Exact match: "abc"

• Fuzzy match: "ab*"

• All values: "**"

  Note

  If you set the tagValue parameter to "**", a default limiter is used. This indicates that a limiter is generated for each resource that matches the tag. For example, if you specify index:"**",search.tps:1, the search speed of each index, rather than that of all indexes, is limited to 1 by default.

priority

The priority.

An integer. Default value: 0.

params

The advanced parameters.

watchMode: specifies whether to enable the watch mode. Valid values: true and false. Default value: false. If you set the watchMode parameter to true, Elasticsearch only records the number of requests that are denied in the related metric but does not perform throttling. You can call an API to view the monitoring data of metrics and check the throttling effect in advance. This prevents unexpected throttling effect caused by inappropriate configurations. For more information, see FAQ.

Operation

Code for the latest version of the plug-in installed on an Elasticsearch V7.10 cluster

Code for the plug-in installed on an Elasticsearch cluster of another version

Configure throttling for the QPS of a specific index

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "twitter"
  }
}

PUT _qos/_ratelimit/<limiterName>
{
   "search.index_patterns" : "twitter",
   "search.max_queries_per_sec" : 1000
}

Configure throttling for the QPS of indexes whose names have a specific prefix

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

PUT _qos/_ratelimit/<limiterName>
{
   "search.index_patterns" : "nginx-log-*",
   "search.max_queries_per_sec" : 1000
}

Configure throttling for the QPS of any index

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "**"
  }
}

Not supported

Configure throttling for the total QPS of all indexes

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "*"
  }
}

PUT _qos/_ratelimit/<limiterName>
{
   "search.index_patterns" : "*",
   "search.max_queries_per_sec" : 1000
}

Operation

Code for the latest version of the plug-in installed on an Elasticsearch V7.10 cluster

Code for the plug-in installed on an Elasticsearch cluster of another version

Query all limiters

GET _qos/limiter

GET _qos/_ratelimit

Query a specific limiter

GET _qos/limiter/<limiterName>

GET _qos/_ratelimit/<limiterName>

Query multiple limiters

GET _qos/limiter/<limiterName1,limiterName2>

GET _qos/_ratelimit/<limiterName1,limiterName2>

Operation

Code for the latest version of the plug-in installed on an Elasticsearch V7.10 cluster

Code for the plug-in installed on an Elasticsearch cluster of another version

Delete a specific limiter

DELETE _qos/limiter/<limiterName>

DELETE _qos/_ratelimit/<limiterName>

Delete multiple limiters

DELETE _qos/limiter/<limiterName1,limiterName2>

DELETE _qos/_ratelimit/<limiterName1,limiterName2>