How to use the aliyun-qos cluster throttling plug-in, throttling plug-in, throttling - Elasticsearch

The aliyun-qos plug-in is a cluster-level read and write throttling plug-in developed by the Alibaba Cloud Elasticsearch team. When upstream services cannot control traffic, you can use the aliyun-qos plug-in to throttle specific indexes based on business priority. This helps keep traffic manageable and ensures your Elasticsearch cluster's stability.

Precautions

The aliyun-qos plug-in is pre-installed and cannot be uninstalled. Its throttling feature is disabled by default. The plug-in is designed to protect cluster stability, not to precisely measure read and write traffic.
Plug-in version: Before using the aliyun-qos plug-in, ensure it is updated to the latest version. Log on to the Kibana console and run GET /_cat/plugins?v to check the plug-in version.
The plug-in version is in the format <cluster version>_ali<internal version number>, for example, 7.10.0_ali1.6.0.2 or 8.17.0_ali2.2.0.4.1. If the plug-in version is not the latest, refer to the following methods to upgrade it:
- For a V7.10 cluster: In the console, upgrade the kernel to version 1.6.0. For more information, see Upgrade cluster versions.
- For other cluster versions: Submit a ticket to Alibaba Cloud Support to upgrade the plug-in. After the upgrade, you must manually restart the Elasticsearch cluster for the new version to take effect.
- If the plug-in version is earlier than rc4, an unsupported_operation_exception error occurs. The aliyun-qos plug-in can be upgraded only on clusters running V6.7.0 or later. You must first upgrade clusters running earlier versions to V6.7.0 or later.

Evaluate thresholds

The aliyun-qos plug-in performs throttling at the cluster level. To minimize performance overhead, it does not precisely measure read and write traffic on all nodes, so the actual traffic might differ from the measured traffic. Before you use the plug-in, evaluate the throttling threshold based on the following rules:

Query requests
Throttling threshold for query requests = End-to-end QPS (queries per second) from the client to Elasticsearch
End-to-end QPS refers only to the number of query requests that reach the client nodes per second.
Write requests
The rule for calculating the write request throttling threshold is similar to that for query requests, but requires adjustment for the number of replica shards.
For example, consider a cluster with two data nodes and one index. The index has one primary shard and one replica shard. Each write operation sends 10 MB of data. Because a replica shard exists, 10 MB of data is written to each data node. Additionally, internal X-Pack tasks such as Monitor, Audit, and Watcher also consume write throughput. Reserve capacity for this traffic when setting the threshold.

Enable throttling

The throttling feature of the aliyun-qos plug-in is disabled by default. You must enable it before use. The command to enable the feature varies by plug-in version.

Latest V7.10 version	Other versions
`PUT _cluster/settings { "persistent": { "apack.qos.limiter.enabled": true } }`	`PUT _cluster/settings { "persistent": { "apack.qos.ratelimit.enabled": "true" } }`

Disable throttling

You can disable the throttling feature by setting the limiter parameter to false or null. The command varies by version.

Method

Latest V7.10 version

Other versions

Set the limiter parameter to false

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.limiter.enabled": false
  }
}

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.ratelimit.enabled": "false"
  }
}

Set the limiter parameter to null

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.limiter.enabled": null
  }
}

PUT _cluster/settings
{
  "persistent": {
    "apack.qos.ratelimit.enabled": null
  }
}

Configure a limiter (latest V7.10 version)

The following limiter configurations apply only to the aliyun-qos plug-in for V7.10 clusters.

A limiter configuration consists of two parts: limiters and tags. The tags section defines the scope of resource limits, while the limiters section defines the specific throttling type and threshold. Limiters can be standard or default. You can create a default limiter by setting a tag value to **. For example, you can set a default throughput for each shard or a default QPS for each application. When a request exceeds a limit, Elasticsearch rejects subsequent requests.

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
     ${action}.${limiter_type}:${threshold}
  },
  "tags": {
    ${tagName}:${tagValue}
  },
  "priority":0,
  "params":{
      "watchMode":true
  }
}

Parameter	Description	Value
action	The action to be throttled. This is used to limit different types of requests.	`write`: Write requests for documents, including `index` and `create` operations. `update`: Document update requests. `delete`: Document delete requests. `search`: Query requests. `search_shards`: Requests to query the total number of shards for an index.
limiter_type	The throttling type. Three categories are supported: rate, concurrency, and per-request limits.	`rate`: Rate limit. Accepts integers only. `qps`: Query rate. Accepts integers only. `tps`: Write rate. Accepts integers only. `throughput`: Data throughput. Supported only for `write`, `update`, and `delete` actions. Units include GB, MB, and KB. The maximum value is 2 GB. `thread_count`: Limits the number of concurrent requests. `concurrent_count`: Request concurrency, calculated based on the specific operations in a request. For example, `search_shards.concurrent_count:20` allows a maximum of 20 concurrent shard queries. `max_per_request`: The maximum number of operations of a certain type within a single request. For example, `update.max_per_request:10` allows a maximum of 10 update operations in a single Bulk request. `max_size_per_request`: The maximum size of a single request. Supported only for `write`, `update`, and `delete` actions.
threshold	The throttling threshold.	An integer greater than or equal to -1. Some types support strings with units. For more information, see the description of `limiter_type`.
tagName	The tag name.	`node`: The name of the current node. `is_master`: Specifies whether the current node is a master node. The corresponding tag value is `true` or `false`. `index`: The index name. For multiple indexes, use an array. If an alias is passed, it is resolved to the actual index name. This tag is available only for subclasses of `IndicesRequest`. `shard`: The shard name, in the format `index[id]`, for example, `test[0]`. This tag is available only for subclasses of `ReplicationRequest`. `index_in_url`: The index string from the URL. If an alias is passed, this tag retains the alias. This tag is available only for subclasses of `IndicesRequest`.
tagValue	The tag value.	A string or an array of strings. If an array is used, the tag matches any value in the array. Exact match, prefix match with a wildcard, and all values are supported. Examples: Exact match: "abc" Prefix match: "ab" All values: "" Using a wildcard value defines a default throttler, which means a specific throttler is generated for any value of this tag. For example, if you specify `index:"*",search.tps:1`, the search rate for each index is throttled to 1 by default, rather than the total search rate being throttled to 1.
priority	The priority of the limiter.	An integer. Default value: 0. A higher value indicates a higher priority. When a request matches multiple default limiters, only the one with the highest priority takes effect.
params	Advanced parameters.	`watchMode`: Specifies whether to enable watch mode. Valid values: `true` and `false` (default). When set to `true`, Elasticsearch records rejected request counts in metrics but does not enforce the limits. This allows you to preview the effect of a rule before applying it, preventing misconfigurations. For more information about the APIs, see the FAQ section.

Limiter configuration examples

Set QPS throttling for queries

You can limit the query QPS on a client node by setting a threshold for an index. When the number of query requests per second exceeds the threshold, Elasticsearch rejects subsequent requests.

The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Actions	Latest V7.10 version	Other versions
Set query QPS throttling for a single index	`PUT /_qos/limiter/<limiterName> { "limiters": { "search.qps": "1000" }, "tags": { "index": "twitter" } }`	`PUT /_qos/_ratelimit/<limiterName> { "search.index_patterns": "twitter", "search.max_queries_per_sec": 1000 }`
Set query QPS throttling for indexes with a specific name prefix	`PUT /_qos/limiter/<limiterName> { "limiters": { "search.qps": "1000" }, "tags": { "index": "nginx-log-*" } }`	`PUT /_qos/_ratelimit/<limiterName> { "search.index_patterns": "nginx-log-*", "search.max_queries_per_sec": 1000 }`
Set query QPS throttling for each index individually	`PUT /_qos/limiter/<limiterName> { "limiters": { "search.qps": "1000" }, "tags": { "index": "**" } }` `index:` represents any index. For example, if a cluster has three indexes A, B, and C, the throttling value for all three indexes is 1000.	Not supported.
Set a total query QPS limit for all indexes	`PUT /_qos/limiter/<limiterName> { "limiters": { "search.qps": "1000" }, "tags": { "index": "" } }` `index:` indicates any index. For example, if a cluster has three indexes named A, B, and C, the total throttling value for these three indexes is 1000. This has the same effect as not setting this tag.	`PUT /_qos/_ratelimit/<limiterName> { "search.index_patterns": "*", "search.max_queries_per_sec": 1000 }`

You can define multiple rules. Throttling is triggered if a request matches any rule.

When the query QPS exceeds the configured limit, the system returns an error message. The error message varies by version:

Latest V7.10 version

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "search blocked, limited by [<limiterName>][search.qps](<limiterId>) threshold:[x]"
      }
    ],
    "type": "status_exception",
    "reason": "search blocked, limited by [<limiterName>][search.qps](<limiterId>) threshold:[x]"
  },
  "status": 429
}

Other versions

{
  "error": {
    "root_cause": [
      {
        "type": "rate_limited_exception",
        "reason": "request indices:data/read/search rejected, limited by [l1:t*:1.0]"
      }
    ],
    "type": "rate_limited_exception",
    "reason": "request indices:data/read/search rejected, limited by [l1:t*:1.0]"
  },
  "status": 429
}

Set TPS throttling for writes

You can limit the number of write requests per second that a client node receives by setting a TPS (transactions per second) threshold. When the number of write requests per second exceeds the threshold, Elasticsearch rejects subsequent requests.

The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Latest V7.10 version	Other versions
`PUT /_qos/limiter/<limiterName> { "limiters": { "write.tps": "100000" }, "tags": { "index": "nginx-log-*" } }`	Not supported

Set throughput throttling for Bulk requests

You can limit the write throughput for Bulk API requests on a client node by setting a limit in bytes per second. When the write throughput exceeds this limit, Elasticsearch rejects subsequent requests.

The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Latest V7.10 version

Other versions

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "write.throughput": "100MB"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

PUT /_qos/_ratelimit/<limiterName>
{
  "bulk.index_patterns": "nginx-log-*",
  "bulk.max_throughput_in_bytes": 104857600
}

You can define multiple rules. Throttling is triggered if a request matches any rule.

Set request size throttling for Bulk requests

You can limit the size of a single Bulk API request on a client node. If a request exceeds this size limit, Elasticsearch rejects it.

The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Latest V7.10 version

Other versions

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "write.max_size_per_request": "1000"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

PUT /_qos/_ratelimit/<limiterName>
{
  "bulk.index_patterns": "nginx-log-*",
  "bulk.max_request_size_in_bytes": 1000
}

You can define multiple rules. Throttling is triggered if a request matches any rule.

When the size of a single write request exceeds the configured limit, the system returns an error message. The error message varies by version:

Latest V7.10 version

{
  "error" : {
    "root_cause" : [
      {
        "type" : "status_exception",
        "reason" : "write_size blocked, limited by [<limiterName>][write.max_size_per_request](<limiterId>) threshold:[x] try acquire [x]"
      }
    ],
    "type" : "status_exception",
    "reason" : "write_size blocked, limited by [<limiterName>][write.max_size_per_request](<limiterId>) threshold:[x] try acquire [x]"
  },
  "status" : 400
}

Other versions

{
  "error": {
    "root_cause": [
      {
        "type": "rate_limited_exception",
        "reason": "request indices:data/write/bulk rejected, limited by [b2:ByteSizePreSeconds:992.0]"
      }
    ],
    "type": "rate_limited_exception",
    "reason": "request indices:data/write/bulk rejected, limited by [b2:ByteSizePreSeconds:992.0]"
  },
  "status": 413
}

Set concurrency throttling for shard queries

You can reduce the load on your cluster by setting the number of concurrent shard queries. The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Latest V7.10 version	Other versions
`PUT /_qos/limiter/<limiterName> { "limiters": { "search_shards.concurrent_count": "10" }, "tags": { "index": "nginx-log-*" } }`	Not supported

You can define multiple rules. Throttling is triggered if a request matches any rule.

Set multiple limiter configurations

You can define multiple limits within a single limiter configuration. The values for index and index_patterns can be full index names or names with a wildcard. The command varies by version.

Latest V7.10 version

Other versions

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000",
    "write.tps": "100000",
    "write.throughput": "1000000",
    "write.max_size_per_request": "1000",
    "search_shards.concurrent_count": "10"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

Not supported

You can define multiple rules. Throttling is triggered if a request matches any rule.

Get limiter configurations

The command to get limiter configurations varies by version.

Actions	Latest V7.10 version	Other versions
Get all limiter configurations	`GET /_qos/limiter`	`GET /_qos/_ratelimit`
Get a single specified limiter configuration	`GET /_qos/limiter/<limiterName>`	`GET /_qos/_ratelimit/<limiterName>`
Get multiple specified limiter configurations. Separate multiple limiter names with commas (,). Wildcards are not supported.	`GET /_qos/limiter/<limiterName1,limiterName2>`	`GET /_qos/_ratelimit/<limiterName1,limiterName2>`

Delete limiter configurations

The command to delete limiter configurations varies by version.

Actions	Latest V7.10 version	Other versions
Delete a single specified limiter configuration	`DELETE /_qos/limiter/<limiterName>`	`DELETE /_qos/_ratelimit/<limiterName>`
Delete multiple specified limiter configurations. Separate multiple limiter names with commas (,). Wildcards are not supported.	`DELETE /_qos/limiter/<limiterName1,limiterName2>`	`DELETE /_qos/_ratelimit/<limiterName1,limiterName2>`

FAQ

Q: How do I obtain monitoring metrics related to throttling?

A: You can use the following APIs:

Get current metric data
- Get current data for all metrics
```
GET /_qos/limiter/nodes/stats
```
- Get current metric data for a specific node
```
GET /_qos/limiter/nodes/{nodeId}/stats
```
- Get current metric data for a specific node and limiter
```
GET /_qos/limiter/nodes/{nodeId}/stats/{limiterIds}
```
Get historical metric data
- Get historical data for all metrics
```
GET /_qos/limiter/metric
```
- Get historical metric data for a specific limiter
```
GET /_qos/limiter/metric/{limiterId}
```

Notes on plug-in upgrades

When you upgrade the aliyun-qos plug-in to the latest version, take note of the following:

Due to differences in the implementation mechanism between old and new versions, the throttling feature may be temporarily unavailable during the upgrade process. It automatically restores after the plug-in on the master node is upgraded.
During the data conversion process, some limiters may fail to convert from the old format. If a conversion fails, run the following command to retry. If the command returns an error, you can run it multiple times until hasError is false.
```
POST /_qos/limiter/ops/upgrade
```
If the preceding command returns an error message (such as unknown action), this indicates that the cluster does not have a legacy throttler. You can ignore the message.