Elasticsearch:Use the aliyun-qos cluster throttling plugin

Preview mode before enforcement

Before enforcing throttling, use watchMode to validate your configuration safely. When watchMode is set to true, the plug-in records denied request counts in metrics without actually rejecting requests. Review the metrics to confirm thresholds are appropriate, then switch to enforcement mode by setting watchMode to false.

Limiter syntax

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
     ${action}.${limiter_type}:${threshold}
  },
  "tags": {
    ${tagName}:${tagValue}
  },
  "priority":0,
  "params":{
      "watchMode":true
  }
}

Limiter parameters

Limiter type reference

Tag name reference

Throttle QPS for read requests

Limit the number of search requests a client node receives per second. Both exact index names and wildcard patterns are supported for tag values. For Elasticsearch V7.10, use the index tag key; for all other versions, use the index_patterns tag key.

Elasticsearch V7.10 — throttle a specific index:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "twitter"
  }
}

Elasticsearch V7.10 — throttle indexes with a name prefix:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

Elasticsearch V7.10 — throttle each index individually (per-index limit):

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "**"
  }
}

index:** applies the threshold to each index separately. For example, if the cluster has indexes A, B, and C, each is limited to 1,000 QPS independently.

Elasticsearch V7.10 — throttle total QPS across all indexes:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000"
  },
  "tags": {
    "index": "*"
  }
}

index:* applies the threshold to the combined QPS of all indexes. You can also omit the tags field to achieve the same effect.

All other versions — throttle a specific index:

PUT _qos/_ratelimit/<limiterName>
{
  "search.index_patterns" : "twitter",
  "search.max_queries_per_sec" : 1000
}

All other versions — throttle indexes with a name prefix:

PUT _qos/_ratelimit/<limiterName>
{
  "search.index_patterns" : "nginx-log-*",
  "search.max_queries_per_sec" : 1000
}

All other versions — throttle total QPS across all indexes:

PUT _qos/_ratelimit/<limiterName>
{
  "search.index_patterns" : "*",
  "search.max_queries_per_sec" : 1000
}

Multiple rules can be active at the same time. A request is throttled if it matches any rule.

QPS throttling errors

If QPS exceeds the configured threshold, the system returns a 429 error. Reduce client-side QPS to resolve this error.

For Elasticsearch V7.10:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "search blocked, limited by [<limiterName>][search.qps](<limiterId>) threshold:[x]"
      }
    ],
    "type": "status_exception",
    "reason": "search blocked, limited by [<limiterName>][search.qps](<limiterId>) threshold:[x]"
  },
  "status": 429
}

For all other versions:

{
  "error": {
    "root_cause": [
      {
        "type": "rate_limited_exception",
        "reason": "request indices:data/read/search rejected, limited by [l1:t*:1.0]"
      }
    ],
    "type": "rate_limited_exception",
    "reason": "request indices:data/read/search rejected, limited by [l1:t*:1.0]"
  },
  "status": 429
}

Throttle TPS for write requests

Limit the number of write requests a client node receives per second. Both exact index names and wildcard patterns are supported for tag values. For Elasticsearch V7.10, use the index tag key; for all other versions, use the index_patterns tag key.

Elasticsearch V7.10:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "write.tps": "100000"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

TPS throttling is not supported for other Elasticsearch versions.

Throttle bulk write throughput (total per second)

Limit the total bytes written per second across all bulk requests to a client node. For more information about bulk requests, see Bulk API. Both exact index names and wildcard patterns are supported for tag values. For Elasticsearch V7.10, use the index tag key; for all other versions, use the index_patterns tag key.

Elasticsearch V7.10:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "write.throughput": "100MB"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

All other versions:

PUT _qos/_ratelimit/<limiterName>
{
  "bulk.index_patterns": "nginx-log-*",
  "bulk.max_throughput_in_bytes" : 104857600
}

Multiple rules can be active at the same time. A request is throttled if it matches any rule.

Throttle bulk write size per request

Limit the maximum bytes written in a single bulk request to a client node. For more information about bulk requests, see Bulk API. Both exact index names and wildcard patterns are supported for tag values. For Elasticsearch V7.10, use the index tag key; for all other versions, use the index_patterns tag key.

Elasticsearch V7.10:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "write.max_size_per_request": "1000"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

All other versions:

PUT _qos/_ratelimit/<limiterName>
{
  "bulk.index_patterns": "nginx-log-*",
  "bulk.max_request_size_in_bytes" : 1000
}

Multiple rules can be active at the same time. A request is throttled if it matches any rule.

Bulk write size throttling errors

If a single bulk request exceeds the configured size threshold, the system rejects it.

For Elasticsearch V7.10 (HTTP 400):

{
  "error" : {
    "root_cause" : [
      {
        "type" : "status_exception",
        "reason" : "write_size blocked, limited by [<limiterName>][write.max_size_per_request](<limiterId>) threshold:[x] try acquire [x]"
      }
    ],
    "type" : "status_exception",
    "reason" : "write_size blocked, limited by [<limiterName>][write.max_size_per_request](<limiterId>) threshold:[x] try acquire [x]"
  },
  "status" : 400
}

For all other versions (HTTP 413):

{
  "error": {
    "root_cause": [
      {
        "type": "rate_limited_exception",
        "reason": "request indices:data/write/bulk rejected, limited by [b2:ByteSizePreSeconds:992.0]"
      }
    ],
    "type": "rate_limited_exception",
    "reason": "request indices:data/write/bulk rejected, limited by [b2:ByteSizePreSeconds:992.0]"
  },
  "status": 413
}

Reduce the bytes in each bulk request to resolve this error.

Throttle concurrent shard queries

Limit the number of concurrent threads used to query the number of primary or replica shards to reduce cluster load. Both exact index names and wildcard patterns are supported for tag values. For Elasticsearch V7.10, use the index tag key; for all other versions, use the index_patterns tag key.

Elasticsearch V7.10:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search_shards.concurrent_count": "10"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

This configuration is not supported for other Elasticsearch versions.

Configure multiple throttle rules in one limiter

Apply multiple throttling rules to a single limiter in one request.

Elasticsearch V7.10:

PUT /_qos/limiter/<limiterName>
{
  "limiters": {
    "search.qps": "1000",
    "write.tps": "100000",
    "write.throughput": "1000000",
    "write.max_size_per_request": "1000",
    "search_shards.concurrent_count": "10"
  },
  "tags": {
    "index": "nginx-log-*"
  }
}

Multiple rules can be active at the same time. A request is throttled if it matches any rule.