Concepts and implementation principle of throttling - Alibaba Cloud Service Mesh

Concepts of throttling

Throttling limits the number of requests sent to a service within a given period of time, such as 300 requests per minute or 10 requests per second. This prevents a service from being overloaded by excessive requests from a specific client IP address or from global clients.

For example, if you limit the number of requests sent to a service to 300 per minute, the 301st request is denied. At the same time, the HTTP 429 status code that indicates excessive requests is returned.

Throttling modes

Envoy proxies support two throttling modes: local throttling and global throttling. Local throttling limits the request rate of each service instance. Global throttling uses a global gRPC service to enforce rate limits across the entire Service Mesh (ASM) instance. You can combine both modes to apply different levels of throttling.

Mode	Description	References
Local throttling	Local throttling is configured on a per Envoy process basis. An Envoy process is a pod in which an Envoy proxy is injected. The configuration of local throttling is simpler than that of global throttling and it does not require an additional component. If you configure both local throttling and global throttling for an ASM instance, the local rate limit is applied first. If the local rate limit is not reached, the global rate limit is applied. Examples: Assume that local throttling limits the number of requests from a specific client to 50 per minute whereas the global throttling limit is 60 requests per minute. When the number of requests from the client exceeds 50, the excessive requests are denied even though the number of requests has not reached the global throttling limit. Assume that local throttling limits the number of requests from a specific client to 50 per minute, and the global throttling limit is 40 requests per minute. When the number of requests from the client exceeds 40, the excessive requests are denied due to global throttling, even though the number of requests has not reached the local throttling limit. If the pod on which the local throttling limit is configured has multiple replicas, each of those replicas has its own limit. That is, requests may be limited on one replica but not limited on another.	Configure local throttling in Traffic Management Center
Global or distributed throttling	Global throttling limits the number of requests sent to multiple services. In this mode, all the services in a cluster share the throttling configuration. Generally, global throttling requires an external component, such as a Redis database. Global throttling is typically used in scenarios where many clients send requests to a smaller number of services. In this case, the requests may interrupt the services. Global throttling can help prevent cascading failures. For example, you can configure global throttling on an ingress gateway to limit the total number of requests sent to an ASM instance. Then, you can configure local throttling to limit the number of requests sent to specific services in the ASM instance.	Configure global rate limiting for application services

How local throttling works

An Envoy proxy uses the token bucket algorithm to implement local throttling. Tokens are added to a bucket at a constant rate. Each incoming request consumes one token. When the bucket is empty, subsequent requests are denied. You generally need to specify the following parameters:

The interval at which the bucket is filled
The number of tokens added to the bucket at each interval

By default, an Envoy proxy returns the HTTP 429 status code when a request is denied and the x-envoy-ratelimited header is set. You can customize the HTTP status code and response header.

Take note of the following concepts when you use the throttling feature:

http_filter_enabled: indicates the percentage of requests for which the local rate limit is checked but not enforced.
http_filter_enforcing: indicates the percentage of requests on which the local rate limit is applied or enforced.

Set the values to percentages. For example, you can set http_filter_enabled to 10% of requests and http_filter_enforcing to 5% of requests. This way, you can test the effect of throttling before it is applied to all the requests.

How global throttling works

Global throttling controls request rates across an ASM instance by using the Envoy rate limit service. The rate limit service centrally processes traffic and enforces rate limits based on predefined rules and quotas.

The configuration of global throttling involves two parts: the Envoy rate_limits filter and the configuration of the rate limit service.

The rate_limits filter contains a list of actions. An Envoy proxy attempts to match each request against each action in the rate_limits filter. A descriptor is generated for each action. A descriptor consists of a set of descriptor entries that correspond to an action. Each descriptor entry is a key-value pair, such as "descriptor-key-1": "descriptor-value-1" and "descriptor-key-2": "descriptor-value-2". For more information, see config-http-filters-rate-limit.
The configuration of the rate limit service is matched against the descriptor entry generated for each request. The configuration of the rate limit service specifies the rate limit for a specific set of descriptor entries. The rate limit service interacts with the Redis cache to determine whether to limit the rates of requests and sends the throttling decision to the Envoy proxy.

Together, the rate_limits filter and the rate limit service implement global throttling. The filter generates a descriptor based on the configured action and sends it to the rate limit service, which determines the applicable limit and returns a throttling response. This mechanism gives you full control over request rates and protects backend services against traffic bursts.