Service Mesh (ASM) supports non-intrusive circuit breaking, throttling, and traffic monitoring. This topic describes how to configure dashboard and alert rules for circuit breaking and throttling.
Circuit breaking
Traffic circuit breaking is an overload protection mechanism that prevents system crashes caused by traffic burst in a short period of time. In the case of east-west calls between cloud-native services, a failure of one service (such as slow responses or increased failure rates) may lead to cascading failures across a series of services in the trace.
You can configure circuit breaking rules for east-west call traffic between services to reject requests from upstream services when the failure rate or the number of response timeouts reaches the corresponding threshold. This protects upstream services and effectively prevents faults from affecting the entire trace and causing the entire system to crash.
After you configure a circuit breaking rule, each ASM proxy calculates the failure rate or the number of response timeouts based on the requests it receives. Therefore, for the same faulty upstream service, the time points at which circuit breaking occurs at different ASM proxies may be slightly different.
Dashboard
The description for each panel is as follows:
Panel | Description |
Requests | Request statistics include the following metrics:
|
Requests OK Percent | The proportion of successful requests to total requests. |
Requests Throttled Percent | The proportion of requests that triggers circuit breaking to total requests. |
You can import the following JSON file into the Grafana dashboard to create a dashboard for displaying requests that triggers circuit breaking in your cluster. You can also modify or customize a dashboard based on the following JSON content.
Sample alert rule
Alert configuration | Example | Description |
Custom PromQL statement | (sum by(cluster, namespace) (increase(envoy_asm_circuit_breaker_total_broken_requests[1m]))) > 10 | The increase statement queries the number of requests that are rejected due to circuit breaking in the last one minute. The number of requests is grouped by the namespace and name of the service that triggers circuit breaking. An alert is reported when the number of requests that are rejected due to circuit breaking within one minute is greater than 10. |
Alert message | Service-level circuit breaking occurred! Namespace: {{$labels.namespace}}, Service that triggers circuit breaking: {{$labels.cluster}}. The number of requests that are rejected due to circuit breaking within the current one minute: {{$value}} | The alert information shows the namespace of the service that triggers the circuit breaking, the service name, and the number of requests that are sent to the service but are rejected due to circuit breaking in the last one minute. |
Global throttling
Global throttling of Envoy is a mechanism used to control the request rates in an ASM instance. It is implemented based on the rate limit service of Envoy. The rate limit service centrally processes traffic of the entire ASM instance and limits the rates of requests based on predefined rules and quotas.
Global throttling can be used together with global throttling to provide different levels of throttling..
Sample dashboard
The description for each panel is as follows:
Panels | Description |
Requests | Request statistics include the following metrics:
|
Requests OK Percent | The proportion of successful requests to total requests. |
Requests Reached Limits Percent | The proportion of requests that triggers throttling to total requests. |
You can import the following JSON file into the Grafana dashboard to create a dashboard for displaying global throttling for your cluster. You can also modify or customize a dashboard based on the following JSON content.
Sample alert rule
Alert configuration | Example | Description |
Custom PromQL statement | sum (increase(envoy_cluster_ratelimit_over_limit[1m])) by (namespace, service_istio_io_canonical_name) > 10 | The increase statement queries the number of requests that are rejected due to throttling in the last one minute. The number of requests is grouped by the namespace and name of the service that triggers throttling. An alert is reported when the number of requests that are rejected due to throttling within one minute is greater than 10. |
Alert message | Throttling triggered! Namespace: {{$labels.namespace}}, Service that triggers throttling: {{$labels.service_istio_io_canonical_name}}. The number of requests that are rejected due to throttling within the current one minute: {{$value}} | The alert information shows the namespace of the service that triggers the throttling, the service name, and the number of requests that are sent to the service but are rejected due to throttling in the last one minute. |
Local throttling
An Envoy proxy uses the token bucket algorithm to implement local throttling. The token bucket algorithm is a method that limits the number of requests sent to services based on a certain number of tokens in a bucket. Tokens fill in the bucket at a constant rate. When a request is sent to a service, a token is removed from the bucket. When the bucket is empty, requests are denied.
Local throttling can be used together with global throttling to provide different levels of throttling.
Sample dashboard
The description for each panel is as follows:
Panel | Description |
Requests | Request statistics include the following metrics:
|
Requests OK Percent | The proportion of successful requests to total requests. |
Requests Reached Limits Percent | The proportion of requests that triggers throttling to total requests. |
You can import the following JSON file into the Grafana dashboard to create a dashboard for displaying local throttling for the cluster. You can also modify or customize a dashboard based on the following JSON content.
Sample alert rule
Alert configuration | Example | Description |
Custom PromQL statements | sum (increase(envoy_http_local_rate_limiter_http_local_rate_limit_enforced[1m])) by (namespace, service_istio_io_canonical_name) > 10 | The increase statement queries the number of requests that are rejected due to throttling in the last one minute. The number of requests is grouped by the namespace and name of the service that triggers throttling. An alert is reported when the number of requests that are rejected due to throttling within one minute is greater than 0. |
Alert message | Throttling triggered! Namespace: {{$labels.namespace}}, Service that triggers throttling: {{$labels.service_istio_io_canonical_name}}. The number of requests that are rejected due to throttling within the current one minute: {{$value}} | The alert information shows the namespace of the service that triggers the throttling, the service name, and the number of requests that are sent to the service but are rejected due to throttling in the last one minute. |
Related operations
Import Grafana dashboard
Log on to the ARMS console.
In the left-side navigation pane, click Integration Management.
On the Integrated Environments tab, select Container Service. Enter a cluster name to search for the environment, and then click the target environment name. Click Dashboard Directory.
On the right side of the Dashboards tab, click the Import button.
Paste the content of the JSON file to the Import via panel json section, and then click the Load button. Keep the default settings and click Import to complete the import.
NoteYou can also import the metrics to dashboard by uploading a JSON file.