Configure circuit breaking for east-west traffic - Alibaba Cloud Service Mesh

Background

Circuit breaking is an overload protection mechanism that prevents a system from crashing due to a sudden surge in traffic. In cloud-native environments with east-west traffic between services, a single failing service (for example, a service that responds slowly or has a high error rate) can cause a chain reaction, leading to a cascading failure across the entire call chain.

A circuit breaking rule for east-west traffic automatically "trips the circuit" and rejects requests to an upstream service when an error rate or response timeout threshold is exceeded. This protects the upstream service, prevents the fault from spreading, and helps avoid a system-wide outage.

After you configure a circuit breaking rule, each mesh proxy independently calculates the traffic error rate or the number of slow requests for the traffic it handles. As a result, different client-side mesh proxies might trip the circuit at slightly different times for the same failing upstream service.

Prerequisites

You have an ASM Professional Edition instance, v1.14.3 or later. For more information, see Create an ASM instance.
Automatic sidecar proxy injection is enabled.
The sleep and httpbin sample applications are deployed. For more information, see Deploy the httpbin application in a data plane cluster and Deploy the sleep service in a data plane cluster.

Step 1: Configure request path routing

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

Create a virtual service by using one of the following methods.

Use the console

On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > VirtualService. On the page that appears, click Create.
Specify the Namespace and Host, and enable Gateways > Apply To All Sidecars.
In the Hosts section, click Add Host to add the httpbin service.
In the HTTP Route section, click Add Route and configure the routes as follows.

Set the route name to error-route. Enable Match Request URI, select Exact as the match type, and enter /status/500 for the match content. Set the destination service name to httpbin.default.svc.cluster.local.

Set the route name to delay-route. Enable Match Request URI, select Prefix as the match type, and enter /delay for the match content. Set the destination service name to httpbin.default.svc.cluster.local.

Set the route name to default-route. Do not add any request matching rules. Set the destination service name to httpbin.default.svc.cluster.local, and leave the version, weight, and port fields empty.

Use YAML

On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > VirtualService. On the page that appears, click Create from YAML.

In the YAML editor, paste the following content and click Create.

Click to view the YAML file

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  hosts:
    - httpbin.default.svc.cluster.local
  http:
    - match:
        - uri:
            exact: /status/500
      name: error-route
      route:
        - destination: 
            host: httpbin.default.svc.cluster.local
    - match:
        - uri:
            prefix: /delay
      name: delay-route
      route:
        - destination:
            host: httpbin.default.svc.cluster.local
    - name: default-route
      route:
        - destination:
            host: httpbin.default.svc.cluster.local

The following table maps request paths to routes.

Request path	Match type	Route	Description
`/status/500`	Exact	`error-route`	Always returns a 500 status code.
`/delay`	Prefix	`delay-route`	Returns a 200 status code after a specified delay. For more information about how to use the /delay path, see delay.
`/*`	Any path	`default-route`	The default route.

Step 2: Configure circuit breaking

This section describes how to configure and test error rate-based circuit breaking and slow request-based circuit breaking.

Error rate

Error rate-based circuit breaking trips the circuit when the error rate of server responses exceeds a threshold within a given time window.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > Circuit Breaking and Degradation.

On the Create page, paste the following content into the YAML editor and click Create.

Click to view the YAML file

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMCircuitBreaker
metadata:
  name: httpbin-error-circuitbreak
  namespace: default
spec:
  configs:
    - breaker_config:
        break_duration: 60s
        custom_response:
          body: error break!
          header_to_add:
            x-envoy-overload: 'true'
          status_code: 499
        error_percent:
          value: 60
        min_request_amount: 5
        window_size: 10s
      match:
        vhost:
          name: httpbin.default.svc.cluster.local
          port: 8000
          route:
            name_match: error-route
  workloadSelector:
    labels:
      app: sleep

The following table describes the parameters in the circuit breaking configuration.

Parameter	Description
workloadSelector.labels	The downstream service workload to which the circuit breaking rule applies. In this example, the downstream service is the sleep service, which is selected by the `app: sleep` label.
break_duration	The duration the circuit remains open after tripping. In this example, the value is 60s.
window_size	The time window for circuit breaking detection. In this example, the value is 10s. This means that if the request error rate on the route exceeds the specified threshold within 10 seconds, the circuit breaker trips and rejects requests.
error_percent	The request error rate threshold required to trip the circuit breaker within the time window. In this example, the value is 60. This means that if the request error rate on the route exceeds 60% within a 10s time window, the circuit breaker trips and rejects requests.
min_request_amount	The minimum number of requests required to trip the circuit breaker within the time window. This parameter prevents the circuit breaker from being tripped by a small number of requests. In this example, the value is 5. This means that the circuit breaker trips only if more than 5 requests are sent on the route within a 10s time window and the error rate exceeds 60%.
custom_response	The custom response returned by the mesh proxy for requests rejected after the circuit breaker trips. `body` is set to `error break!`. The response body is `error break!`. `header_to_add` is set to `x-envoy-overload: 'true'`. The `x-envoy-overload: 'true'` header is added to the response when the circuit is open. `status_code` is set to `499`. The response code for the request is `499` after the circuit is open.
match.vhost	The route entry for which circuit breaking is configured. The route entry must match a specific route entry declared in the virtual service. `name`: The domain name of the upstream service in the call chain. In this example, this parameter is set to `httpbin.default.svc.cluster.local`, which is the domain name of the httpbin service (the upstream service of the sleep service). `port`: The service port of the upstream service. In this example, this parameter is set to `8000`, which is the service port of the httpbin service. `route.name_match`: The actual name of the route entry in the virtual service. The circuit breaking configuration takes effect on this route. In this example, this parameter is set to `error-route`, which is the route entry created in Step 1. The requests that match this route entry always return a 500 status code, which ensures that the circuit breaker can be tripped.

Connect to the ACK cluster by using kubectl and run the following command:

for i in {1..100};  do kubectl exec -it deploy/sleep -- curl httpbin:8000/status/500 -I | grep 'HTTP';  echo ''; sleep 0.1; done;

Expected output:

Click to view details

HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
...

As the output shows, the circuit breaker is tripped on the sixth request. Subsequent requests return the custom 499 response code. The circuit remains open for 60s.

While the circuit is open, try to access other paths of the httpbin service.

for i in {1..100};  do kubectl exec -it deploy/sleep -- curl httpbin:8000/status/503 -I | grep 'HTTP';  echo ''; sleep 0.1; done;

Expected output:

Click to view details

HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
HTTP/1.1 503 Service Unavailable
...

Requests to other paths of the service are not affected by the circuit breaking configuration on the error-route route entry and can still return the response content of the httpbin service.

Slow request

Slow request-based circuit breaking trips the circuit when the number of "slow requests"—requests that exceed a response time threshold—surpasses a limit within a given time window.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > Circuit Breaking and Degradation.

On the Create page, paste the following content into the YAML editor and click Create.

Click to view the YAML file

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMCircuitBreaker
metadata:
  name: httpbin-slow-circuitbreak
  namespace: default
spec:
  configs:
    - breaker_config:
        break_duration: 60s
        custom_response:
          body: delay break!
          header_to_add:
            x-envoy-overload: 'true'
          status_code: 498
        slow_request_rt: 0.5s
        max_slow_requests: 5
        min_request_amount: 5
        window_size: 10s
      match:
        vhost:
          name: httpbin.default.svc.cluster.local
          port: 8000
          route:
            name_match: delay-route
  workloadSelector:
    labels:
      app: sleep

The following table describes the parameters in the circuit breaking configuration.

Parameter	Description
workloadSelector.labels	The downstream service workload to which the circuit breaking rule applies. In this example, the downstream service is the sleep service, which is selected by the `app: sleep` label.
break_duration	The duration the circuit remains open after tripping. In this example, the value is 60s.
window_size	The time window for circuit breaking detection. In this example, the value is 10s. This means that if the number of slow requests on the route exceeds the specified threshold within 10 seconds, the circuit breaker trips and rejects requests.
slow_request_rt	The response time threshold for a slow request. In this example, the value is 0.5s. This means that a request with a response time greater than 0.5s is considered a slow request.
max_slow_requests	The number of slow requests required to trip the circuit breaker within the time window. In this example, the value is 5. This means that if more than 5 slow requests occur within a 10s time window, the circuit breaker trips and rejects requests.
min_request_amount	The minimum number of requests required to trip the circuit breaker within the time window. This parameter prevents the circuit breaker from being tripped by a small number of requests. In this example, the value is 5. This means that the circuit breaker trips only if more than 5 requests are sent on the route within a 10s time window and the number of slow requests exceeds 5.
custom_response	The custom response returned by the mesh proxy for requests rejected after the circuit breaker trips. `body` is set to `delay break!`. The response body is `delay break!`. `header_to_add` is set to `x-envoy-overload: 'true'`. The `x-envoy-overload: 'true'` header is added to the response when the circuit is open. `status_code` is set to `498`. The response code for the request is `498` after the circuit is open.
match.vhost	The route entry for which circuit breaking is configured. The route entry must match a specific route entry declared in the virtual service. `name`: The domain name of the upstream service in the call chain. In this example, this parameter is set to `httpbin.default.svc.cluster.local`, which is the domain name of the httpbin service (the upstream service of the sleep service). `port`: The service port of the upstream service. In this example, this parameter is set to `8000`, which is the service port of the httpbin service. `route.name_match`: The actual name of the route entry in the virtual service. The circuit breaking configuration takes effect on this route. In this example, this parameter is set to `delay-route`, which is the route entry created in Step 1. You can manually specify a response time of more than 0.5 seconds for requests that match this route. This ensures that the circuit breaker can be tripped.

Connect to the ACK cluster by using kubectl and run the following command:

for i in {1..100};  do kubectl exec -it deploy/sleep -- curl httpbin:8000/delay/1 -I | grep 'HTTP';  echo ''; sleep 0.1; done;

Expected output:

Click to view details

HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 200 OK
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
HTTP/1.1 498 Unknown
...

As the output shows, the circuit breaker is tripped on the sixth request. Subsequent requests return the custom 498 response code. The circuit remains open for 60s.

While the slow request-based circuit breaker is active, run the following command to test the error rate-based circuit breaker that you configured in the previous section.

for i in {1..100};  do kubectl exec -it deploy/sleep -- curl httpbin:8000/status/500 -I | grep 'HTTP';  echo ''; sleep 0.1; done;

Expected output:

Click to view details

HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 500 Internal Server Error
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
HTTP/1.1 499 Unknown
...

The output shows that the circuit breaking rules that are configured on different route entries operate independently. This allows you to configure specific circuit breaking policies for different types of east-west traffic.

Related operations

View service-level circuit breaking metrics

In ASM v1.22.6.28 and later, you can view and use service-level circuit breaking metrics for ASMCircuitBreaker.

Metric	Type	Description
envoy_asm_circuit_breaker_total_broken_requests	Counter	The total number of requests that are rejected by ASMCircuitBreaker.

You can configure the proxyStatsMatcher for the sidecar proxy to report the metrics.

When you configure proxyStatsMatcher, select Regular Expression Match and set the value to .*circuit_breaker.*. For more information, see proxyStatsMatcher.
Redeploy the httpbin application to apply the new proxy configuration. For more information, see Redeploy workloads.
Perform Step 1 and Step 2 again to reconfigure circuit breaking.

Run the following command to view the service-level circuit breaking metrics for the httpbin service:

kubectl exec -it deploy/httpbin -c istio-proxy -- curl localhost:15090/stats/prometheus|grep asm_circuit_breaker

Expected output:

# TYPE envoy_asm_circuit_breaker_total_broken_requests counter
envoy_asm_circuit_breaker_total_broken_requests{cluster="outbound|8000||httpbin.default.svc.cluster.local",uuid="af7cf7ad-67e8-49c5-b5fe-xxxxxxxxx"} 1430
# TYPE envoy_total_asm_circuit_breakers gauge
envoy_total_asm_circuit_breakers{} 1

Configure metric collection and alerting

After you configure the reporting of service-level circuit breaking metrics, you can collect these metrics in Prometheus and create alert rules for timely notifications of circuit breaking events. The following example shows how to configure service-level circuit breaking metric collection and alerting by using Managed Service for Prometheus.

In Managed Service for Prometheus, add the Alibaba Cloud ASM component to the data plane cluster or upgrade the component to the latest version to ensure that Managed Service for Prometheus can collect the exposed circuit breaking metrics. For more information about how to add a component, see Component management. If you have already integrated a self-managed Prometheus instance to monitor the service mesh to collect service mesh metrics, you do not need to perform additional operations.

Create an alert rule for service-level circuit breaking. For more information, see Create a Prometheus alert rule by using a custom PromQL statement. The following table provides an example of how to configure key parameters for an alert rule. You can configure other parameters based on your business requirements as described in the documentation.

Parameter	Example	Description
Custom PromQL statement	(sum by(cluster, namespace) (increase(envoy_asm_circuit_breaker_total_broken_requests[1m]))) > 0	Queries the number of rejected requests within the last minute and groups the results by the namespace and name of the upstream service. An alert is triggered when the number of rejected requests in one minute is greater than 0.
Alert content	Service-level circuit breaking occurred! Namespace: {{$labels.namespace}}, Target service where circuit breaking occurred: {{$labels.cluster}}. Number of requests rejected due to circuit breaking in the last minute: {{ $value }}	The format of the alert message. The message displays the namespace and name of the affected service, and the number of rejected requests to that service in the last minute.