System protection provides node-level traffic protection for a range of unexpected situations. For example, when an interface that has no traffic protection rule configured encounters a traffic spike, system protection acts as a safety net that ensures the stability of your application.
For the relationship between system protection and traffic protection, see System protection vs. traffic protection.
Protection features
Microservices Governance provides the following system protection features for server-side traffic and client-side traffic. Use the trigger metric and the load characteristics of your application to decide which features to configure.
| Feature | Trigger metric | Applies to | Minimum agent version |
| Adaptive overload protection | CPU utilization | All server-side interfaces | 3.1.4 or later |
| Total QPS throttling | Total QPS of a node | All server-side interfaces | 4.2.0 or later |
| Total concurrency throttling | Total concurrency of a node | All server-side interfaces | 4.2.0 or later |
| Abnormal call circuit breaking | Error ratio of an interface | All client-side interfaces, except interfaces that have interface-level circuit breaking rules configured | 4.2.0 or later |
| Slow call circuit breaking | Slow call ratio of an interface | All client-side interfaces, except interfaces that have interface-level circuit breaking rules configured | 4.2.0 or later |
Limits
Before you configure system protection, note the following limits:
Precedence — Adaptive overload protection, total QPS throttling, and total concurrency throttling have a lower priority than traffic protection rules.
Response status code — When throttling is triggered, system protection returns the 429 status code. Custom status codes are not supported.
Prerequisites
Connect your application to Microservices Governance. For more information, see Connect an ACK microservice application to the MSE governance center and Connect an ECS microservice application to the MSE governance center.
The agent of your application runs a version that supports the features that you want to configure.
Procedure
-
Log on to the MSE console, and select a region in the top navigation bar.
-
In the left-side navigation pane, choose Microservices Governance > Application Governance.
-
On the Application list page, click the resource card of the desired application. In the left-side navigation pane, click Traffic management.
Click the System protection tab. For each feature that you want to use, set Mode and configure the thresholds that are described in the corresponding section of this topic.
To confirm that a feature protects traffic as expected, check the event list on the left side of the System protection tab and the trend chart on the right side. To stop a feature from throttling traffic, set its Mode to the disabled option.
Adaptive overload protection
How it works
Adaptive overload protection uses CPU utilization to measure system load and adaptively adjusts the throttling ratio for server-side traffic. Even during unexpected traffic spikes, it keeps CPU utilization relatively stable within the configured threshold.
Scenarios
Adaptive overload protection provides CPU-based fallback protection for server-side interfaces. Use it for CPU-bound applications, in which a spike on an unexpected interface drives up system CPU utilization and affects the response time (RT) of core interfaces.
Console
The System protection tab lists adaptive overload protection events on the left side. The right side shows the average CPU utilization trend of the application nodes over the last 5 minutes.
The top of the page contains the Mode setting for adaptive overload protection. In the CPU utilization line chart on the right, a blue dashed line marks the protection threshold, which is the protection watermark threshold.
Events are reported at the node level based on algorithm state changes. They include throttling start, throttling ongoing, and throttling end events.
Click the View link in the Actions column of an event to query the CPU utilization data of the corresponding node IP address. The timeline rewinds to the time when the event was reported, so that you can observe the CPU utilization and the throttling probability of the node when the event was triggered.
| Parameter | Description |
| Mode |
|
| CPU utilization | The expected CPU utilization threshold. Adaptive overload protection combines the actual system CPU utilization with the configured CPU utilization threshold and uses an algorithm to adaptively adjust the throttling probability of interfaces. Under heavy load, the system rejects a portion of requests to keep CPU utilization fluctuating within a small range around the configured threshold. Steady-state CPU utilization varies with the business of each application. Determine the maximum steady-state CPU utilization by stress testing or from historical data, and then add a margin to that value. |
| Exceptions | For more information, see Configure exceptions. |
Total QPS throttling
How it works
Total QPS throttling tracks the total QPS at the node level, which is the sum of the QPS of all server-side interfaces on a single node, and throttles requests that exceed the threshold.
Scenarios
Not all systems are strongly CPU-bound. Some applications degrade in performance even at low CPU load because of memory, network, or other constraints. Total QPS throttling throttles traffic based on the total QPS of a node, which provides a traffic-based protection method.
Use total QPS throttling when a spike on an unexpected interface leads to contention for scarce resources, which affects core interfaces.
Console
The System protection tab lists total QPS throttling events on the left side. The right side shows the trend of the average total QPS request data of the application nodes over the last 5 minutes.
The line chart on the right displays the Total QPS, Passed QPS, and Blocked QPS trend curves together with the threshold reference line, which helps you observe the throttling effect.
Events are reported at the node and interface level based on the nodes and interfaces on which total QPS throttling actually occurred. Events are reported every 5 minutes and cover the throttling that occurred in the preceding 5 minutes.
Click the View link in the Actions column of an event to query the total QPS of the corresponding node IP address. The timeline rewinds to a point near the time when the event was reported, so that you can check whether the total QPS of the node and the throttling behavior meet your expectations. The interface details page and the node details page provide more detailed information, such as data at the interface and node level.
| Parameter | Description |
| Mode |
|
| Total QPS threshold | The total QPS threshold at the node level. Determine the steady-state total QPS of a node by stress testing or from historical data, and then add a margin to that value. |
| Exceptions | For more information, see Configure exceptions. |
Total concurrency throttling
How it works
Total concurrency throttling tracks the total concurrency at the node level, which is the sum of the concurrent requests of all server-side interfaces on a single node, and throttles requests that exceed the threshold.
Scenarios
In scenarios with a high call RT, generally more than 1 second, QPS throttling shows an obvious problem. When contended system resources such as thread pools, memory, and connection pools are occupied, requests queue up and the RT of the interface rises further. If you rely only on QPS-based throttling, a small number of requests still enter the system every second, but the queued requests cannot be processed within seconds. The queue grows, the RT rises further, and the RT of both new and existing requests increases significantly.
With concurrency throttling, new requests are rejected immediately while a certain number of requests remain unfinished. Requests are throttled, but after the system finishes the current requests, new requests pass with a shorter queuing time. Overall, both the success rate and the average RT of requests improve significantly.
Use total concurrency throttling when a spike on an unexpected interface leads to contention for scarce resources, queue buildup, and a higher RT for all requests.
Console
The System protection tab lists total concurrency throttling events on the left side. The right side shows the trend of the average total concurrent request data of the application nodes over the last 5 minutes.
Events are reported at the node and interface level based on the nodes and interfaces on which total concurrency throttling actually occurred. Events are reported every 5 minutes and cover the throttling that occurred in the preceding 5 minutes.
Click the View link in the Actions column of an event to query the total concurrency of the corresponding node IP address. The timeline rewinds to a point near the time when the event was reported, so that you can check whether the total concurrency of the node and the throttling behavior meet your expectations. The interface details page and the node details page provide more detailed information, such as data at the interface and node level.
| Parameter | Description |
| Mode |
|
| Total concurrency threshold | The total concurrency threshold at the node level. Determine the steady-state total concurrency of a node by stress testing or from historical data, and then add a margin to that value. |
| Exceptions | For more information, see Configure exceptions. |
Abnormal call circuit breaking
How it works
Abnormal call circuit breaking tracks the error ratio of each client-side interface. When the error ratio exceeds the configured threshold, the circuit breaker trips for that interface. During circuit breaking, requests to the interface fail fast. At intervals, the system passes a probe request. When a probe request succeeds, circuit breaking ends.
Scenarios
Abnormal call circuit breaking mainly covers two scenarios:
Timeout errors — A high timeout error ratio on a client-side interface usually indicates that the service provider has a problem. This causes requests to pile up in the caller, which is your application, and affects the other interfaces of your application. Circuit breaking makes these requests fail fast while the provider is abnormal, which prevents the pileup.
Non-timeout errors — When the non-timeout error ratio of a client-side interface is too high, abnormal call circuit breaking throws a throttling exception that you can handle. This provides a degradation effect and improves the user experience in abnormal situations.
Console
The System protection tab lists abnormal call circuit breaking events on the left side. The right side shows the top 10 interfaces by error ratio for the application over the last 5 minutes.
Events are reported at the node and interface level based on the nodes and interfaces on which abnormal call circuit breaking actually occurred. Events are reported every 5 minutes and cover the throttling that occurred in the preceding 5 minutes.
| Parameter | Description |
| Mode |
|
| Circuit breaking ratio threshold (%) | The circuit breaking ratio threshold at the interface level. |
| Exceptions | For more information, see Configure exceptions. |
Advanced settings
| Parameter | Description |
| Statistical window duration (seconds) | The length of the statistical time window. Valid values: 1 second to 120 minutes. |
| Circuit breaking duration (s) | The period for which circuit breaking lasts after it is triggered. After a resource enters the circuit breaking state, all requests fail fast within the configured circuit breaking duration. |
| Minimum number of requests | The minimum number of requests required to trigger circuit breaking. If the number of requests in the current statistical window is less than this value, the rule is not triggered even if the circuit breaking condition is met. |
| Circuit breaking recovery strategy | The recovery strategy that the circuit breaker uses when it enters the recovery phase, which is the half-open state.
|
Slow call circuit breaking
How it works
Slow call circuit breaking tracks the slow call ratio of each client-side interface. When the slow call ratio exceeds the configured threshold, the circuit breaker trips for that interface. During circuit breaking, requests to the interface fail fast. At intervals, the system passes a probe request. When a probe request succeeds, circuit breaking ends.
Scenarios
Slow call circuit breaking covers basically the same scenario as the timeout scenario of abnormal call circuit breaking. The difference is that you can dynamically adjust the RT criterion that defines a slow call, without depending on the timeout configuration.
Console
The System protection tab lists slow call circuit breaking events on the left side. The right side shows the top 10 interfaces by average RT for the application over the last 5 minutes.
Events are reported at the node and interface level based on the nodes and interfaces on which slow call circuit breaking actually occurred. Events are reported every 5 minutes and cover the throttling that occurred in the preceding 5 minutes.
| Parameter | Description |
| Mode |
|
| Slow call RT (ms) | Requests with a response time longer than this value are considered slow calls. |
| Degradation threshold (%) | When the percentage of requests with an RT longer than the configured slow call RT exceeds this threshold, circuit breaking is triggered. |
| Exceptions | For more information, see Configure exceptions. |
Advanced settings
| Parameter | Description |
| Statistical window duration (seconds) | The length of the statistical time window. Valid values: 1 second to 120 minutes. |
| Circuit breaking duration (s) | The period for which circuit breaking lasts after it is triggered. After a resource enters the circuit breaking state, all requests fail fast within the configured circuit breaking duration. |
| Minimum number of requests | The minimum number of requests required to trigger circuit breaking. If the number of requests in the current statistical window is less than this value, the rule is not triggered even if the circuit breaking condition is met. |
| Circuit breaking recovery strategy | The recovery strategy that the circuit breaker uses when it enters the recovery phase, which is the half-open state.
|
Exceptions
How it works
You can configure exceptions for all system protection features. System protection always passes the interfaces in the exception list without checking the rules.
Exceptions require agent version 4.2.0 or later.
Scenarios
In most cases, you only need to configure exceptions for health check endpoints and critical system interfaces. Exceptions for health check endpoints prevent the node health status from being affected. Critical system interfaces have their own interface-level throttling limits and are expected to stay unaffected by system-level throttling.
Console
The System protection tab lists the interfaces that you can select directly on the left side. These are the interfaces called recently. For an interface that is not listed on the left, enter its name in the input box and press Enter to add it to the selected interfaces.
You can remove selected interfaces one by one by clicking × or clear them all by clicking Remove all. System protection does not apply the configured rules to the selected interfaces.
System protection vs. traffic protection
Both system protection and traffic protection keep an application in a steady state, but they cover different scenarios and cause different levels of traffic loss.
System protection provides traffic protection based on node-level metrics. It keeps the application itself in a steady state and covers most scenarios. However, system protection works from the perspective of the application and treats all interfaces equally, whereas interfaces in the same application differ in importance and in their impact on system load. Traffic protection lets you configure a different threshold for each interface, so it covers more scenarios and throttles as little traffic as possible while providing the same protection.
Overall, both system protection and traffic protection are effective. Traffic protection performs better in scenario coverage and traffic loss, and system protection is easier to configure. Therefore, the best practice is to combine both: use system protection to ensure application stability, and use traffic protection to reduce loss, which is the throttled traffic, through fine-grained configuration without compromising the protection effect.
References
For more information about traffic protection policies, see Traffic protection.