The traffic scheduling suite of Service Mesh (ASM) supports priority-based request scheduling policies. When the system is overloaded, high-priority requests are processed first. You can use AverageLatencySchedulingPolicy to prioritize requests based on configurable priority levels.
Background information
A priority-based request scheduling policy compares real-time latency with the historical average to detect traffic overload, then uses token bucket and priority mechanisms to schedule requests. The workflow is as follows:
-
Overload detection: This policy compares the average latency in the past period with the current latency to determine whether the system is overloaded.
-
Adjustment of the token issuance rate: If an overload occurs, the monitoring data obtained in the previous step is sent to a dedicated controller, which will control the fill rate of the token bucket.
-
Request scheduling: Different requests have different priorities. When an overload occurs, requests with higher priorities have a greater chance of obtaining tokens.
This policy queues requests when services are congested due to high concurrency and rising response latencies. Unlike standard throttling policies that directly reject requests, this policy places them in a priority queue. The token bucket mechanism limits the request rate, and request processing order is adjusted based on priority levels.
Prerequisites
-
A Container Service for Kubernetes (ACK) managed cluster is added to your ASM instance, and the version of your ASM instance is V1.21.6.44 or later. For more information, see Add a cluster to an ASM instance.
-
Automatic sidecar proxy injection is enabled for the default namespace in the ACK cluster. For more information, see Manage global namespaces.
-
You have connected to the ACK cluster by using kubectl. For more information, see Connect to an ACK cluster using kubectl.
-
The ASM traffic scheduling suite is enabled. For more information, see Enable the ASM traffic scheduling suite.
-
The HTTPBin application is deployed and can be accessed by using an ASM gateway. For more information, see Deploy the HTTPBin application.
-
(Optional) The features of generating and collecting access logs of the ASM gateway are enabled. For more information, see Generate and collect ASM gateway access logs.
Step 1: Create AverageLatencySchedulingPolicy
-
Use kubectl to connect to the ASM instance. For more information, see Access Istio resources with kubectl.
-
Create an AverageLatencySchedulingPolicy.yaml file that contains the following content:
apiVersion: istio.alibabacloud.com/v1 kind: AverageLatencySchedulingPolicy metadata: name: workload-prioritization namespace: istio-system spec: load_scheduling_core: aimd_load_scheduler: load_scheduler: workload_latency_based_tokens: true selectors: - service: httpbin.default.svc.cluster.local scheduler: workloads: - label_matcher: match_labels: http.request.header.user_type: "guest" parameters: priority: 50.0 name: "guest" - label_matcher: match_labels: http.request.header.user_type: "subscriber" parameters: priority: 200.0 name: "subscriber"The following table describes some of the fields.
Field
Description
workload_latency_based_tokens
Indicates whether to dynamically adjust the number of tokens based on the average latency of the workload. The time window for estimating the average latency is the last 30 minutes.
service
The service on which the scheduling policy takes effect.
workloads
Two types of requests are defined based on user_type in the request headers: guest and subscriber. The priority of a request of the guest type is 50, and that of a request of the subscriber type is 200.
For more information about the fields supported by AverageLatencySchedulingPolicy, see AverageLatencySchedulingPolicy field reference.
Step 2: Perform tests
In this example, the load testing tool fortio is used. For more information, see Install fortio.
First, simulate normal service requests to establish a baseline average latency:
fortio load -c 20 -qps 100000 -t 60m http://${IP address of the ASM gateway}/status/200Three minutes after running the preceding command, open two new terminals and send test requests. Keep the fortio process running throughout the test and do not close its terminal.
In the two terminals, run the following load testing commands. Start both tests at the same time.
fortio load -c 40 -qps 100000 -H "user_type:guest" -t 3m http://${IP address of the ASM gateway}/status/201fortio load -c 40 -qps 100000 -H "user_type:subscriber" -t 3m http://${IP address of the ASM gateway}/status/202The two commands use different request paths so that you can distinguish the test results in access logs.
Step 3: Analyze the test results
The following code blocks show the last few lines of the test output in the current environment:
Code 201 : 26852 (97.8 %)
Code 503 : 601 (2.2 %)
Response Header Sizes : count 27453 avg 242.91564 +/- 36.35 min 0 max 249 sum 6668763
Response Body/Total Sizes : count 27453 avg 246.17754 +/- 14.56 min 149 max 249 sum 6758312
All done 27453 calls (plus 40 warmup) 262.318 ms avg, 152.4 qpsCode 202 : 52765 (100.0 %)
Response Header Sizes : count 52765 avg 248.86358 +/- 0.5951 min 248 max 250 sum 13131287
Response Body/Total Sizes : count 52765 avg 248.86358 +/- 0.5951 min 248 max 250 sum 13131287
All done 52765 calls (plus 40 warmup) 136.472 ms avg, 292.9 qpsThe results indicate that:
-
All subscriber requests succeeded with no HTTP 503 errors, while 2.2% of guest requests returned HTTP 503.
-
The average latency for subscriber requests was about 136 ms, compared to about 262 ms for guest requests. The queries per second (QPS) also differed significantly between the two user types.
This confirms that AverageLatencySchedulingPolicy prioritizes subscriber requests when latency increases.
The test results in this topic are for reference only. Actual data depends on your environment. Only the relative differences between the data points are analyzed here.
(Optional) Step 4: Analyze the test results based on the access logs of the ASM gateway
-
Log on to the ASM console. In the left-side navigation pane, choose .
-
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose .
-
Click the name of the ASM gateway to go to the Gateway overview page. Then, click Gateway Logs to view the access logs of the ASM gateway.
-
The access paths of the guest and subscriber users are different. You can use the access paths to retrieve the access results of the two users respectively:
References
You can verify whether AverageLatencySchedulingPolicy takes effect in Grafana. Make sure the Prometheus instance for Grafana is configured with the ASM traffic scheduling suite.
Import the following content into Grafana to create a dashboard for AverageLatencySchedulingPolicy.
The dashboard is as follows.
