How to use ASMAdaptiveConcurrency to implement adaptive concurrency control - Alibaba Cloud Service Mesh

ASMAdaptiveConcurrency dynamically adjusts the number of allowed concurrent requests based on sampled request data and rejects requests when concurrency exceeds service capacity, protecting your service from overload.

Prerequisites

An ASM instance is created and runs version 1.12.4.19 or later. For more information, see Create an ASM instance.
The cluster is added to the ASM instance.
A kubectl client is connected to the cluster. For more information, see Connect to an ACK cluster using kubectl.

Background information

When a service exceeds its load capacity, it must reject the excess requests to prevent cascading failures. Use a Service Mesh DestinationRule to configure basic circuit breaking. However, this approach requires you to specify a threshold that triggers circuit breaking, such as a specific number of pending requests. The Service Mesh data plane then rejects requests only when network traffic exceeds the circuit breaking configuration. In real-world scenarios, the load capacity of a service is difficult to estimate accurately.

ASMAdaptiveConcurrency uses an adaptive concurrency control algorithm instead of a fixed threshold. The algorithm periodically compares the sampled latency with the measured ideal latency. Based on a series of calculations, the algorithm dynamically adjusts the concurrency limit. This keeps the concurrency limit close to the capacity of the service and rejects the requests that exceed the limit. A rejected request returns HTTP 503 and the error message reached concurrency limit.

During each periodic MinRTT measurement, the number of connections is limited to a small value (min_concurrency). Therefore, after you enable AdaptiveConcurrency for a service, also use a DestinationRule to enable retries for the service. Retries allow the sidecar to resend the requests that are rejected during MinRTT calculation. Resent requests are more likely to return a result.

Step 1: Deploy the sample applications

Deploy testserver as the service to be protected and gotest as the load generator.

testserver — Processes 500 requests concurrently, queues the requests that exceed this limit, and takes 1000 ms to process each request.
gotest — Sends 200 requests at a time per replica. In gotest.yaml, replicas is set to 0 so that no load is generated before adaptive concurrency control is in place. You scale out gotest in Step 4.

Create a file named testserver.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testserver
  name: testserver
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testserver
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: testserver
    spec:
      containers:
      - args:
        - -m
        - "500"
        - -t
        - "1000"
        command:
        - /usr/local/bin/limited-concurrency-http-server
        image: registry.cn-hangzhou.aliyuncs.com/acs/asm-limited-concurrency-http-server:v0.1.1-gee0b08f-aliyun
        imagePullPolicy: IfNotPresent
        name: testserver
        ports:
        - containerPort: 8080
          protocol: TCP

Use -m to specify the number of concurrent requests that the application can handle and use -t to specify the request processing time.

Run the following command to deploy the testserver application.
```
kubectl apply -f testserver.yaml
```

Create a file named testservice.yaml with the following content.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: testserver
  name: testserver
  namespace: default
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: metrics
      port: 15020
      protocol: TCP
      targetPort: 15020
  selector:
    app: testserver
  type: ClusterIP

Run the following command to deploy the Service of testserver.
```
kubectl apply -f testservice.yaml
```

Create a file named gotest.yaml with the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: gotest
  name: gotest
  namespace: default
spec:
  replicas: 0
  selector:
    matchLabels:
      app: gotest
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gotest
    spec:
      containers:
      - args:
        - -c
        - "200"
        - -n
        - "10000"
        - -u
        - testserver:8080
        command:
        - /root/go-stress-testing-linux
        image: xocoder/go-stress-testing-linux:v0.1
        imagePullPolicy: Always
        name: gotest
        resources:
          limits:
            cpu: 500m

Run the following command to deploy the gotest application.

kubectl apply -f gotest.yaml

Step 2: Create an ASMAdaptiveConcurrency resource

Use kubectl on the control plane to access Istio resources.

Create a file named adaptiveconcurrency.yaml with the following content.

apiVersion: istio.alibabacloud.com/v1beta1
kind: ASMAdaptiveConcurrency
metadata:
    name: sample-adaptive-concurrency
    namespace: default
spec:
    workload_selector:
        labels:
            app: testserver
    sample_aggregate_percentile:
        value: 60
    concurrency_limit_params:
        max_concurrency_limit: 500
        concurrency_update_interval: 15s
    min_rtt_calc_params:
        interval: 60s
        request_count: 100
        jitter:
            value: 15
        min_concurrency: 50
        buffer:
            value: 25

The preceding file uses example values for the testserver application rather than default values: max_concurrency_limit is set to 500 to match the 500 concurrent requests that testserver can process, and min_concurrency is set to 50, which is far below that capacity, so that testserver can respond with the ideal latency during MinRTT calculation. The following table describes each parameter and its default value.

Parameter	Type	Description	Required
workload_selector	WorkloadSelector	The workload selector that specifies how to match the workloads to which the configuration applies.	Yes
workload_selector.labels	map	The labels of the workloads (pods) to be matched. The labels are used to match workloads.	Yes
sample_aggregate_percentile	Percent	The sampling percentile. The latency at this percentile of the sampled requests is used as the SampleRTT in calculations. Specify the percentile in the value field.	Yes
sample_aggregate_percentile.value	int	The percentage. Valid values: 0 to 100.	Yes
concurrency_limit_params	ConcurrencyLimitParams	The configurations related to the concurrency limit.	Yes
concurrency_limit_params.max_concurrency_limit	int	The maximum concurrency limit. Default value: 1000.	No
concurrency_limit_params.concurrency_update_interval	duration	The interval at which the maximum concurrency is calculated. For example, 15s.	Yes
min_rtt_calc_params	MinRTTCalcParams	The configurations for calculating MinRTT.	Yes
min_rtt_calc_params.interval	duration	The interval at which the ideal round-trip time is calculated. For example, 60s.	No
min_rtt_calc_params.request_count	int	The number of requests whose data is used to calculate the ideal round-trip time. Default value: 50.	No
min_rtt_calc_params.jitter	Percent	The random delay that is added when the ideal round-trip time is calculated. Specify the percentage in the value field.	No
min_rtt_calc_params.jitter.value	int	The percentage of the random delay. For example, if interval is 120s and jitter is 50, the interval is random(120, 120 + (120 * 50%)). Default value: 15.	No
min_rtt_calc_params.min_concurrency	int	The number of concurrent requests that is used when the ideal round-trip time is calculated. This value is also the initial concurrency value of the controller. Set this value far below the bottleneck of the service so that the service can return a response with the ideal latency at this concurrency. Default value: 3.	No
min_rtt_calc_params.buffer	Percent	The acceptable range of latency fluctuation. Specify the percentage in the value field.	No
min_rtt_calc_params.buffer.value	int	The percentage of acceptable latency fluctuation. For example, if a fluctuation of 10% is acceptable for a latency of about 100 ms, set this value to 10. Default value: 25.	No

Run the following command to create the ASMAdaptiveConcurrency resource.

kubectl apply -f adaptiveconcurrency.yaml

Step 3: Enable Managed Service for Prometheus (optional)

Managed Service for Prometheus provides an intuitive view of how the concurrency controller runs and simplifies parameter tuning. Complete this step if you want to observe the concurrency controller on a Grafana dashboard, because the dashboard in Step 4 uses the metrics that you export here. If you only need to know whether the concurrency limit takes effect, you can skip this step: the requests that are rejected by the concurrency limit return HTTP 503 and the error message reached concurrency limit.

Enable Managed Service for Prometheus. For more information, see Connect and configure Managed Service for Prometheus.

Create a file named servicemonitor.yaml with the following content. The ServiceMonitor allows Managed Service for Prometheus to collect data from testserver.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: testserver-envoy-metrics
  namespace: default
spec:
  endpoints:
    - interval: 5s
      path: /stats/prometheus
      port: metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: testserver

In the KubeConfig environment of the cluster, run the following command to create the ServiceMonitor.

kubectl apply -f servicemonitor.yaml

Step 4: Verify that adaptive concurrency control takes effect

Scale out gotest to send more requests than testserver can process concurrently, and then check how the concurrency controller responds. The dashboard operations in this section require the metrics that you export in Step 3.

Set the number of replicas of gotest to 5. One replica of the gotest application sends 200 requests. Therefore, five replicas send 1,000 requests.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Cluster List page, click the name of the destination cluster or click Details in the Actions column.
3. In the left navigation pane of the cluster management page, choose Workload > Deployments.
4. On the Deployments page, set Namespace to default. In the Actions column of the gotest application, choose More > View in YAML.
5. In the Edit YAML dialog box, set replicas to 5 and then click Update.
Import a dashboard for Grafana by using the JSON definition provided in the Appendix of this topic, and then view the running status of the concurrency controller. For more information, see the ARMS documentation.
In the dashboard, select the cluster and set service to testserver and pod to ALL.
Expected result: The gotest application sends 1,000 requests to the testserver application, but the testserver application always limits the number of concurrent requests that it receives to fewer than 500. This indicates that the adaptive concurrency control of ASMAdaptiveConcurrency takes effect.

Appendix: Grafana dashboard definition

Use the following JSON definition to import the dashboard for Grafana in Step 4.

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "description": "monitoring ASM Adaptive Concurrency",
  "editable": true,
  "gnetId": 6693,
  "graphTooltip": 0,
  "id": 3239002,
  "iteration": 1651922323976,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 22,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_rq_blocked{service=\"$service\", pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "RqBlocked",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 24,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size{service=\"$service\", pod=\"$pod\"}",
          "format": "time_series",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "HeadRoom",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "hiddenSeries": false,
      "id": 26,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "ConcurrencyLimit",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "hiddenSeries": false,
      "id": 28,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_gradient{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Gradient",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 32,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "MinRTT(msec)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ms",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$cluster",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 16
      },
      "hiddenSeries": false,
      "id": 34,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_sample_rtt_msecs{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "SampleRTT(msec)",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ms",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "test-adaptive-concurrency_1217520382582089",
      "fieldConfig": {
        "defaults": {
          "custom": {}
        },
        "overrides": []
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 24
      },
      "hiddenSeries": false,
      "id": 30,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "7.4.0-pre",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_min_rtt_calculation_active{service=\"$service\",pod=\"$pod\"}",
          "interval": "",
          "legendFormat": "{{service}}-{{pod}}",
          "queryType": "randomWalk",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "MinRTTCalc",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "5s",
  "schemaVersion": 26,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": "edas120_1217520382582089",
          "value": "edas120_1217520382582089"
        },
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "cluster",
        "options": [],
        "query": "prometheus",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": null,
        "current": {
          "isNone": true,
          "selected": false,
          "text": "None",
          "value": ""
        },
        "datasource": "$cluster",
        "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
        "error": null,
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "service",
        "options": [],
        "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_burst_queue_size,service)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": "$cluster",
        "definition": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
        "error": null,
        "hide": 0,
        "includeAll": true,
        "label": null,
        "multi": true,
        "name": "pod",
        "options": [],
        "query": "label_values(envoy_http_inbound_0_0_0_0_8080_adaptive_concurrency_gradient_controller_concurrency_limit, pod)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "ASM Adaptive Concurrency",
  "uid": "000000084",
  "version": 3
}