Documentation
Home Page Alibaba Cloud Service Mesh ASM Sidecar Mode Use Cases Best Practices Configure Grafana for circuit breaking and rate limiting

Configure Grafana for circuit breaking and rate limiting

Updated at:
Copy as MD
Product

Service Mesh (ASM) supports non-intrusive circuit breaking, throttling, and traffic monitoring. This topic covers common dashboard configurations and alert rules for circuit breaking and throttling.

Circuit breaking

Circuit breaking is an overload protection mechanism that prevents a system from crashing due to sudden traffic spikes. In an architecture with east-west traffic between cloud-native services, a failure in a single service, such as slow responses or a high error rate, can trigger cascading failures across the call chain.

By configuring a circuit breaking rule for east-west traffic between services, you can automatically reject requests from upstream services when the request failure rate or the count of response timeouts reaches a threshold. This approach protects upstream services and prevents failures from spreading across the entire call chain.

After you configure a circuit breaking rule, each ASM proxy independently calculates the request failure rate or the count of response timeouts based on the requests it proxies. Therefore, for the same failing upstream service, different client-side ASM proxies might trip the circuit breaker at slightly different times.

Dashboard example

image

The following table describes each panel:

Panel

Description

Requests

Provides an overview of requests, including the following metrics:

  • Total requests (total_requests), which corresponds to the Requests total panel.

  • Throttled requests (throttled_requests), which corresponds to the Requests throttled panel.

  • Successful requests (ok_requests), which corresponds to the Requests OK panel.

Requests OK percent

The percentage of successful requests out of total requests.

Requests throttled percent

The percentage of throttled requests out of total requests.

You can import the following JSON into Grafana to create a circuit breaking dashboard for your cluster. You can also modify this JSON or use it as a template to create your own dashboard.

JSON content

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 168,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "stacking": {
              "group": "A",
              "mode": "none"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "total_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "blue",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "unlimited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "limited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-red",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none",
          "targets": []
        }
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "format": "time_series",
          "instant": false,
          "legendFormat": "total_requests",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30 - sum(rate(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[60s])) * 30",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "legendFormat": "ok_requests",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(rate(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[60s])) * 30",
          "hide": false,
          "instant": false,
          "legendFormat": "throttled_requests",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "Requests",
      "transparent": true,
      "type": "aliyun-timeseries-panel"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 0,
        "y": 8
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Total",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 4,
        "y": 8
      },
      "id": 5,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[$__range])) - sum(increase(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 8,
        "y": 8
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Throttled",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "red",
                "value": null
              },
              {
                "color": "green",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 16
      },
      "id": 6,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "(sum(increase(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[$__range])) - sum(increase(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[$__range]))) / sum(increase(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[$__range])) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK Percent",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 10
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 16
      },
      "id": 7,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_asm_circuit_breaker_total_broken_requests{namespace=~\"$namespace\",pod_name=~\"$pod\",cluster=~\".*$service.*\"}[$__range])) / sum(increase(istio_requests_total{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[$__range])) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Throttled Percent",
      "type": "gauge"
    }
  ],
  "refresh": false,
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "datesource",
          "value": "datesource"
        },
        "hide": 0,
        "includeAll": false,
        "label": "datasource",
        "multi": false,
        "name": "DS_PROMETHEUS",
        "options": [],
        "query": "prometheus",
        "queryValue": "datesource",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (namespace))",
        "hide": 0,
        "includeAll": true,
        "label": "namespace",
        "multi": false,
        "name": "namespace",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (namespace))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*namespace=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (pod_name))",
        "description": "",
        "hide": 0,
        "includeAll": true,
        "label": "pod",
        "multi": false,
        "name": "pod",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (pod_name))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*pod_name=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (cluster))",
        "hide": 0,
        "includeAll": true,
        "multi": false,
        "name": "service",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_asm_circuit_breaker_total_broken_requests) by (cluster))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*_.*_.*_(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "2025-03-19T07:40:00.388Z",
    "to": "2025-03-19T07:43:09.493Z"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Circuit Breaker",
  "uid": "5d5bcd97-33a9-499f-94f6-aea7081b9777",
  "version": 5,
  "weekStart": ""
}

Alert rule example

Configuration

Example

Description

Custom PromQL expression

(sum by(cluster, namespace) (increase(envoy_asm_circuit_breaker_total_broken_requests[1m]))) > 10

Calculates the number of throttled requests over the last minute, grouped by cluster and namespace. This triggers an alert if the count exceeds 10.

Alert content

Service-level circuit breaking has occurred! Namespace: {{$labels.namespace}}, Target service: {{$labels.cluster}}. Number of throttled requests in the last minute: {{ $value }}

This defines the alert message format. The message includes the namespace, the target service, and the count of throttled requests in the last minute.

Global throttling

In an Envoy proxy, global throttling is a mechanism that controls the request rate across an entire service mesh. It relies on Envoy's rate limit service to centrally process traffic and enforce limits based on predefined rules and quotas.

global throttling can be combined with local throttling to provide layered protection.

Dashboard example

image

The following table describes each panel.

Panel

Description

Requests

Provides an overview of requests, including the following metrics:

  • Total requests (total_requests), which corresponds to the Requests Total panel.

  • Throttled requests (limited_requests), which corresponds to the Requests Reached Limits panel.

  • Allowed requests (unlimited_requests), which corresponds to the Requests OK panel.

Requests OK Percent

The percentage of allowed requests.

Requests Reached Limits Percent

The percentage of throttled requests.

You can import the following JSON into a Grafana dashboard to create a global throttling dashboard for your cluster. You can also modify this JSON or use it to create your own.

Dashboard JSON

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 166,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "stacking": {
              "group": "A",
              "mode": "none"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "total_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "blue",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "unlimited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "limited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-red",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none",
          "targets": []
        }
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30 + sum(rate(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "format": "time_series",
          "instant": false,
          "legendFormat": "total_requests",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "legendFormat": "unlimited_requests",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(rate(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "hide": false,
          "instant": false,
          "legendFormat": "limited_requests",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "Requests",
      "transparent": true,
      "type": "aliyun-timeseries-panel"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 0,
        "y": 8
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) + sum(increase(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Total",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 4,
        "y": 8
      },
      "id": 5,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 8,
        "y": 8
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Reached Limits",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "red",
                "value": null
              },
              {
                "color": "green",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 16
      },
      "id": 6,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) / (sum(increase(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) + sum(increase(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK Percent",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 10
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 16
      },
      "id": 7,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) / (sum(increase(envoy_cluster_ratelimit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) + sum(increase(envoy_cluster_ratelimit_over_limit{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Reached Limits Percent",
      "type": "gauge"
    }
  ],
  "refresh": "",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "datasource",
          "value": "datasource"
        },
        "hide": 0,
        "includeAll": false,
        "label": "datasource",
        "multi": false,
        "name": "DS_PROMETHEUS",
        "options": [],
        "query": "prometheus",
        "queryValue": "datasource",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (namespace) or sum(envoy_cluster_ratelimit_ok) by (namespace))",
        "hide": 0,
        "includeAll": true,
        "label": "namespace",
        "multi": false,
        "name": "namespace",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (namespace) or sum(envoy_cluster_ratelimit_ok) by (namespace))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*namespace=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (pod_name) or sum(envoy_cluster_ratelimit_ok) by (pod_name))",
        "description": "",
        "hide": 0,
        "includeAll": true,
        "label": "pod",
        "multi": false,
        "name": "pod",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (pod_name) or sum(envoy_cluster_ratelimit_ok) by (pod_name))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*pod_name=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (service_istio_io_canonical_name) or sum(envoy_cluster_ratelimit_ok) by (service_istio_io_canonical_name))",
        "hide": 0,
        "includeAll": true,
        "label": "service",
        "multi": false,
        "name": "service",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_cluster_ratelimit_over_limit) by (service_istio_io_canonical_name) or sum(envoy_cluster_ratelimit_ok) by (service_istio_io_canonical_name))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*service_istio_io_canonical_name=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Global Rate Limit",
  "uid": "d8f95bbc-efb6-41e7-8929-25551b22f3d2",
  "version": 25,
  "weekStart": ""
}

Alert rule example

Alert configuration

Example

Description

PromQL expression

sum (increase(envoy_cluster_ratelimit_over_limit[1m])) by (namespace, service_istio_io_canonical_name) > 10

Calculates the number of throttled requests over the last minute, grouped by namespace and service name. Triggers an alert if the count exceeds 10.

Alert message

Throttling triggered! Namespace: {{$labels.namespace}}, Target service: {{$labels.service_istio_io_canonical_name}}. Number of throttled requests in the last minute: {{ $value }}

Defines the alert message format. The message includes the namespace, service name, and the number of throttled requests in the last minute.

Local throttling

The Envoy proxy uses the token bucket algorithm to implement local throttling. The algorithm limits the number of requests sent to a service by maintaining a bucket of tokens, refilled at a constant rate. Each request to the service consumes one token. If the bucket is empty, the Envoy proxy rejects the request.

You can combine local throttling with global throttling to provide layered protection.

Dashboard example

image

The following table describes each panel:

Panel

Description

Requests

Shows an overview of requests, including these metrics:

  • total requests (total_requests), shown in the Requests Total panel.

  • throttled requests (limited_requests), shown in the Requests Reached Limits panel.

  • allowed requests (unlimited_requests), shown in the Requests OK panel.

Requests OK Percent

The percentage of total requests that are allowed.

Requests Reached Limits Percent

The percentage of total requests that are throttled.

Import the following JSON into a Grafana dashboard to create a local throttling dashboard for your cluster. You can also modify this JSON or use it as a template to create a custom dashboard.

Dashboard JSON

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 167,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "stacking": {
              "group": "A",
              "mode": "none"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "total_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "blue",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "unlimited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-green",
                  "mode": "fixed"
                }
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "limited_requests"
            },
            "properties": [
              {
                "id": "color",
                "value": {
                  "fixedColor": "dark-red",
                  "mode": "fixed"
                }
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none",
          "targets": []
        }
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(envoy_http_local_rate_limiter_http_local_rate_limit_enabled{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "format": "time_series",
          "instant": false,
          "legendFormat": "total_requests",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "sum(rate(envoy_http_local_rate_limiter_http_local_rate_limit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "legendFormat": "unlimited_requests",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(rate(envoy_http_local_rate_limiter_http_local_rate_limit_enforced{namespace=~\"$namespace\",pod_name=~\"$pod\",service_istio_io_canonical_name=~\"$service\"}[60s])) * 30",
          "hide": false,
          "instant": false,
          "legendFormat": "limited_requests",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "Requests",
      "transparent": true,
      "type": "aliyun-timeseries-panel"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 0,
        "y": 8
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_enabled{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Total",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 4,
        "y": 8
      },
      "id": 5,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 4,
        "x": 8,
        "y": 8
      },
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_enforced{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range]))",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Reached Limits",
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "red",
                "value": null
              },
              {
                "color": "green",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 16
      },
      "id": 6,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_ok{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) / sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_enabled{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests OK Percent",
      "type": "gauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "neutral": -1
          },
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 10
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 16
      },
      "id": 7,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.9",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_enforced{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) / sum(increase(envoy_http_local_rate_limiter_http_local_rate_limit_enabled{namespace=~\"$namespace\", pod_name=~\"$pod\", service_istio_io_canonical_name=~\"$service\"}[$__range])) * 100",
          "instant": false,
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Requests Reached Limits Percent",
      "type": "gauge"
    }
  ],
  "refresh": "",
  "schemaVersion": 38,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "datasource",
          "value": "datasource"
        },
        "hide": 0,
        "includeAll": false,
        "label": "datasource",
        "multi": false,
        "name": "DS_PROMETHEUS",
        "options": [],
        "query": "prometheus",
        "queryValue": "datasource",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (namespace))",
        "hide": 0,
        "includeAll": true,
        "label": "namespace",
        "multi": false,
        "name": "namespace",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (namespace))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*namespace=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (pod_name))",
        "description": "",
        "hide": 0,
        "includeAll": true,
        "label": "pod",
        "multi": false,
        "name": "pod",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (pod_name))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*pod_name=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      },
      {
        "allValue": ".*",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "ykIxRIhNk"
        },
        "definition": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (service_istio_io_canonical_name))",
        "hide": 0,
        "includeAll": true,
        "label": "service",
        "multi": false,
        "name": "service",
        "options": [],
        "query": {
          "query": "query_result(sum(envoy_http_local_rate_limiter_http_local_rate_limit_enabled) by (service_istio_io_canonical_name))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "/.*service_istio_io_canonical_name=\"(.*)\".*/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "browser",
  "title": "Local Rate Limit",
  "uid": "859fe8e7-c7dd-46f4-a1c0-c0c7010bcf19",
  "version": 7,
  "weekStart": ""
}

Alert rule example

Parameter

Example

Description

PromQL expression

sum (increase(envoy_http_local_rate_limiter_http_local_rate_limit_enforced[1m])) by (namespace, service_istio_io_canonical_name) > 10

Queries the number of throttled requests over the last minute, grouped by namespace and service name. The alert fires if the count exceeds 10.

Alert message

Throttling triggered! Namespace: {{$labels.namespace}}, Target service: {{$labels.service_istio_io_canonical_name}}. Number of throttled requests in the last minute: {{ $value }}

Defines the format for the alert message. It displays the namespace, service name, and the number of throttled requests in the last minute.

Related operations

Import a Grafana dashboard

  1. Log on to the ARMS console.

  2. In the navigation pane on the left, click Integration Management.

  3. On the Integrated Environments tab, select Container Service, search for the target environment by its cluster name, and click it. Click Dashboard Directory.

    The Dashboard Directory is in the Basic Information section of the Container Service details page. Click its value to open the Grafana dashboard directory.

  4. On the right side of the Dashboards tab, click Import.

  5. Paste the JSON content into the Import via panel json box and click Load. Keep the default settings and click Import.

    Note

    You can also import the dashboard by uploading a JSON file.

Previous：Connect to Consul service registriesNext：Use proxyless Service Mesh in gRPC services