Configure buckets for histogram metrics - Alibaba Cloud Service Mesh

Prometheus histogram metrics track data distribution -- such as request latency or response size -- across predefined buckets. Each bucket counts observations at or below its upper bound, enabling percentile calculations and SLO-based alerting.

Default bucket boundaries cover a wide range but may not match your monitoring needs. For example, if your SLO requires 95% of requests to complete within 300 ms, placing a bucket boundary at that threshold lets you measure compliance directly. Removing unused boundaries also reduces the number of time series stored in Prometheus.

In Service Mesh (ASM), you override histogram bucket boundaries for both Istio and Envoy metrics by adding a pod annotation.

How it works

Add the sidecar.istio.io/statsHistogramBuckets annotation to your application pods. The annotation value is a JSON object that maps metric prefixes to arrays of bucket boundaries.

Istio matches metrics by prefix. A single prefix entry applies to every histogram metric that shares that prefix. For example, configuring istiocustom applies the custom buckets to all three Istio histogram metrics (istio_request_duration_milliseconds, istio_request_bytes, and istio_response_bytes).

Annotation format:

{
  "<metric-prefix>": [<boundary>, <boundary>, ...],
  "<metric-prefix>": [<boundary>, <boundary>, ...]
}

Supported metrics

Metric type	Metric prefix	Covered metrics
Istio	`istiocustom`	`istio_request_duration_milliseconds`, `istio_request_bytes`, `istio_response_bytes`
Envoy	`cluster_manager`	Cluster manager statistics
Envoy	`listener_manager`	Listener manager statistics
Envoy	`server`	Server-level statistics
Envoy	`cluster.xds-grpc`	xDS gRPC cluster statistics

For details, see Istio Standard Metrics and Envoy Statistics.

Default bucket boundaries

When no custom buckets are configured, Envoy uses the following default boundaries:

[0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 300000, 600000, 1800000, 3600000]

Prerequisites

Before you begin, ensure that you have:

A cluster added to an ASM instance of V1.19 or later. For more information, see Add a cluster to an ASM instance

Configure histogram buckets

You can apply custom bucket boundaries through either a YAML manifest (recommended) or a kubectl patch command.

Option 1: YAML manifest (recommended)

Add the sidecar.istio.io/statsHistogramBuckets annotation to the pod template in your Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
      annotations:
        sidecar.istio.io/statsHistogramBuckets: '{"istiocustom":[1,5,10],"cluster.xds-grpc":[1,5,10]}'
    spec:
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/httpbin:0.1.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Apply the manifest:

kubectl apply -f deployment.yaml

Option 2: kubectl patch

Patch an existing pod to add or update bucket boundaries:

kubectl patch pod <POD_NAME> -p '{"metadata":{"annotations":{"sidecar.istio.io/statsHistogramBuckets": {"istiocustom":[1,5,10],"cluster.xds-grpc":[1,5,10]}}}}'

Replace <POD_NAME> with the name of your pod.

Demo: Modify xds-grpc metric buckets

This walkthrough deploys a sample HTTPBin application, inspects default Envoy histogram buckets, applies custom boundaries, and verifies the change.

Step 1: Deploy the sample application

Create the following Deployment. For more information, see Deploy the HTTPBin application.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: httpbin
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: httpbin
         version: v1
     template:
       metadata:
         labels:
           app: httpbin
           version: v1
       spec:
         containers:
         - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/httpbin:0.1.0
           imagePullPolicy: IfNotPresent
           name: httpbin
           ports:
           - containerPort: 80

Confirm that the pod is running:

   kubectl get pod

Expected output:

   NAME                      READY   STATUS    RESTARTS   AGE
   httpbin-fd686xxxx         2/2     Running   0          2m16s

Step 2: Inspect the default buckets

Query the envoy_cluster_upstream_cx_connect_ms_bucket metric from the Envoy admin endpoint:

kubectl exec -it httpbin-fd686xxxx -c istio-proxy -- curl localhost:15000/stats/prometheus | grep envoy_cluster_upstream_cx_connect_ms_bucket

Expected output:

envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="0.5"} 10
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1"} 10
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="25"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="50"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="100"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="250"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="500"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="2500"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="30000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="60000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="300000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="600000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1800000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="3600000"} 11
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="+Inf"} 11

The le labels show the current bucket boundaries: [0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 300000, 600000, 1800000, 3600000].

Step 3: Apply custom buckets

Patch the HTTPBin deployment to replace the default xds-grpc buckets with [1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]:

kubectl patch deployment httpbin -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/statsHistogramBuckets":"{\"cluster.xds-grpc\":[1,5,10,25,50,100,250,500,1000,2500,5000,10000]}"}}}}}'

Patching the deployment triggers a rolling update. Wait for the new pod to reach Running state:

kubectl get pod

Expected output:

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-85b555xxxx-xxxxx   2/2     Running   0          2m2s

Step 4: Verify the new buckets

Query the metric again from the new pod:

kubectl exec -it httpbin-85b555xxxx-xxxxx -c istio-proxy -- curl localhost:15000/stats/prometheus | grep envoy_cluster_upstream_cx_connect_ms_bucket

Expected output:

envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1"} 0
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="25"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="50"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="100"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="250"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="500"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="1000"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="2500"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="5000"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="10000"} 1
envoy_cluster_upstream_cx_connect_ms_bucket{cluster_name="xds-grpc",le="+Inf"} 1

The output confirms the buckets are now [1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]. The sub-millisecond boundaries (0.5) and the large boundaries above 10000 have been removed.

References

Istio Standard Metrics -- Istio metric types and their dimensions.
Envoy Statistics -- Envoy-level statistics and how to enable additional stats collection.