Merge Istio and application monitoring metrics - Alibaba Cloud Service Mesh

Standard Prometheus configurations use prometheus.io annotations to scrape one metrics endpoint per pod. When Service Mesh (ASM) injects a sidecar proxy, the proxy exposes Istio metrics on a separate endpoint, which conflicts with the application's existing scrape target. Metrics merging solves this by combining both sets of metrics into a single endpoint that Prometheus scrapes without additional configuration.

After you enable metrics merging, each sidecar proxy fetches your application's metrics (using the original prometheus.io annotations), combines them with Istio metrics, and exposes everything at :15020/stats/prometheus. ASM updates the pod's prometheus.io annotations to point to this merged endpoint, so Prometheus picks up both application and Istio metrics automatically.

If metrics merging does not fit your environment, configure Prometheus to scrape the application and sidecar endpoints separately.

When not to use metrics merging

Important

Evaluate the following scenario before you enable this feature. If it applies, configure Prometheus to scrape application and sidecar endpoints separately instead.

Sensitive application metrics. Merged metrics are exposed through the sidecar proxy. Any user with permission to view Istio metrics can also view your application metrics.

How metrics merging works

The following diagram shows the data flow after metrics merging is enabled:

+---------------------------------------------------------+
|  Pod                                                    |
|                                                         |
|  +----------------+       +--------------------------+  |
|  |  Application   |       |  Sidecar proxy           |  |
|  |                |       |                          |  |
|  |  :9080         |------>|  Fetches app metrics     |  |
|  |  /metrics      |       |  from original endpoint  |  |
|  +----------------+       |                          |  |
|                           |  Merges with Istio       |  |
|                           |  proxy metrics           |  |
|                           |                          |  |
|                           |  :15020                  |  |
|                           |  /stats/prometheus       |------> Prometheus
|                           +--------------------------+  |
+---------------------------------------------------------+

When ASM enables metrics merging on a pod:

The original prometheus.io annotations (scrape target, port, path) are saved to the ISTIO_PROMETHEUS_ANNOTATIONS environment variable in the sidecar container.
The prometheus.io annotations are added to all pods on the data plane and overwritten to point to the merged endpoint (:15020/stats/prometheus). If these annotations already exist, they will be overwritten.
The sidecar proxy reads the saved annotations, fetches application metrics from the original endpoint, combines them with its own Istio metrics, and serves everything at the merged endpoint.

Before you begin

Before you begin, make sure that you have:

An ASM instance that runs version 1.17.2.30 or later. To create or upgrade an instance, see Create an ASM instance or Update an ASM instance
An application deployed with standard prometheus.io annotations. For deployment instructions, see Deploy an application in an ASM instance

Step 1: Verify your application's Prometheus annotations

Before you enable metrics merging, confirm that your application pods include prometheus.io annotations. The following Deployment manifest shows a typical configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: productpage-v1
  labels:
    app: productpage
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: productpage
      version: v1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9080"
        prometheus.io/path: "/metrics"
      labels:
        app: productpage
        version: v1
    spec:
      ......

Run kubectl get pod <pod-name> -o yaml and check that the following annotations are present:

prometheus.io/path: /metrics
prometheus.io/port: '9080'
prometheus.io/scrape: 'true'

Step 2: Enable metrics merging

Metrics merging is disabled by default. Enable it at the ASM instance level (applies to all workloads) or per workload.

Option A: Enable globally for an ASM instance

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
In the Config Info section, click Enable next to Merge Istio's Metrics with Application Metrics. In the confirmation dialog, click OK.

Option B: Enable for a specific workload

Add the following annotation to the pod spec of the target workload:

prometheus.istio.io/merge-metrics: "true"

Use this approach to enable merging on specific services while keeping it disabled globally, or to override the instance-level setting on individual workloads.

Step 3: Verify the merged endpoint

After you enable metrics merging, restart the affected pods so the updated sidecar configuration takes effect. Then inspect the pod to confirm the changes.

Run kubectl get pod <pod-name> -o yaml and verify the following:

Updated annotations. The prometheus.io annotations now point to the merged endpoint:

prometheus.io/path: /stats/prometheus
prometheus.io/port: '15020'
prometheus.io/scrape: 'true'

New environment variable. The sidecar container includes an ISTIO_PROMETHEUS_ANNOTATIONS environment variable that preserves the original application scrape configuration:

env:
  - name: ISTIO_PROMETHEUS_ANNOTATIONS
    value: '{"scrape":"true","path":"/metrics","port":"9080"}'

Disable metrics merging

To disable metrics merging, choose one of the following methods based on how you enabled it:

Instance level: In the ASM console, go to ASM Instance > Base Information. In the Config Info section, click Disable next to Merge Istio's Metrics with Application Metrics. In the confirmation dialog, click OK.
Workload level: Add the following annotation to the target pod spec:
```
prometheus.istio.io/merge-metrics: "false"
```

Note

The per-workload annotation also works as an override. If metrics merging is enabled at the instance level, add this annotation to exclude specific workloads that have sensitive metrics.

Alibaba Cloud Service Mesh:Merge Istio metrics with application metrics