Collect metrics from a specific virtual node

When multiple virtual nodes share a single node IP address, Prometheus collects cAdvisor metrics from that IP and returns data for all virtual nodes at once—causing metric duplication. To avoid this, ACK provides a node-scoped endpoint that lets Prometheus target one virtual node by name.

How it works

ACK exposes two cAdvisor endpoints on the kubelet:

Endpoint	Returns
`<nodeIP>:10250/metrics/cadvisor`	Metrics for all virtual nodes on that IP
`<nodeIP>:10250/metrics/cadvisor?nodeName=<nodeName>`	Metrics for pods managed by the specified virtual node only

Adding the nodeName query parameter scopes the response to a single virtual node, eliminating duplicate data when multiple virtual nodes share the same IP.

Configure Prometheus

ACK supports three Prometheus setups. Choose the one that matches your deployment.

Alibaba Cloud Managed Service for Prometheus

No configuration changes are required. The node-scoped endpoint is supported by default.

Community Edition Prometheus Operator

If you use the Community Edition Prometheus Operator with ack-prometheus-operator from the application marketplace, add the following ServiceMonitor custom resource (CR).

This configuration does two things: it keeps only endpoints whose target name starts with virtual-kubelet, and adds nodeName as a query parameter so each virtual node is scraped individually.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: virtual-kubelet
  namespace: monitoring
  labels:
    k8s-app: kubelet
    # Required label for automatic management by Prometheus Operator.
    release: prometheus-operator
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    interval: 15s
    scheme: https
    path: /metrics/cadvisor
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
    relabelings:
    # Keep only virtual node endpoints (names starting with "virtual-kubelet").
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      action: keep
    # Set the nodeName query parameter to the virtual node's name.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      targetLabel: __param_nodeName
      replacement: ${1}
      action: replace

Important

If the cluster already uses kubelet-based service discovery to collect cAdvisor metrics, the virtual node IP is also scraped by the existing configuration, which causes duplicate data. Add the following drop rule to the existing kubelet ServiceMonitor to exclude virtual node endpoints from that job.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
  - path: /metrics/cadvisor
    port: https-metrics
    ...
    relabelings:
    # Drop endpoints whose target name starts with "virtual-kubelet".
    # This prevents duplicate collection from <virtual node IP>:10250/metrics/cadvisor.
    - action: drop
      regex: (^virtual-kubelet.*)
      sourceLabels:
      - __meta_kubernetes_endpoint_address_target_name

Open-source Prometheus

Locate your Prometheus configuration file (typically /etc/prometheus/prometheus.yml or your custom configuration folder) and add the following scrape job.

The relabeling rules are divided into two groups: the first group is standard kubelet scrape configuration, and the last two rules are virtual-node-specific—they filter to virtual node endpoints only and inject the nodeName query parameter.

scrape_configs:

  # ... other job configurations ...

  - job_name: monitoring/virtual-kubelet/0
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics/cadvisor
    scheme: https
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    relabel_configs:
    # --- Standard kubelet service discovery rules ---
    - source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: kubelet
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: https-metrics
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: service
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      target_label: pod
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_container_name]
      separator: ;
      regex: (.*)
      target_label: container
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: job
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: (.+)
      target_label: job
      replacement: ${1}
      action: replace
    - separator: ;
      regex: (.*)
      target_label: endpoint
      replacement: https-metrics
      action: replace
    # --- Virtual-node-specific rules ---
    # Keep only endpoints whose target name starts with "virtual-kubelet".
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      replacement: $1
      action: keep
    # Set the nodeName query parameter to the virtual node's name.
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      target_label: __param_nodeName
      replacement: ${1}
      action: replace
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - kube-system

Important

If the cluster already uses kubelet-based service discovery to collect cAdvisor metrics, add the following drop rule to the existing scrape job (job name: monitoring/ack-prometheus-operator-kubelet/0). This prevents Prometheus from also scraping <virtual node IP>:10250/metrics/cadvisor and producing duplicate metrics.

scrape_configs:

  # ... other job configurations ...

  - job_name: monitoring/ack-prometheus-operator-kubelet/0
    honor_labels: true
    honor_timestamps: true
    ...
    relabel_configs:
    ...
    # Drop virtual node endpoints to prevent duplicate collection of /metrics/cadvisor.
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      replacement: $1
      action: drop