Container Service for Kubernetes:Collect metrics from a specific virtual node

Alibaba Cloud Managed Service for Prometheus

No configuration changes are required. The node-scoped endpoint is supported by default.

Community edition Prometheus Operator

If you use the community edition Prometheus Operator with ack-prometheus-operator from App Marketplace, add the following ServiceMonitor custom resource (CR).

This configuration keeps only endpoints whose target name starts with virtual-kubelet, and adds nodeName as a query parameter so each virtual node is scraped individually.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: virtual-kubelet
  namespace: monitoring
  labels:
    k8s-app: kubelet
    # Required label for automatic management by Prometheus Operator.
    release: prometheus-operator
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    interval: 15s
    scheme: https
    path: /metrics/cadvisor
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
    relabelings:
    # Keep only virtual node endpoints (names starting with "virtual-kubelet").
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      action: keep
    # Set the nodeName query parameter to the virtual node's name.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      targetLabel: __param_nodeName
      replacement: ${1}
      action: replace

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
  - path: /metrics/cadvisor
    port: https-metrics
    ...
    relabelings:
    # Drop endpoints whose target name starts with "virtual-kubelet".
    # This prevents duplicate collection from <virtual node IP>:10250/metrics/cadvisor.
    - action: drop
      regex: (^virtual-kubelet.*)
      sourceLabels:
      - __meta_kubernetes_endpoint_address_target_name

Open-source Prometheus

Locate your Prometheus configuration file (typically /etc/prometheus/prometheus.yml or under your custom configuration directory) and add the following scrape job.

The relabeling rules fall into two groups. The first group is the standard kubelet scrape configuration. The last two rules are virtual-node-specific and keep only virtual node endpoints. They also inject the nodeName query parameter.

scrape_configs:

  # ... other job configurations ...

  - job_name: monitoring/virtual-kubelet/0
    honor_timestamps: true
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics/cadvisor
    scheme: https
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    relabel_configs:
    # --- Standard kubelet service discovery rules ---
    - source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: kubelet
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: https-metrics
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: service
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      target_label: pod
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_pod_container_name]
      separator: ;
      regex: (.*)
      target_label: container
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: (.*)
      target_label: job
      replacement: ${1}
      action: replace
    - source_labels: [__meta_kubernetes_service_label_k8s_app]
      separator: ;
      regex: (.+)
      target_label: job
      replacement: ${1}
      action: replace
    - separator: ;
      regex: (.*)
      target_label: endpoint
      replacement: https-metrics
      action: replace
    # --- Virtual-node-specific rules ---
    # Keep only endpoints whose target name starts with "virtual-kubelet".
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      replacement: $1
      action: keep
    # Set the nodeName query parameter to the virtual node's name.
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      target_label: __param_nodeName
      replacement: ${1}
      action: replace
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - kube-system

scrape_configs:

  # ... other job configurations ...

  - job_name: monitoring/ack-prometheus-operator-kubelet/0
    honor_labels: true
    honor_timestamps: true
    ...
    relabel_configs:
    ...
    # Drop virtual node endpoints to prevent duplicate collection of /metrics/cadvisor.
    - source_labels: [__meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: (^virtual-kubelet.*)
      replacement: $1
      action: drop