How to collect metrics from a specific virtual node - Container Service for Kubernetes

Modify the Prometheus monitoring configuration to collect metrics from a specific virtual node.

Overview

In the virtual node architecture, multiple virtual nodes in the same cluster share the same node IP. As a result, collecting data from a single virtual node returns the data of all virtual nodes. Because Prometheus usually collects the metrics of all nodes through the kubelet Service, duplicate metrics occur. To resolve this issue, Container Service for Kubernetes (ACK) lets you collect metrics from a specific virtual node. It retains the original collection endpoint <nodeIP>:10250/metrics/cadvisor and adds an endpoint that specifies the virtual node name <nodeIP>:10250/metrics/cadvisor?nodeName=<nodeName>. After you specify a virtual node name, the virtual node returns metrics only for the pods that it manages.

Prerequisites

The ACK Virtual Node add-on is installed and is version v2.11.0 or later. For more information, see ACK Virtual Node.

Modify the Prometheus monitoring configuration

You can modify the Prometheus monitoring configuration to collect metrics from a specific virtual node. Container Service for Kubernetes supports three configuration scenarios: Alibaba Cloud Managed Service for Prometheus, the community edition Prometheus Operator solution (or the Container Service for Kubernetes App Marketplace ack-prometheus-operator), and open source Prometheus.

Alibaba Cloud Managed Service for Prometheus

Supported by default. No additional action is required.

Community edition Prometheus Operator

If you use the community edition Prometheus Operator solution or the Container Service for Kubernetes App Marketplace ack-prometheus-operator, add the following ServiceMonitor CR configuration.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: virtual-kubelet
  namespace: monitoring
  labels:
    k8s-app: kubelet
    # Add this label for automatic management by prometheus-operator.
    release: prometheus-operator
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    interval: 15s
    scheme: https
    path: /metrics/cadvisor
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
    relabelings:
    # Keep only the endpoints of virtual nodes.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      action: keep
    # Add the query parameter that specifies nodeName.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      targetLabel: __param_nodeName
      replacement: ${1}
      action: replace

If your cluster already uses service discovery based on the kubelet Service to collect cAdvisor metrics, add the following configuration to remove the collection of <Virtual Node IP>:10250/metrics/cadvisor and avoid collecting duplicate data.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
  - path: /metrics/cadvisor
    port: https-metrics
    ...
    relabelings:
    # This relabeling rule drops all endpoints whose target name starts with virtual-kubelet.
    - action: drop
      regex: (^virtual-kubelet.*)
      sourceLabels:
      - __meta_kubernetes_endpoint_address_target_name

Open source Prometheus

In open source Prometheus, find the Prometheus configuration file, which is usually located in /etc/prometheus/prometheus.yml or under your custom configuration directory, and then add the following scrape configuration.

scrape_configs:

...Other job configurations.

- job_name: monitoring/virtual-kubelet/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    target_label: __param_nodeName
    replacement: ${1}
    action: replace
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system

scrape_configs:

...Other job configurations.

- job_name: monitoring/ack-prometheus-operator-kubelet/0
  honor_labels: true
  honor_timestamps: true
  ...
  relabel_configs:
  ...
  # Remove the collection of the /metrics/cadvisor endpoint for virtual nodes.
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    replacement: $1
    action: drop