Container Service for Kubernetes:Collect the metrics of the specified virtual node

Alibaba Cloud Managed Service for Prometheus

Default support is provided. No additional action is required.

Community Edition Prometheus Operator

When using the Community Edition Prometheus Operator solution in conjunction with the Container Service for Kubernetes application marketplace ack-prometheus-operator, you must include the following ServiceMonitor CR configuration.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: virtual-kubelet
  namespace: monitoring
  labels:
    k8s-app: kubelet
    # Add this label to automatically manage prometheus-operator.
    release: prometheus-operator
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: kubelet
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    interval: 15s
    scheme: https
    path: /metrics/cadvisor
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    tlsConfig:
      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecureSkipVerify: true
    relabelings:
    # Retain only the virtual node endpoint.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      action: keep
    # Add parameters to query based on the specified nodeName.
    - sourceLabels: [__meta_kubernetes_endpoint_address_target_name]
      regex: (^virtual-kubelet.*)
      targetLabel: __param_nodeName
      replacement: ${1}
      action: replace

If the cluster is already using service discovery based on the kubelet service to collect cAdvisor metrics, you must add the following configuration to exclude the collection configuration for <Virtual Node IP>:10250/metrics/cadvisor to prevent duplicate data collection.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
...
spec:
  endpoints:
  - path: /metrics/cadvisor
    port: https-metrics
    ...
    relabelings:
    # The relabeling rule discards the endpoints of all targets whose names start with virtual-kubelet.
    - action: drop
      regex: (^virtual-kubelet.*)
      sourceLabels:
      - __meta_kubernetes_endpoint_address_target_name

Open-source Prometheus

For open-source Prometheus, locate the Prometheus configuration file, typically found at /etc/prometheus/prometheus.yml or within your custom configuration folder, and add the following collection configuration.

scrape_configs:

...Other job configuration.

- job_name: monitoring/virtual-kubelet/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    target_label: __param_nodeName
    replacement: ${1}
    action: replace
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system

If the cluster is already using service discovery based on the kubelet service to collect cAdvisor metrics, you must add the following configuration to exclude the collection configuration for <Virtual Node IP>:10250/metrics/cadvisor to prevent duplicate data collection.

scrape_configs:

...Other job configuration.

- job_name: monitoring/ack-prometheus-operator-kubelet/0
  honor_labels: true
  honor_timestamps: true
  ...
  relabel_configs:
  ...
  // Discard the endpoint for collecting the /metrics/cadviso metrics of virtual nodes.
  - source_labels: [__meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: (^virtual-kubelet.*)
    replacement: $1
    action: drop