Monitor ACK Pro control plane components using a self-managed Prometheus - Container Service for Kubernetes

Prerequisites

Before you begin, ensure that you have:

A self-managed Prometheus instance with access to the ACK Pro cluster API Server.
Read permission on the API Server /metrics path.
The Prometheus instance deployed inside or outside the cluster.

ACK Pro managed clusters expose metrics for the following control plane components:

Component	Metrics reference
`kube-apiserver`	Metrics for the kube-apiserver component
`etcd`	Metrics for the etcd component
`kube-scheduler`	Metrics of kube-scheduler
`kube-controller-manager`	Metrics for the kube-controller-manager component
`cloud-controller-manager`	Metrics of cloud-controller-manager

Step 1: Add Prometheus scrape jobs

Add a scrape job to your prometheus.yaml for each control plane component. The default global interval is 15s; each component job uses 30s to reduce API Server load.

The Prometheus configuration documentation covers full prometheus.yaml syntax.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: ack-api-server
    ......

  - job_name: ack-etcd
    ......

  - job_name: ack-scheduler
    ......

The following sections list the scrape configuration and alerting rules for each component. If you deploy Prometheus by using the Prometheus Operator pattern, see the ack-prometheus-operator add-on documentation in the ACK App Marketplace and the Prometheus Operator community documentation for custom scrape configuration.

Step 2: Configure in-cluster monitoring

If your self-managed Prometheus runs inside the cluster, route each scrape job through the API Server as a proxy.

Identify your cluster's networking architecture

Determine whether your cluster uses ENI (Elastic Network Interface) direct connection or CLB (Classic Load Balancer) forwarding:

kubectl get endpoints kubernetes

ENI direct connection: The output lists two or more IP addresses, such as 10.0.0.1:6443, 10.0.0.2:6443. Prometheus connects directly to each API Server replica.
CLB forwarding: The output shows a single IP address (the internal IP of the CLB). All traffic routes through the load balancer.

Use the correct endpoint based on this result when you configure each scrape job below.

kube-apiserver

The API Server is the entry point for all control plane metrics. Other component jobs route through it as a proxy.

Scrape configuration:

- job_name: ack-api-server
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [default]
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_component]
      action: keep
      regex: apiserver
    - source_labels: [__meta_kubernetes_service_label_provider]
      action: keep
      regex: kubernetes
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: https

Recommended alerting rules:

- alert: AckApiServerWarning
  expr: (absent(up{job="ack-api-server",pod!=""}) or (count(up{job="ack-api-server",pod!=""}) <= 1)) == 1
  for: 5m
  labels:
    severity: critical
  annotations:
    message: "APIServer is not available. Please check the Prometheus job and target status."

The pod!="" filter targets actual control plane pods, excluding the service-level endpoint. The <= 1 threshold triggers when one or zero API Server replicas are healthy, indicating degraded high availability. The for: 5m clause prevents false alarms from brief restarts.

etcd

Collects etcd metrics through the API Server proxy. The honor_labels: true setting preserves original component labels from the proxy.

Scrape configuration:

- job_name: ack-etcd
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  honor_labels: true
  params:
    hosting: ["true"]
    job: ["etcd"]
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [default]
  authorization:
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: false
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_component]
      action: keep
      regex: apiserver
    - source_labels: [__meta_kubernetes_service_label_provider]
      action: keep
      regex: kubernetes
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: https
    - source_labels: [__meta_kubernetes_service_label_component]
      action: replace
      target_label: job
      replacement: ${1}

Recommended alerting rules:

- alert: AckETCDLeaderMissing
  expr: sum_over_time(etcd_server_has_leader[5m]) == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    message: "Etcd cluster has no leader in the last 5 minutes. Check if the cluster is overloaded."

- alert: AckETCDDown
  expr: (absent(up{job="ack-etcd",pod!=""}) or (count(up{job="ack-etcd",pod!=""}) <= 2)) == 1
  for: 5m
  labels:
    severity: critical
  annotations:
    message: "Etcd is unavailable. Check the Prometheus job and target status."

AckETCDLeaderMissing fires when no etcd leader is elected over a 5-minute window, indicating a split-brain or overloaded cluster. AckETCDDown uses a threshold of <= 2 rather than <= 0 because etcd requires a quorum of at least three members — two or fewer cannot reach consensus.

kube-scheduler

Monitors scheduler health and scheduling latency.

Scrape configuration:

- job_name: ack-scheduler
  scrape_interval: 30s
  scheme: https
  params:
    hosting: ["true"]
    job: ["ack-scheduler"]
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [default]
  authorization:
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: false
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_component]
      action: keep
      regex: apiserver
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: https
    - source_labels: [__meta_kubernetes_service_label_component]
      action: replace
      target_label: job
      replacement: ${1}

Recommended alerting rules:

- alert: AckSchedulerWarning
  expr: (absent(up{job="ack-scheduler",pod!=""}) or (count(up{job="ack-scheduler",pod!=""}) <= 0)) == 1
  for: 3m
  labels:
    severity: critical
  annotations:
    message: "Scheduler is unavailable. Check the Prometheus job and target status."

The for: 3m duration is shorter than the API Server and etcd alerts because scheduler unavailability directly blocks new pod scheduling and requires faster detection.

kube-controller-manager (KCM)

Monitors controllers for core Kubernetes objects such as nodes, namespaces, and deployments.

Scrape configuration:

- job_name: ack-kcm
  scrape_interval: 30s
  scheme: https
  params:
    hosting: ["true"]
    job: ["ack-kube-controller-manager"]
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [default]
  authorization:
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_component]
      action: keep
      regex: apiserver
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: https
    - source_labels: [__meta_kubernetes_service_label_component]
      action: replace
      target_label: job
      replacement: ${1}

Recommended alerting rules:

- alert: AckKCMWarning
  expr: (absent(up{job="ack-kcm",pod!=""}) or (count(up{job="ack-kcm",pod!=""}) <= 0)) == 1
  for: 3m
  labels:
    severity: critical
  annotations:
    message: "KCM is unavailable. Check the Prometheus job and target status."

cloud-controller-manager (CCM)

Monitors the CCM, which integrates the cluster with Alibaba Cloud infrastructure. This job uses bearer_token_file for authentication instead of authorization.credentials_file used by other components.

Scrape configuration:

- job_name: ack-cloud-controller-manager
  scrape_interval: 30s
  scheme: https
  params:
    hosting: ["true"]
    job: ["ack-cloud-controller-manager"]
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names: [default]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_component]
      action: keep
      regex: apiserver
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      action: keep
      regex: https

Recommended alerting rules:

- alert: AckCCMWarning
  expr: (absent(up{job="ack-cloud-controller-manager",pod!=""}) or (count(up{job="ack-cloud-controller-manager",pod!=""}) <= 0)) == 1
  for: 3m
  labels:
    severity: critical
  annotations:
    message: "CCM is unavailable. Check the Prometheus job and target status."

The pod!="" filter ensures the alert targets only valid control plane pods, avoiding false alarms during brief network interruptions or rolling restarts.

Step 3: Configure Prometheus alerting rules

The Prometheus alerting rules documentation covers syntax and advanced options such as the for and keep_firing_for clauses.

Verify the setup

Confirm that Prometheus scrapes all components successfully. If Prometheus runs outside the target cluster, configure an out-of-cluster scrape job that uses kubernetes_sd_configs with the api_server endpoint, a bearer_token, and tls_config to authenticate against the cluster's API Server. For details, see the Prometheus community documentation.

Log on to your Prometheus console and go to the Graph page.
Run the up query. Verify that up{job="ack-api-server"} and all other component jobs return 1.
Run a component-specific query such as apiserver_request_total to confirm that time-series data populates correctly.