Container Service for Kubernetes:Collect control plane component metrics with a self-hosted Prometheus

How it works

The Metrics Aggregator component collects and aggregates metrics from control plane components on the managed side. It exposes a unified metrics interface through the API Server's internal endpoint (an internal-facing SLB).

A Headless Service is deployed in the cluster with its Endpoints pointing to the internal SLB IP address of the managed API Server. A ServiceMonitor — a CustomResourceDefinition (CRD) of the Prometheus Operator — is configured with service discovery rules so Prometheus automatically discovers and monitors this service.

When Prometheus scrapes metrics, it discovers the target through the ServiceMonitor, resolves the Headless Service's Endpoints, and accesses the metric aggregation interface via the internal SLB. No public endpoints are required.

Prerequisites

Before you begin, ensure that you have:

• An ACK One registered cluster with the ack-stub component at v1.15.0.4 or later

Step 1: Install the Metrics Aggregator component

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.

3. On the Add-ons page, search for Metrics Aggregator. On the component card, click Install and follow the on-screen instructions.

Step 2: Prepare authentication credentials

Prometheus uses TLS mutual authentication to connect to the API Server. Extract the required certificates from your kubeconfig and store them in a Kubernetes Secret.

1. Get the cluster kubeconfig and connect using kubectl.

   Use a kubeconfig with the minimum required permissions to reduce security exposure.

2. Extract the CA certificate, client certificate, and client private key from the kubeconfig.

   # 1. Extract the CA certificate (to verify the API Server)
   kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
   # 2. Extract the client certificate (to identify the client to the API Server)
   kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
   # 3. Extract the client private key
   kubectl config view --raw -o jsonpath='{.users[0].user.client-key-data}' | base64 -d > client.key

3. Create a Secret in the monitoring namespace to store the certificates.

   kubectl create secret generic demo-metrics-tls \
     --namespace monitoring \
     --from-file=ca.crt=./ca.crt \
     --from-file=tls.crt=./client.crt \
     --from-file=tls.key=./client.key

Step 3: Create monitoring resources

1. Find the API Server internal endpoint. On the Cluster Information page, click the Basic Information tab and locate API Server Internal Endpoint. The metrics access address follows the format https://<YOUR_SLB_IP>:6443.

2. Save the following YAML content as ack-metrics-monitoring.yaml. Replace <YOUR_SLB_IP> with the IP address from the previous step.

   # Create an Endpoints object that manually points to the internal IP address and port of the API Server.
   apiVersion: v1
   kind: Endpoints
   metadata:
     name: demo-metrics-service   # This name must exactly match the Service name below to establish a link.
     namespace: monitoring
   subsets:
   - addresses:
     - ip: <YOUR_SLB_IP>          # Replace with the internal IP address of the API Server endpoint.
     ports:
     - port: 6443
       name: https-metrics
       protocol: TCP
   ---
   # Create a Headless Service and bind it to the Endpoints object above to provide a stable service discovery target for the ServiceMonitor.
   apiVersion: v1
   kind: Service
   metadata:
     name: demo-metrics-service
     namespace: monitoring
     labels:
       app: demo-metrics          # This label must match the selector in the ServiceMonitor below.
   spec:
     clusterIP: None              # Define a Headless Service. It is not assigned a virtual IP and resolves directly to the IP in the Endpoints.
     ports:
     - name: https-metrics
       port: 6443
       targetPort: 6443
       protocol: TCP
   ---
   # Define a Prometheus scrape configuration.
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: demo-control-plane-metrics
     namespace: monitoring
     # This label must match your Prometheus instance's serviceMonitorSelector for automatic discovery.
     labels:
       app: prometheus-operator
   spec:
     endpoints:
       - interval: 30s
         params:
           hosting:
             - 'true'
         path: /metrics
         port: https-metrics
         scheme: https
         # References the Secret created in Step 2.
         tlsConfig:
           ca:
             secret:
               key: ca.crt
               name: demo-metrics-tls
           cert:
             secret:
               key: tls.crt
               name: demo-metrics-tls
           insecureSkipVerify: false
           keySecret:
             key: tls.key
             name: demo-metrics-tls
     selector:
       matchLabels:
         app: demo-metrics
     namespaceSelector:
       matchNames:
         - monitoring

3. Apply the file to create the resources.

   kubectl apply -f ack-metrics-monitoring.yaml

Verify the collection status

1. Confirm that all resources were created successfully.

   # Check the Service, Endpoints, and Secret
   kubectl get service,endpoints,secret -n monitoring | grep demo-metrics
   # Check the ServiceMonitor
   kubectl get servicemonitor -n monitoring

2. Port-forward the Prometheus service to your local machine.

   kubectl port-forward svc/ack-prometheus-operator-prometheus 9090 -n monitoring

3. Open http://localhost:9090 in your browser.

4. Go to Status > Targets. Find the target group serviceMonitor/monitoring/demo-control-plane-metrics. If the State column shows UP, metric scraping is working correctly.