Container Service for Kubernetes:Collect control plane component metrics with a self-hosted Prometheus

How it works

In an ACK One registered cluster, control plane components, such as kube-scheduler and cloud-controller-manager, run on the Alibaba Cloud-managed infrastructure and cannot be accessed directly. To allow your self-hosted Prometheus to scrape metrics from these components, Container Service for Kubernetes (ACK) uses the following process:

Metric aggregation and exposure: The Metrics Aggregator component on the managed side collects and aggregates metrics from various control plane components. It then exposes a unified metrics interface through the API Server's internal endpoint (an internal-facing SLB).

Service discovery mechanism: A Headless Service is deployed in the cluster with its Endpoints pointing to the internal SLB IP address of the managed API Server. A ServiceMonitor, which is a CustomResourceDefinition (CRD) of the Prometheus Operator, is configured with service discovery rules that allow Prometheus to automatically discover and monitor this service.

Scrape process: Prometheus discovers the target service through the ServiceMonitor, parses its endpoints, then accesses the managed metric aggregation interface via the internal SLB to scrape the control plane component metrics.

This process requires no public endpoints, ensuring security and improving network performance.

Note: Integrate Managed Service for Prometheus with your cluster to get automated data monitoring and scraping, real-time Grafana dashboards, and configurable alerting. This service lets you create alerting rules and receive notifications through channels such as email, SMS, and DingTalk.

Prerequisites

• The version of the ack-stub component in your ACK One registered cluster must be v1.15.0.4 or later.

Step 1: Install the Metrics Aggregator component

Install the Metrics Aggregator component to aggregate metrics from control plane components.

1. Log on to the ACK console. In the navigation pane on the left, choose Clusters.

2. On the Clusters page, click the name of the target cluster. In the navigation pane on the left, click Add-ons.

3. On the Add-ons page, search for Metrics Aggregator. On the component card, click Install and follow the on-screen instructions to complete the installation.

Step 2: Prepare authentication credentials

1. Obtain the cluster KubeConfig and use kubectl to connect to the cluster.

   Note: Use a KubeConfig with the minimum required permissions to enhance security.

2. Extract the certificates from the KubeConfig. Prometheus uses these certificates to establish a secure TLS connection with the API Server.

   # 1. Extract the CA certificate (to verify the API Server)
   kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ca.crt
   # 2. Extract the client certificate (to identify the client to the API Server)
   kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d > client.crt
   # 3. Extract the client private key
   kubectl config view --raw -o jsonpath='{.users[0].user.client-key-data}' | base64 -d > client.key

3. Create a Secret to store the certificates and private key.

   kubectl create secret generic demo-metrics-tls \
     --namespace monitoring \
     --from-file=ca.crt=./ca.crt \
     --from-file=tls.crt=./client.crt \
     --from-file=tls.key=./client.key

Step 3: Create monitoring resources

1. On the Cluster Information page, click the Basic Information tab. Find the API Server Internal Endpoint. The metrics access address uses the format https://<YOUR_SLB_IP>:6443.

2. Create a Service for the API Server to allow Prometheus to discover the service and scrape its metrics.

   In the following YAML content, replace <YOUR_SLB_IP> with the IP address of the API Server internal endpoint from the previous step. Save the content as a file named ack-metrics-monitoring.yaml.

   # Create an Endpoints object that manually points to the internal IP address and port of the API Server.
   apiVersion: v1
   kind: Endpoints
   metadata:
     name: demo-metrics-service   # This name must exactly match the Service name below to establish a link.
     namespace: monitoring
   subsets:
   - addresses:
     - ip: <YOUR_SLB_IP>          # Replace with the internal IP address of the API Server endpoint.
     ports:
     - port: 6443
       name: https-metrics
       protocol: TCP
   ---
   # Create a Headless Service and bind it to the Endpoints object above to provide a stable service discovery target for the ServiceMonitor.
   apiVersion: v1
   kind: Service
   metadata:
     name: demo-metrics-service
     namespace: monitoring
     labels:
       app: demo-metrics          # This label must match the selector in the ServiceMonitor below.
   spec:
     clusterIP: None              # Define a Headless Service. It is not assigned a virtual IP and resolves directly to the IP in the Endpoints.
     ports:
     - name: https-metrics
       port: 6443
       targetPort: 6443
       protocol: TCP
   ---
   # Define a Prometheus scrape configuration.
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: demo-control-plane-metrics
     namespace: monitoring
     # This label must match your Prometheus instance's serviceMonitorSelector for automatic discovery.
     labels:
       app: prometheus-operator 
   spec:
     endpoints:
       - interval: 30s
         params:
           hosting:
             - 'true'
         path: /metrics
         port: https-metrics
         scheme: https
         # References the Secret created in Step 2.
         tlsConfig:
           ca:
             secret:
               key: ca.crt
               name: demo-metrics-tls
           cert:
             secret:
               key: tls.crt
               name: demo-metrics-tls
           insecureSkipVerify: false
           keySecret:
             key: tls.key
             name: demo-metrics-tls
     selector:
       matchLabels:
         app: demo-metrics
     namespaceSelector:
       matchNames:
         - monitoring

3. Apply the YAML file to create the resources.

   kubectl apply -f ack-metrics-monitoring.yaml

Verify the collection status

1. Verify that the resources were created successfully.

   # Check the Service, Endpoints, and Secret
   kubectl get service,endpoints,secret -n monitoring | grep demo-metrics
   # Check the ServiceMonitor
   kubectl get servicemonitor -n monitoring

2. Port-forward the self-hosted Prometheus service to your local machine.

   kubectl port-forward svc/ack-prometheus-operator-prometheus 9090 -n monitoring

3. Open http://localhost:9090 in your browser to access the Prometheus UI.

4. Navigate to the Status > Targets page. Find the target group for serviceMonitor/monitoring/demo-control-plane-metrics. If the State column shows UP, metric scraping is successful.