How to connect a registered cluster to Prometheus - Container Service for Kubernetes

You can use Prometheus to monitor the status of a registered cluster in real time, view data on dashboards, and receive real-time alerts. This topic describes how to connect Prometheus to a registered cluster.

Prerequisites

An ACK One registered cluster has been created, and a self-managed Kubernetes cluster from your data center is connected to the ACK One registered cluster. For more information, see Create an ACK One registered cluster.
If your registered cluster connects to an Alibaba Cloud Virtual Private Cloud (VPC) through a leased line, complete the configuration by following the instructions in Configure routes for a registered cluster to access cloud products over an internal network. This ensures that the cluster can access the Registry Address of ACK components and that Prometheus can report monitoring data over the internal network.

Step 1: Grant Resource Access Management (RAM) permissions to the ack-arms-prometheus component

Configure using onectl

Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.

Run the following command to grant RAM permissions to the ack-arms-prometheus component.

onectl ram-user grant --addon arms-prometheus

Expected output:

Ram policy ack-one-registered-cluster-policy-arms-prometheus granted to ram user ack-one-user-ce313528c3 successfully.

Configure using the console

Create a RAM user.

Create a custom policy. The policy document for the ack-arms-prometheus component is as follows.

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "arms:Describe*",
                "arms:List*",
                "arms:Get*",
                "arms:Search*",
                "arms:Check*",
                "arms:Query*",
                "arms:ListEnvironments",
                "arms:DescribeAddonRelease",
                "arms:InstallAddon",
                "arms:DeleteAddonRelease",
                "arms:ListEnvironmentDashboards",
                "arms:ListAddonReleases",
                "arms:CreateEnvironment",
                "arms:UpdateEnvironment",
                "arms:InitEnvironment",
                "arms:DescribeEnvironment",
                "arms:InstallEnvironmentFeature",
                "arms:ListEnvironmentFeatures"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Grant permissions to the RAM user.
Create an AccessKey pair for the RAM user.
Warning
We recommend that you configure AccessKey pair-based policies for network access control, limiting AccessKey invocation sources to trusted network environments to enhance AccessKey security.
Use an AccessKey to create a Secret resource named alibaba-addon-secret in the registered cluster.
Run the following command to create the Secret for the ack-arms-prometheus component.
```
kubectl create namespace arms-prom
kubectl -n arms-prom create secret generic alibaba-addon-secret --from-literal='access-key-id=<your AccessKey ID>' --from-literal='access-key-secret=<your AccessKey Secret>'
```
Note
Replace <your AccessKey ID> and <your AccessKey Secret> with your AccessKey ID and AccessKey secret.

Step 2: Install the ack-arms-prometheus component

Install using onectl

Run the following command to install the ack-arms-prometheus component.

onectl addon install arms-prometheus

Expected output:

Addon arms-prometheus, version **** installed.

Install using the console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the navigation pane on the left, choose Applications > Helm.
If the Helm page contains installation records for arms-prometheus and arms-prom, delete them before you proceed.
In the left-side navigation pane of the details page, click Add-ons.
On the Add-ons page, click the Logs and Monitoring tab.
Find the ack-arms-prometheus card and click Install in the lower-right corner. Then, click OK.

After the installation is complete, log on to the Prometheus console. In the console, navigate to the cluster instance to view monitoring data and define alert rules. For more information, see Connect a data source to Prometheus Monitoring and Create a Prometheus alert rule.

Switch the ack-arms-prometheus component to collect data from CAdvisor port 10250

By default, the ack-arms-prometheus component collects data from CAdvisor port 10255. If only kubelet port 10250 is open in your registered cluster, follow these steps to switch the data collection port for the component.

Use the following YAML content to deploy and configure a ServiceMonitor in the registered cluster.

Click to view the YAML content

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    arms.prometheus.io/discovery: 'true'
  name: arms-prom-cadvisor-10250
  namespace: arms-prom
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

Wait for about 15 seconds to 1 minute, and then verify the collected data.

Verify the collected data after you switch the port.
1. Log on to the ARMS console.
2. In the left navigation pane, choose Managed Service for Prometheus > Instances.
3. On the Prometheus Monitoring page, select the region where the cluster is located from the top menu bar.
4. On the Instances page, click the name of the target instance to go to the Integration Center page. In the navigation pane on the left, click Indicator Collection, and then click the Self-Monitoring tab.
  Verify that the arms-prom/arms-prom-cadvisor-10250 collection job exists and that its status is Normal.