How to connect Prometheus Monitoring to a registered cluster - Container Service for Kubernetes

You can use Prometheus Monitoring to monitor the status of registered clusters in real time, view visualized monitoring dashboards, and receive alerts in real time. This topic describes how to integrate Prometheus Monitoring with registered clusters.

Prerequisites

An ACK One registered cluster is connected to a self-managed Kubernetes cluster in your data center. For more information, see Create an ACK One registered cluster.
For a registered cluster that is connected to a cloud VPC by using a leased line, see Configure a route for a registered cluster to access cloud products over an internal network to complete the configuration. This ensures that the cluster can access the image addresses of ACK add-ons over the internal network and establishes the network path for reporting monitoring data to Prometheus Monitoring.

Step 1: Grant RAM permissions to the ack-arms-prometheus add-on

onectl

Install and configure onectl on your local machine. For more information, see Manage registered clusters using onectl.

Run the following command to grant RAM permissions to the ack-arms-prometheus add-on.

onectl ram-user grant --addon arms-prometheus

Expected output:

Ram policy ack-one-registered-cluster-policy-arms-prometheus granted to ram user ack-one-user-ce313528c3 successfully.

Console

Create a RAM user.

Create a custom policy. The ack-arms-prometheus add-on requires the following policy content:

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "arms:Describe*",
                "arms:List*",
                "arms:Get*",
                "arms:Search*",
                "arms:Check*",
                "arms:Query*",
                "arms:ListEnvironments",
                "arms:DescribeAddonRelease",
                "arms:InstallAddon",
                "arms:DeleteAddonRelease",
                "arms:ListEnvironmentDashboards",
                "arms:ListAddonReleases",
                "arms:CreateEnvironment",
                "arms:UpdateEnvironment",
                "arms:InitEnvironment",
                "arms:DescribeEnvironment",
                "arms:InstallEnvironmentFeature",
                "arms:ListEnvironmentFeatures",
                "cms:GetIntegrationVersionForCS",
                "cms:CreateIntegrationPolicy",
                "cms:ListAddonReleases",
                "cms:UpdateAddonRelease",
                "cms:CreateAddonRelease",
                "cms:GetPrometheusInstance",
                "cms:ListIntegrationPolicyStorageRequirements",
                "log:PostLogStoreLogs"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Grant permissions to the RAM user.
Create an AccessKey for the RAM user.

Warning
For enhanced security, configure a network access control policy for the AccessKey to restrict access to trusted network environments and improve security. For more information, see AccessKey-based network access restriction policies.
Use the AccessKey to create a Secret resource named alibaba-addon-secret in the registered cluster.
Run the following commands to create the Secret.
```
kubectl create namespace arms-prom
kubectl -n arms-prom create secret generic alibaba-addon-secret --from-literal='access-key-id=<your AccessKey ID>' --from-literal='access-key-secret=<your AccessKey Secret>'
```
Note
Replace <your AccessKey ID> and <your AccessKey Secret> with your AccessKey ID and AccessKey secret.

Step 2: Install the ack-arms-prometheus add-on

onectl

Run the following command to install the ack-arms-prometheus add-on.

onectl addon install arms-prometheus

Expected output:

Addon arms-prometheus, version **** installed.

Console

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your target cluster. On the left-side navigation pane, choose Applications > Helm.
If the Helm page contains installation records for arms-prometheus and arms-prom, delete them before you proceed.
In the left navigation pane of the cluster management page, choose O&M Management > Component Management.
On the Add-ons page, click the Logs and Monitoring tab.
Find the ack-arms-prometheus card and click Install in the lower-right corner. Then, click OK.

After the installation is complete, log on to the Prometheus Monitoring console to view monitoring data for the cluster instance and define alert rules. For more information, see Connect and configure Prometheus Monitoring and Create a Prometheus alert rule.

Switch to CAdvisor port 10250

By default, the ack-arms-prometheus add-on collects data from CAdvisor port 10255. If only kubelet port 10250 is open in your registered cluster, follow these steps to switch the data collection port.

Use the following YAML content to deploy and configure a ServiceMonitor in the registered cluster.

Click to view the YAML content

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    arms.prometheus.io/discovery: 'true'
  name: arms-prom-cadvisor-10250
  namespace: arms-prom
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

Wait for 15 to 60 seconds, and then verify the collected data.

Verify the collected data after you switch the port.
1. Log on to the ARMS console.
2. In the left navigation pane, choose Managed Service for Prometheus > Instances.
3. On the top navigation bar of the Prometheus Monitoring page, select the region where the cluster is deployed.
4. On the Instance page, click the name of the target instance to open the Integration Center page. On the left-side navigation pane, click Indicator Collection and then click the Self-Monitoring tab.
  Verify that the arms-prom/arms-prom-cadvisor-10250 collection job exists and its status is Normal.