How to enable Managed Service for Prometheus in a registered cluster - Container Service for Kubernetes

You can use Managed Service for Prometheus to monitor the status of the registered cluster in real time based on the visualized monitoring dashboards and real-time alerts. This topic describes how to enable Managed Service for Prometheus for a registered cluster.

Prerequisites

An ACK One registered cluster is created and connected to a self-managed Kubernetes cluster in the on-premises data center. For more information, see Create a registered cluster.
A route is configured if the registered cluster is connected to a virtual private cloud (VPC) by using an Express Connect circuit. For more information, see Configure routes to route packets from registered clusters to cloud resources over the internal network. This ensures that the cluster can access the ACK component image addresses through the internal network and establish a network link for reporting monitoring data to Managed Service for Prometheus.

Step 1: Configure RAM permissions for the ack-arms-prometheus component

Use onectl

Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.

Run the following command to configure RAM permissions for the ack-arms-prometheus component:

onectl ram-user grant --addon arms-prometheus

Expected output:

Ram policy ack-one-registered-cluster-policy-arms-prometheus granted to ram user ack-one-user-ce313528c3 successfully.

Use the console

Create a RAM user.

Create a custom policy. The following policy includes the permissions required by ack-arms-prometheus:

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "arms:Describe*",
                "arms:List*",
                "arms:Get*",
                "arms:Search*",
                "arms:Check*",
                "arms:Query*",
                "arms:ListEnvironments",
                "arms:DescribeAddonRelease",
                "arms:InstallAddon",
                "arms:DeleteAddonRelease",
                "arms:ListEnvironmentDashboards",
                "arms:ListAddonReleases",
                "arms:CreateEnvironment",
                "arms:UpdateEnvironment",
                "arms:InitEnvironment",
                "arms:DescribeEnvironment",
                "arms:InstallEnvironmentFeature",
                "arms:ListEnvironmentFeatures"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Attach the custom policy to the RAM user.
Create an AccessKey pair for the RAM user.
Warning
We recommend that you configure AccessKey pair-based policies for network access control, limiting AccessKey invocation sources to trusted network environments to enhance AccessKey security.
Use the AccessKey pair to create a Secret named alibaba-addon-secret in the registered cluster.
Run the following command to create a Secret for ack-arms-prometheus:
```
kubectl -n arms-prom create secret generic alibaba-addon-secret --from-literal='access-key-id=<your AccessKey ID>' --from-literal='access-key-secret=<your AccessKey Secret>'
```
Note
Replace <your AccessKey ID> and <your AccessKey Secret> with the AccessKey pair that you created in the preceding step.

Step 2: Install the ack-arms-prometheus component

Use onectl

Run the following command to install the ack-arms-prometheus component:

onectl addon install arms-prometheus

Expected output:

Addon arms-prometheus, version **** installed.

Use the console

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of a cluster and choose Applications > Helm in the left-side navigation pane.
If the arms-prometheus and arms-prom Helm releases are displayed on the Helm page, delete the Helm releases. Then, perform the following steps to re-install arms-prometheus and arms-prom.
In the left-side navigation pane of the details page, choose Operations > Add-ons.
On the Add-ons page, click the Logs and Monitoring tab.
Find the ack-arms-prometheus component and click Install in the lower-right corner. Click OK.

After the component is installed, go to the Managed Service for Prometheus console. Click the Prometheus instance that is named after the registered cluster. On the details page of the Prometheus instance, you can then view monitoring data and create alert rules. For more information, see Use Managed Service for Prometheus and Create a Prometheus alert rule.

Configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor

By default, ack-arms-prometheus collects metrics from port 10255 on cAdvisor. If port 10250 on the kubelet in your registered cluster is open, you can perform the following steps to configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor (Container Advisor):

Create a ServiceMonitor in the registered cluster based on the following YAML template.

Show the YAML file content

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    arms.prometheus.io/discovery: 'true'
  name: arms-prom-cadvisor-10250
  namespace: arms-prom
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/cadvisor
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      honorLabels: true
      path: /metrics/probes
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          targetLabel: metrics_path
      scheme: https
      tlsConfig:
        caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecureSkipVerify: true
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet

Wait 15 seconds to 1 minute before you start metric collection.

Check whether metrics are collected.
1. Log on to the ARMS console.
2. In the left-side navigation pane, choose Managed Service for Prometheus > Instances.
3. In the top navigation bar of the Instances page, select the region where the cluster is deployed.
4. Click the name of the Prometheus instance that you want to view to go to the integration details page. In the left-side navigation pane, click Service Discovery. On the page that appears, click the Targets tab.
  Check whether a collection task named arms-prom/arms-prom-cadvisor-10250 exists and runs as expected.