All Products
Search
Document Center

Container Service for Kubernetes:Enable Managed Service for Prometheus for a registered cluster

Last Updated:Nov 10, 2023

You can use Managed Service for Prometheus to centrally manage registered Kubernetes clusters that are deployed in different geolocations. This topic describes how to enable Managed Service for Prometheus for a registered cluster.

Prerequisites

Step 1: Configure RAM permissions for the ack-arms-prometheus component

Use onectl

  1. Install onectl on your on-premises machine. For more information, see Use onectl to manage registered clusters.

  2. Run the following command to configure RAM permissions for the ack-arms-prometheus component:

    onectl ram-user grant --addon arms-prometheus

    Expected output:

    Ram policy ack-one-registered-cluster-policy-arms-prometheus granted to ram user ack-one-user-ce313528c3 successfully.

Use the console

If the external Kubernetes cluster is registered to ACK over the Internet, you must specify an AccessKey pair in the registered cluster before you install ack-arms-prometheus. This way, ack-arms-prometheus can use the AccessKey pair to access Alibaba Cloud services. If the external Kubernetes cluster is registered to ACK over the internal network, you do not need to specify an AccessKey pair in the registered cluster.

Run the following command to check whether the external Kubernetes cluster is registered to ACK over the internal network:

kubectl -n kube-system get deploy ack-cluster-agent -o=jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="INTERNAL_ENDPOINT")].value}'
  • If true is included in the output, the external Kubernetes cluster is registered to ACK over the internal network.

  • If false is included in the output, the external Kubernetes cluster is registered to ACK over the Internet. In this case, perform the following steps to specify an AccessKey pair in the registered cluster:

Specify an AccessKey pair in the registered cluster

  1. Create a RAM user. For more information, see Create a RAM user.

  2. Create a custom policy. For more information, see Create a custom policy. The following policy content includes the permissions that are required by ack-arms-prometheus:

    {
        "Version": "1",
        "Statement": [
            {
                "Action": [
                    "arms:Describe*",
                    "arms:List*",
                    "arms:Get*",
                    "arms:Search*",
                    "arms:Check*",
                    "arms:Query*"
                ],
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
    }
  3. Attach the custom policy to the RAM user. For more information, see Grant permissions to RAM users.

  4. Create an AccessKey pair for the RAM user. For more information, see Create an AccessKey pair.

  5. Use the AccessKey pair to create a Secret named alibaba-addon-secret in the registered cluster.

    Run the following command to create a Secret for Logtail:

    kubectl -n arms-prom create secret generic alibaba-addon-secret --from-literal='access-key-id=<your AccessKey ID>' --from-literal='access-key-secret=<your AccessKey Secret>'
    Note

    Replace <your AccessKey ID> and <your AccessKey Secret> with the AccessKey pair that you created in the preceding step.

Step 2: Install the ack-arms-prometheus component

Use onectl

Run the following command to install the ack-arms-prometheus component:

onectl addon install arms-prometheus

Expected output:

Addon arms-prometheus, version **** installed.

Use the console

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of a cluster and choose Applications > Helm in the left-side navigation pane.

    If the arms-prometheus and arms-prom Helm releases are displayed on the Helm page, delete the Helm releases. Then, perform the following steps to re-install arms-prometheus and arms-prom.

  3. In the left-side navigation pane of the details page, choose Operations > Add-ons.

  4. On the Add-ons page, click the Logs and Monitoring tab.

  5. Find the ack-arms-prometheus component and click Install in the lower-right corner. Click OK.

After the component is installed, go to the Managed Service for Prometheus console. Click the Prometheus instance that is named after the registered cluster. On the details page of the Prometheus instance, you can then view monitoring data and create alert rules. For more information, see Managed Service for Prometheus and Create a Prometheus alert rule.

Configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor

By default, ack-arms-prometheus collects metrics from port 10255 on cAdvisor. If port 10250 on the kubelet in your registered cluster is open, you can perform the following steps to configure ack-arms-prometheus to collect metrics from port 10250 on cAdvisor (Container Advisor):

  1. Create a ServiceMonitor in the registered cluster based on the following YAML template.

    Show the YAML file content

    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      annotations:
        arms.prometheus.io/discovery: 'true'
      name: arms-prom-cadvisor-10250
      namespace: arms-prom
    spec:
      endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          honorLabels: true
          port: https-metrics
          relabelings:
            - sourceLabels:
                - __metrics_path__
              targetLabel: metrics_path
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecureSkipVerify: true
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          honorLabels: true
          path: /metrics/cadvisor
          port: https-metrics
          relabelings:
            - sourceLabels:
                - __metrics_path__
              targetLabel: metrics_path
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecureSkipVerify: true
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          honorLabels: true
          path: /metrics/probes
          port: https-metrics
          relabelings:
            - sourceLabels:
                - __metrics_path__
              targetLabel: metrics_path
          scheme: https
          tlsConfig:
            caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecureSkipVerify: true
      jobLabel: k8s-app
      namespaceSelector:
        matchNames:
          - kube-system
      selector:
        matchLabels:
          k8s-app: kubelet

    Wait 15 seconds to 1 minute before you start metric collection.

  2. Check whether metrics are collected.

    1. Log on to the ARMS console.

    2. In the left-side navigation pane, choose Managed Service for Prometheus > Prometheus Instances.

    3. In the top navigation bar of the Managed Service for Prometheus page, select the region where the cluster is deployed. Click the name of the Prometheus instance that you want to view to go to the Integration Center page.

    4. In the left-side navigation pane, click Service Discovery. On the page that appears, click the Targets tab.

      Check whether a collection task named arms-prom/arms-prom-cadvisor-10250 exists and runs as expected.采集数据