Prometheus is an open source monitoring tool for cloud-native applications. This topic describes how to deploy Prometheus to a Kubernetes cluster.

Prerequisites

Background information

A monitoring system monitors the following two types of objects:
  • Resources: resource usage of nodes and applications. In a Kubernetes cluster, a monitoring system monitors the resource usage of nodes, pods, and the cluster.
  • Applications: internal metrics of applications. For example, the monitoring system counts the number of online users that use an application in real time, and uses the exposed ports of the application to monitor services and generate alerts.
In a Kubernetes cluster, a monitoring system monitors the following objects:
  • Built-in components of the Kubernetes cluster, such as kube-apiserver, kube-controller-manager, and etcd.
  • Static resource entities, such as node resource status and kernel events.
  • Dynamic resource entities, which are the entities of abstract workloads in Kubernetes, such as Deployments, DaemonSets, and pods.
  • Custom objects in applications, which include the custom data and metrics in applications.

To monitor cluster components and static resource entities, specify monitoring methods in the configuration file.

To monitor dynamic resource entities, you can deploy Prometheus to a Kubernetes cluster.

Procedure

  1. Deploy Prometheus.
    1. Log on to the Container Service console.
    2. In the left-side navigation pane, choose Marketplace > App Catalog. On the page that appears, select ack-prometheus-operator.
    3. In the Deploy section of the App Catalog - ack-prometheus-operator page, select the target cluster, and click Create to deploy Prometheus. Prometheus-operator
      Verify the deployment result.
      1. Run the following command to map Prometheus in the cluster to local port 9090.
        kubectl port-forward svc/ack-prometheus-operator-prometheus 9090:9090 -n monitoring
      2. Use a browser to visit the Prometheus page at localhost:9090.
      3. In the top navigation bar, choose Status > Targets to view all data collection tasks.Data collection tasksTasks in the UP state are running properly.Targets
  2. View the aggregate data.
    1. Run the following command to map Grafana in the cluster to local port 3000.
      kubectl -n monitoring port-forward svc/ack-prometheus-operator-grafana 3000:80
    2. To view aggregate data, use a browser to visit localhost:3000, and select a dashboard.
      Dashboard
  3. View alert rules and set silent alerts.
    • View alert rules
      To view the alert rules, use a browser to visit localhost:9090, and click Alerts in the top navigation bar.
      • If an alert rule is in red, alerts are triggered based on this rule.
      • If an alert rule is in green, no alerts are triggered based on this rule.
      Alerts
    • Set silent alerts
      Run the following command. Use a browser to visit localhost:9093, and click Silence to set silent alerts.
      kubectl --namespace monitoring port-forward svc/alertmanager-operated 9093
      Silence

You can follow the preceding steps to deploy Prometheus. The following examples describe configurations in different scenarios.

Alert configuration

To set alert methods or notification templates, perform the following steps to configure the config element in the alertmanager module.

  • Set alert methods
    Prometheus that is deployed to a Kubernetes cluster can send notifications by DingTalk and email. You can perform the following steps to set alert methods.
    • Configure DingTalk notifications

      On the App Catalog - ack-prometheus-operator page, click the Parameters tab. In the dingtalk module, set enabled to true. In the token element of the module, enter webhook URLs of your DingTalk chatbots. In the alertmanager module, set the receiver parameter in the config element to the DingTalk alert name in the receivers element. The default value is webhook1.

      For example, if you have two DingTalk chatbots, perform the following steps:
      1. Replace the parameter values in the token element with webhook URLs of DingTalk chatbots.
        Copy the webhook URLs of your DingTalk chatbots and use the copied URLs to replace the parameter values of dingtalk1 and dingtalk2 in the token element. In the following example, use the webhook URLs to replace https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxx.
        Token configuration
      2. Modify the value of the receiver parameter.

        In the alertmanager module, find the receiver parameter in the config element, and set the parameter to the DingTalk alert name in the receivers element. In this example, the parameter values are webhook1 and webhook2.

      3. Modify the url parameter.
        The following example is based on chatbots dingtalk1 and dingtalk2. Modify the url parameter based on the actual chatbot names.
        Webhook configuration
      Note You can add the configurations of more DingTalk chatbots in the receivers element.
    • Configure email notifications
      On the App Catalog - ack-prometheus-operator page, click the Parameters tab. Set the parameters in the second red box of the following figure. In the alertmanager module, set the receiver parameter in the config element to the email alert name in the receivers element. The default name is mail.Configure email notifications
  • Set alert notification templates
    In the templateFiles element of the alertmanager module on the Parameters tab, you can customize alert notification templates.Template settings

Storage configuration

Monitoring data that is generated by Prometheus can be stored in Time Series Database (TSDB) or on disks. You can perform the following steps to configure data storage.
  • Store data in TSDB
    On the App Catalog - ack-prometheus-operator page, click the Parameters tab. In the tsdb module, set enabled to true. Then, set the url parameter for remoteRead and remoteWrite. .TDSP
  • Store data on disks
    By default, Prometheus data in Kubernetes is stored on disks. You can configure disk storage in the prometheus or alertmanager module on the Parameters tab. On the App Catalog - ack-prometheus-operator page, click the Parameters tab. Find the storage element in the alertmanager module or the storageSpec element in the prometheus module, and set related parameters. storageClassName specifies the disk category, accessModes specifies the access mode, and storage specifies the storage capacity.DATA
    Note Assume that you want to configure an SSD to store the data. In the storageSpec element, set storageClassName to alicloud-disk-ssd, accessModes to ReadWriteOnce, and storage to 50Gi.DISKS

    To verify the configuration, you can choose Storage & Snapshots > Disks in the ECS console.

    For information about how to reuse a disk, see Use Alibaba Cloud disks as volumes.

Enable the Prometheus adapter

To enable the Prometheus adapter, you can set enabled to true in the prometheusAdapter module and customize metrics. This allows Kubernetes cluster to automatically scale the number of pods based on the custom metrics to improve resource usage.

You can perform the following steps to enable the Prometheus adapter. On the App Catalog - ack-prometheus-operator page, click the Parameters tab. In the prometheusAdapter module, set enabled to true.adapter
You can run the following command to verify the configuration. For information about how to customize metrics, see Prometheus adapter.
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Mount a custom ConfigMap to Prometheus

This topic describes how to mount a ConfigMap to the /etc/prometheus/configmaps/ path of a pod.

If you deploy Prometheus for the first time, perform the following steps.

On the App Catalog - ack-prometheus-operator page, click the Parameters tab. In the configMaps element of the prometheus module, enter the name of the custom ConfigMap.ConfigMap

If Prometheus has been deployed to the cluster, perform the following steps.

  1. In the left-side navigation pane, choose Applications > Releases. The Releases page appears.
  2. On the Helm tab, find the target application, and click Update on the right. The Update Release page appears.
  3. In the configMaps element of the prometheus and alertmanager modules, enter the name of the custom ConfigMap and click Update.
    conguration
    Note Assume that you want to mount a custom ConfigMap named special-config and this ConfigMap contains the configuration file of Prometheus. To use the configuration file as the value of the --config.file parameter when the Prometheus pod starts, enter special-config in the configMaps element of the prometheus module. After you perform this operation, the ConfigMap is mounted to the Prometheus pod in the /etc/prometheus/configmaps/ path.
    The following figure shows an example of the YAML file for special-config.config
    The following figure shows how to set the configMaps element in the prometheus module.configmap

Configure Grafana

  • Mount a dashboard to Grafana
    To mount a dashboard as a ConfigMap to a Grafana pod, perform the following steps: On the App Catalog - ack-prometheus-operator page, click the Parameters tab. Set the parameters in the extraConfigmapMounts element.dashboard
    Note
    • Make sure that a dashboard is stored as a ConfigMap in the Kubernetes cluster.

      This ConfigMap must have a label that is also attached to other ConfigMaps.

    • In the extraConfigmapMounts element of the grafana module, set the ConfigMap name and mount path.
    • Set mountPath to /tmp/dashboards/.
    • Set configMap to the name of your custom ConfigMap.
    • Set name to the name of the JSON file that stores the dashboard information.
  • Enable persistence of dashboards

    In the ACK console, choose Applications > Releases. Find ack-prometheus-operator, and click Update on the right. In the Update Release dialog box, find the persistence element in the grafana module, and set parameters as shown in the following figure.

    Enable persistence of Grafana dashboards

    You can export Grafana dashboards in JSON format. For more information, see Export a Grafana dashboard.

FAQ

  • What can I do if I fail to receive DingTalk alert notifications?
    1. Obtain the webhook URL of your DingTalk chatbot. For more information, see Scenario 1: Use DingTalk to implement monitoring and alerting of Kubernetes events.
    2. On the Parameters tab, find the dingtalk module, set enabled to true, and enter the webhook URL of your DingTalk chatbot in the token element. For more information, see Configure DingTalk alert notifications in Alert configuration.
  • What can I do if the following error message appears when I deploy Prometheus to a cluster?
    <input tabindex="-1" class="dnt" readonly="readonly" value="Do Not Translate">
    Can't install release with errors: rpc error: code = Unknown desc = object is being deleted: customresourcedefinitions.apiextensions.k8s.io "xxxxxxxx.monitoring.coreos.com" already exists
    According to the error message, the cluster fails to clear custom resource definitions (CRDs) of the previous deployment. Run the following commands to delete CRDs and then deploy Prometheus again.
    kubectl delete crd prometheuses.monitoring.coreos.com
    kubectl delete crd prometheusrules.monitoring.coreos.com
    kubectl delete crd servicemonitors.monitoring.coreos.com
    kubectl delete crd alertmanagers.monitoring.coreos.com
  • What can I do if I do not receive email alert notifications?

    Make sure that the value of smtp_auth_password is the SMTP authorization code instead of the logon password of the email account. Make sure that the STMP server endpoint includes a port number.

  • What can I do if the following error message appears when I update a YAML file?error

    If the configuration file of Tiller is overlarge, the cluster cannot be accessed. To solve this issue, you can delete some annotations in the configuration file and mount the file to a pod as a ConfigMap. You can enter the name of the custom ConfigMap in the configMaps element of the prometheus and alertmanager modules. For more information, see the second method in Mount a custom ConfigMap to Prometheus.

  • How do I enable the features of the deployed Prometheus?

    Choose Releases > Helm > Update, and find the YAML file of Prometheus. Set enabled to true for the features that you want to enable. Then, click Update to enable the features.

  • How do I select data storage: TSDB or disks?
    TSDB storage is available to restricted regions. However, disk storage is supported in all regions. The following figure shows the data retention policy.policy