After a self-managed Kubernetes cluster is connected to Application Real-Time Monitoring Service (ARMS) Prometheus, you cannot directly configure Prometheus in the Prometheus console. Instead, you need to use the kubectl command to manage the Prometheus.yaml configuration file, delete metrics, specify the number of agent replicas, and upgrade the Helm version.

Prerequisites

A self-managed Kubernetes cluster is connected to ARMS Prometheus. For more information, see Create a Prometheus instance for a Kubernetes cluster.

Create and modify the promethues.yaml file

  1. Create a promethues.yaml file. You can configure service discovery to enable remote read/write addresses. Examples:
    apiVersion: v1
kind: ConfigMap
metadata:
  name:  
arms-prom-prometheus-yaml
  namespace: arms-prom
  labels:
    target: arms
    type: prometheus-yaml
data:
  promYaml: |
    remote_write:
// Set the url parameter to the Remote Write URL that you obtained. 
          - url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_write"
          basic_auth:   
            // Set the username and password parameters to the AccessKey ID and AccessKey secret of your Alibaba Cloud account. 
            username: access-key-id
            password: access-key-secret
    Note
    • // Set the url parameter to the Remote Write URL that you obtained. For more information about how to obtain the Remote Write URL, see Remote Write and Remote Read URLs.
    • Set the username and password parameters to the AccessKey ID and AccessKey secret of your Alibaba Cloud account. For more information about how to obtain an AccessKey pair, see Obtain an AccessKey pair.
  2. Go to the directory where the promethues.yaml file is stored and run the following command:
    kubectl apply -f prometheus.yaml

Drop Prometheus metrics

If there are custom metrics that you no longer want your Prometheus instance to monitor, you can drop these metrics. Dropped metrics no longer incur charges.

Create a drop-metric.yaml file to specify the metrics that you want to drop. Examples:

apiVersion: v1
kind: ConfigMap
metadata:
  name: arms-prom-drop-metric
  namespace: arms-prom
  labels:
    target: arms
    type: drop-metric
data:
  dropMetric: |
    container_memory_rss
    container_memory_failures_total
    apiserver_request_total
    kube_pod_container_status_waiting_reason
    container_cpu_load_average_10s
    container_memory_max_usage_bytes
Note Metrics listed below dropMetric: are provided for reference. You can specify the metrics based on your business requirements.

Set the number of Prometheus agent replicas

If Prometheus agent replicas are insufficient, memory overflows and restarts may occur continuously. You can adjust the number of Prometheus agent replicas based on your business requirements.

Run the following command to change the number of agent replicas:

kubectl scale deployment arms-prometheus-ack-arms-prometheus --replicas 3 -n arms-prom
Note The number 3 in the command is the currently set number of agent replicas. You can change the number based on your business requirements. We recommend that you set the number equal to or greater than 3.

Upgrade Helm charts

ARMS Prometheus allows you to upgrade Helm charts. For more information, see Upgrade the Helm version.

  1. Run the following command to view the current version of your Helm chart: 
    helm list -n arms-prom
  2. Run the following command to view the latest available version of Helm charts: 
    helm search repo ack-arms-prometheus -n arms-prom

    The following figure shows an example of the command output.

    jl
  3. As shown in the preceding figure, if the 1 is the same as the 2, you do not need to upgrade the Helm chart. Otherwise, run the following command to upgrade your Helm chart. 
    helm upgrade arms-prometheus aliyun/ack-arms-prometheus -n arms-prom
  4. Run the following command to view the historical versions of your Helm chart: 
    helm history arms-prometheus -n arms-prom

    The following figure shows an example of the command output.

    jkl