Application Real-Time Monitoring Service:Integrate self-managed Prometheus alerts

Background information

Open source Prometheus reports alerts to Alertmanager. Configure <alertmanager_config> in Prometheus.yaml to send alerts to ARMS Alert Management. For the configuration reference, see [alertmanager_config].

ARMS Alert Management provides an Alertmanager-compatible endpoint for open source Prometheus. When you create a Prometheus integration, ARMS Alert Management generates an integration address and a bearer token that your Prometheus instance uses to send alerts.

Prerequisites

• Access to the ARMS console. You obtain the bearer token for authentication from the integration settings page after you create the integration.

• For the Prometheus-Operator method: kubectl installed and configured with access to your Kubernetes cluster.

• For other self-managed Prometheus instances: write access to the prometheus.yaml configuration file on your Prometheus host.

Step 1: Create an integration and connect a self-managed Prometheus instance

2. In the New Prometheus Integration dialog box, enter the name and description of the Prometheus integration and the auto-recovery time of alert events, and then click Save.

   Note

   Automatic recovery time after timeout: If an alert event is not triggered again within the specified period of time, the alert event is automatically cleared.

   3. Select the connection method based on your Prometheus deployment. If your Prometheus is deployed through Prometheus-Operator, select Prometheus instances managed by Prometheus-Operator. Otherwise, select Other Self-managed Prometheus Instances. Then click OK.

Prometheus instances managed by Prometheus-Operator

• Create the arms-alertmanager-config.yaml file on your local machine, and then paste the following content into the file.

- scheme: https
  bearer_token: ******
  static_configs:
    - targets: ['alerts.aliyuncs.com']

• Use the arms-alertmanager-config.yaml file to create a secret named arms-am-config.

kubectl create secret generic arms-am-config --from-file=arms-alertmanager-config.yaml

• Create the prometheus-patch.yaml file on your local machine, and then paste the following content into the file.

spec:
  additionalAlertManagerConfigs:
    key: arms-alertmanager-config.yaml
    name: arms-am-config

• Run the following command to get the Prometheus object.

kubectl get prometheus

Sample response:

NAME                                                VERSION   REPLICAS   AGE
prometheus-operator-160879-prometheus               v2.18.2   1          4d1h

• Run the following command to apply the prometheus-patch.yaml file to the Prometheus object. Replace prometheus-operator-160879-prometheus with the name returned in the previous step.

kubectl patch prometheus prometheus-operator-160879-prometheus --patch "$(cat prometheus-patch.yaml)" --type=merge

Other Self-managed Prometheus Instances

• Modify the prometheus.yaml configuration file.

  The following example applies to Prometheus 2.x. For the configuration formats of other versions, see Prometheus documentation.

alerting:
  alertmanagers:
  - api_version: v1
    authorization:
      type: Bearer
      credentials: <bearer_token>
    scheme: https
    static_configs:
    - targets: ["alerts.aliyuncs.com"]

• In your browser, open the Prometheus configuration page and verify that the alerting section contains the Alertmanager configuration with the target alerts.aliyuncs.com.

Integrate other Prometheus-compatible observability systems with ARMS Alert Management

Thanos uses the --alertmanagers.config parameter of the thanos rule command to specify the Alertmanager configuration file. Enter the integration address and token described in the preceding steps. For more information about the configuration, see Thanos documentation.

VictoriaMetric uses the -notifier.config parameter of the ./bin/vmalert command to specify the notifier configuration file. Enter the integration address and token described in the preceding steps. For more information about the configuration, see VictoriaMetric documentation.

(Optional) Edit the integration

ARMS Alert Management presets the field mappings between the self-managed Prometheus alert source and ARMS Alert Management. You can add or modify the field mappings on the Edit Integration page as needed.

In the left-side navigation pane, choose Alert Management > Integrations. On the Alert Integration tab, you can perform the following operations on the integrations that you created:

• To update the key of an integration, in the Actions column on the right side of the target integration, choose MoreUpdate Key. In the dialog box that appears, click Confirm.