Alertmanager is a service that you can use to handle alerts. This service is provided by Prometheus, an open source monitoring system. After you configure the alert ingestion system of Log Service as a receiver in Alertmanager, Alertmanager sends alerts to Log Service.

Prerequisites

An alert ingestion application whose Protocol is Alertmanager is created. For more information, see Configure webhook URLs for alert ingestion.

Configure Alertmanager

In the configuration file of Alertmanager, add a custom receiver to route. Then, add the configuration of the custom receiver to receivers. Configuration template:
route:
  receiver: '{RECIEVER_NAME}'
  ...
  
...

receivers:
- name: '{RECIEVER_NAME}'
  webhook_configs:
  - url: 'http://{ALIYUN_SLS_ENDPOINT}/event/webhook/RAMAK_{ACCESS_KEY_ID}/{WEBHOOK_APP_ID}'
  
- name: ...
Parameter Description
receiver The name of the custom receiver.
name The name of the custom receiver. This name must be the same as the name that you specify for the receiver parameter.
url The URL of the custom receiver. Enter the full URL of the webhook URL that is generated after you create an alert ingestion service and an alert ingestion application in the alert ingestion system of Log Service. For more information, see Obtain webhook URLs.
Note If your Alertmanager server is deployed on an Elastic Compute Service (ECS) instance, we recommend that you select the region where the ECS instance resides and use the internal endpoint that is accessible over a LAN or virtual private cloud (VPC) when you configure the region information. If your Alertmanager server is not deployed on an ECS instance, you can select a region and use the public endpoint that is accessible over the Internet.

Alertmanager alerts

The following sample code provides an example of an Alertmanager alert:
Note If the alerts field includes multiple alerts, each alert is converted to a Log Service alert. In the alerts field, each alert is enclosed in a separate pair of braces {}. 
{
  "version": "4",
  "groupKey": "alertname,job",
  "truncatedAlerts": 0,
  "status": "firing",
  "receiver": "webhook",
  "groupLabels": {
    "alertname": "Alert",
    "job": "promethueus"
  },
  "commonLabels": {
    "instance": "localhost:9090"
  },
  "commonAnnotations": {},
  "externalURL": "http://example.com/#/alerts",
  "alerts": [
    {
      "annotations": {
        "description": "description info",
        "summary": "High request latency"
      },
      "endsAt": "2020-10-28T12:28:52.710Z",
      "startsAt": "2020-10-28T12:23:37.710Z",
      "generatorURL": "",
      "labels": {
        "alertname": "Alert",
        "instance": "localhost:9090",
        "job": "prometheus",
        "severity": "page"
      }
    },
    ...
  ]
}

Field mappings

After an Alertmanager alert is ingested into Log Service, the alert is converted to a Log Service alert based on field mappings. The following sample code provides an example of a Log Service alert: 

{
  "aliuid": "{The ID of the Alibaba Cloud account to which the alert ingestion application belongs}",
  "alert_instance_id": "{Automatically generated}",
  "project": "{The project to which Alert Center belongs}",
  "region": "{The region of the project to which Alert Center belongs}",
  "alert_id": "Alert",
  "alert_type": "sls_pub",
  "alert_name": "Alert",
  "next_eval_interval": 0,
  "alert_time": 1603859020,
  "fire_time": 1603859017,
  "resolve_time": 0,
  "status": "firing",
  "labels": {
    "instance": "localhost:9090",
    "job": "prometheus"
  },
  "annotations": {
    "__pub_alert_region__": "{The region of the endpoint to which the alert is sent}",
    "__config_app__": "sls_pub_alert",
    "__pub_alert_service__": "{The ID of the alert ingestion service}",
    "__pub_alert_app__": "{The ID of the alert ingestion application}",
    "__pub_alert_protocol__": "alert_manager",
    "desc": "description info",
    "summary": "High request latency"
  },
  "severity": 2,
  "policy": {
    "alert_policy_id": "{The ID of the alert policy that is specified for the alert ingestion application}",
    "action_policy_id": "{The ID of the action policy that is specified for the alert ingestion application}",
    "repeat_interval": "{The cycle that is specified for the alert ingestion application}"
  },
...
The following table describes the field mappings between Log Service and Alertmanager alerts.
Log Service Alertmanager Description
aliuid None The ID of the Alibaba Cloud account to which the alert ingestion application belongs.
alert_id alertname The ID of the alert monitoring rule.

If the labels field in the Alertmanager alert includes the alertname field, the value of the alert_id field is the same as the value of the alertname field. Otherwise, the alert_id field is empty.

alert_type None The type of the alert. The value is fixed as sls_pub.
alert_name alertname The name of the alert monitoring rule.

If the labels field in the Alertmanager alert includes the alertname field, the value of the alert_name field is the same as the value of the alertname field. Otherwise, the alert_name field is empty.

status status The status of the alert.
next_eval_interval None The interval at which the alert is evaluated. The value is fixed as 0.
alert_time None The time at which the alert is triggered. The time indicates when Log Service receives the Alertmanager alert.
fire_time startsAt The time at which the alert is first triggered.
resolve_time endsAt The time at which the alert is cleared.

If the value of the status field in the Alertmanager alert is firing, the value of the resolve_time field is 0. Otherwise, the value of the resolve_time field is the timestamp indicated by the endsAt field.

labels labels The labels of the alert.

If you add a label on the Enrichment tab when you create the alert ingestion application, the label is added to the labels field.

annotations annotations The annotations of the alert. The following fields are added to the annotations field in the Log Service alert.
  • __config_app__: "sls_pub_alert"
  • __pub_alert_service__: {The ID of the alert ingestion service}
  • __pub_alert_app__: {The ID of the alert ingestion application}
  • __pub_alert_protocol__: "alert manager"
  • __pub_alert_region__: {The region of the endpoint to which the alert is sent}
  • desc:

    If the annotations field in the Alertmanager alert includes the desc, description, and summary fields, the value of the desc field in the Log Service alert is a combination of the values of the three fields.

If you add an annotation on the Enrichment tab when you create the alert ingestion application, the annotation is added to the annotations field.

severity severity The severity of the alert.
Note If an Alertmanager alert contains the severity field, Log Service maps the value of this field to the corresponding severity after the alert is ingested into Log Service. If an Alertmanager alert does not contain the severity field, Log Service uses the Medium severity for the alert. For more information, see Severity levels.
policy None The alert policy that is specified for the alert ingestion application. For more information, see Data structure of the policy variable.
project None The project to which Alert Center belongs. For more information, see Project.
drill_down_query
  • generatorURL
  • externalURL and groupLabels
 The URL of the Alertmanager alert.
  • If an Alertmanager alert contains the generatorURL field, the value of the drill_down_query field is the same as the value of the generatorURL field.
  • If an Alertmanager alert does not contain the generatorURL field and the groupLabels field is empty, the value of the drill_down_query field is the same as the value of the externalURL field.
  • If an Alertmanager alert does not contain the generatorURL field and the groupLabels field is not empty, the value of the drill_down_query field is a combination of the values of the externalURL and groupLabels fields.

If you click the URL, you are redirected to the alert management page of Alertmanager. On this page, Alertmanager automatically displays alerts that match the content of the groupLabels field.