Prometheus is an open source monitoring system that provides the Alert Manager feature. Alert Manager is a service that handles alerts. After you configure the alert ingestion system of Log Service as a receiver in Alert Manager, you can send alerts that are generated by Alert Manager to Log Service.

Prerequisites

An alert ingestion application is created and the Protocol parameter is set to AlertManager. For more information, see Configure webhook URLs for alert ingestion.

Configure Alert Manager

In the configuration file of Alert Manager, add a receiver to the route parameter. Then, configure a notification method in the receivers parameter. Sample configuration template:
route:
  receiver: '{RECIEVER_NAME}'
  ...
  
...

receivers:
- name: '{RECIEVER_NAME}'
  webhook_configs:
  - url: 'http://{ALIYUN_SLS_ENDPOINT}/event/webhook/RAMAK_{ACCESS_KEY_ID}/{WEBHOOK_APP_ID}'
  
- name: ...
Parameter Description
receiver The name of the custom receiver.
name The name of the custom receiver. The name must be the same as the name that you set in the receiver parameter.
url The receiving end of alerts. Set the value to the endpoint that is generated after you create an alert ingestion service and alert ingestion application. The value is the complete URL of the endpoint. For more information, see Obtain webhook URLs.
Note If your Alert Manager server is deployed on an Elastic Compute Service (ECS) instance, we recommend that you select the region where the ECS instance resides. Then, use the related LAN or virtual private cloud (VPC) endpoint. You can also use the Internet webhook URL of a region.

Parse Alert Manager alerts

The following code provides an example of an Alert Manager alert.
Note If the alerts field includes multiple alerts, the system maps the content of the alerts field to multiple Log Service alerts. A group of entries that are enclosed in braces {} indicates an alert.
{
  "version": "4",
  "groupKey": "alertname,job",
  "truncatedAlerts": 0,
  "status": "firing",
  "receiver": "webhook",
  "groupLabels": {
    "alertname": "Alert",
    "job": "promethueus"
  },
  "commonLabels": {
    "instance": "localhost:9090"
  },
  "commonAnnotations": {},
  "externalURL": "http://example.com/#/alerts",
  "alerts": [
    {
      "annotations": {
        "description": "description info",
        "summary": "High request latency"
      },
      "endsAt": "2020-10-28T12:28:52.710Z",
      "startsAt": "2020-10-28T12:23:37.710Z",
      "generatorURL": "",
      "labels": {
        "alertname": "Alert",
        "instance": "localhost:9090",
        "job": "prometheus",
        "severity": "page"
      }
    },
    ...
  ]
}

Field mapping

After an Alert Manager alert is ingested to Log Service, the alert is converted to an alert that is supported by Log Service by using field mapping. The following code provides an example of an Alert Manager alert:

{
  "aliuid": "{The ID of the Alibaba Cloud account to which the alert ingestion application belongs}",
  "alert_instance_id": "{The alert instance ID that is automatically generated}",
  "project": "{The project to which Alert Center belongs}",
  "region": "{The region of the endpoint to which the alert is sent}",
  "alert_id": "Alert",
  "alert_type": "sls_pub",
  "alert_name": "Alert",
  "next_eval_interval": 0,
  "alert_time": 1603859020,
  "fire_time": 1603859017,
  "resolve_time": 0,
  "status": "firing",
  "labels": {
    "instance": "localhost:9090",
    "job": "prometheus"
  },
  "annotations": {
    "__pub_alert_region__": "{The region of the endpoint to which the alert is sent}",
    "__config_app__": "sls_pub_alert",
    "__pub_alert_service__": "{The ID of the alert ingestion service}",
    "__pub_alert_app__": "{The ID of the alert ingestion application}",
    "__pub_alert_protocol__": "alert_manager",
    "desc": "description info",
    "summary": "High request latency"
  },
  "severity": 2,
  "policy": {
    "alert_policy_id": "{The alert policy that is specified for the alert ingestion application}",
    "action_policy_id": "{The action policy that is specified for the alert ingestion application}",
    "repeat_interval": "{The cycle that is specified for the alert ingestion application}"
  },
  "drill_down_query": "http://example.com:9090/#/alerts?filter=%7B\"alertname\"%3D\"Alert\"%2C%20\"job\"%3D\"Promethues\"%7D"
},
...
The following table describes the mappings between the content of a Log Service alert and the content of an Alert Manager alert.
Log Service Alert Manager Description
aliuid N/A The ID of the Alibaba Cloud account to which the alert ingestion application belongs.
alert_id alertname The ID of the alert rule.

If the labels field of the Alert Manager alert includes the alertname field, the value of the alert_id is set to the value of the alertname field. Otherwise, the value of the alert_id field is empty.

alert_type N/A The alert type. Valid value: sls_pub.
alert_name alertname The name of an alert rule.

If the labels field of an Alert Manager alert includes the alertname field, the value of the alert_name field is set to the value of the alertname field. Otherwise, the value of the alert_name field is empty.

status status The alert status.
next_eval_interval N/A The interval at which the alert is evaluated. Valid value: 0.
alert_time N/A The time when the alert is first triggered. The time indicates when Log Service receives the Alert Manager alert.
fire_time startsAt The time when the alert is triggered.
resolve_time endsAt The time when the alert is cleared.

If the status parameter of an Alert Manager alert is set to firing, the resolve_time parameter is set to 0. Otherwise, the resolve_time parameter is set to the timestamp of the endsAt parameter.

labels labels The labels of the alert.

If you add a label on the Enrichment tab when you create the alert ingestion application, the specified label is added to the labels field.

annotations annotations The annotation of the alert. The following fields are added to the annotations field of a Log Service alert.
  • __config_app__: "sls_pub_alert"
  • __pub_alert_service__: {The ID of the alert ingestion service}
  • __pub_alert_app__: {The ID of the alert ingestion application}
  • __pub_alert_protocol__: "alert manager"
  • __pub_alert_region__: {The region of the endpoint to which the alert is sent}
  • desc:

    If the annotations field of an Alert Manager alert includes the desc, description, and summary fields, the value of the desc field is the combination of the values of the preceding three fields.

If you add an annotation on the Enrichment tab when you create the alert ingestion application, the specified annotation is added to the annotations field.

severity severity The severity of the alert.
Note If the severity field exists in a Grafana alert, Log Service maps the severity of the alert to the corresponding severity after the alert is ingested into Log Service. If the severity field does not exist, the default severity of the Grafana alert is set to medium. For more information, see Severity levels.
policy N/A The alert policy that is specified for the alert ingestion application. For more information, see Data structure of the policy variable.
project N/A The project to which Alert Center belongs. For more information, see Project.
drill_down_query
  • generatorURL
  • externalURL and groupLabels
The URL of the Alert Manager alert.
  • If an Alert Manager alert includes the generatorURL field, the value of the drill_down_query field is set to the value of the generatorURL field.
  • If an Alert Manager alert does not include the generatorURL field and the value of the groupLabels field is empty, the value of the drill_down_query field is set to the value of the externalURL field.
  • If an Alert Manager alert does not include the generatorURL field and the value of the groupLabels field is not empty, the value of the drill_down_query field is the combination of the value of the externalURL field and the value of the groupLabels field.

After you click the URL, you are redirected to the Alert Management page of Alert Manager. On this page, Alert Manager automatically shows alerts that match the content of the groupLabels field.