You can configure the alert ingestion system of Log Service as an Alertmanager component in Loki. After you complete the configuration, Loki sends alerts to the alerting system of Log Service. Then, the alerting system denoises the alerts and sends alert notifications.

Prerequisites

An alert ingestion application is created. The Protocol parameter of the alert ingestion application is set to Loki. For more information, see Configure webhook URLs for alert ingestion.

Background information

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system. You can ingest Loki alerts into the alerting system of Log Service. You can also use Grafana or Alertmanager to ingest Loki alerts into the alerting system of Log Service.

Configure Loki

In the configuration file of Loki, add the alertmanager_url parameter to the ruler parameter. Set the alertmanager_url parameter to the full URL of a webhook URL that is generated after you create an alert ingestion service and an alert ingestion application. For more information, see Obtain webhook URLs. 

ruler:
  alertmanager_url: {ALIYUN_SLS_ENDPOINT}/event/webhook/RAMAK_{ACCESS_KEY_ID}/{WEBHOOK_APP_ID}
  ...

Loki alerts

The following example shows the content of a Loki alert:
[
    {
        "annotations": {
            "description": "description info",
            "summary": "High request latency"
        },
        "endsAt": "2020-10-28T12:28:52.710Z",
        "startsAt": "2020-10-28T12:23:37.710Z",
        "generatorURL": "http://127.0.0.1:9090/graph?g0.expr=go_threads%7Binstance%3D%22localhost%3A9090%22%2Cjob%3D%22prometheus%22%7D+%3E+0\\u0026g0.tab=1",
        "labels": {
            "alertname": "HighErrorRate",
            "instance": "localhost:9090",
            "job": "prometheus",
            "severity": "page"
        }
    }
]

Field mappings

After you ingest a Loki alert into Log Service, the alert fields of Loki are converted to the alert fields of Log Service. The following example shows the fields of a Log Service alert:
{
    "aliuid": "{The ID of the Alibaba Cloud account to which the alert ingestion application belongs}",
    "alert_instance_id": "{The alert instance ID that is automatically generated}",
    "project": "{The project to which Alert Center belongs}",
    "region": "{The region to which the project belongs}",
    "alert_id": "HighErrorRate",
    "alert_type": "sls_pub",
    "alert_name": "HighErrorRate",
    "next_eval_interval": 78,
    "alert_time": 1603859020,
    "fire_time": 1603859017,
    "resolve_time": 0,
    "status": "firing",
    "labels": {
        "alertname": "HighErrorRate",
        "instance": "localhost:9090",
        "job": "prometheus"
    },
    "annotations": {
        "__pub_alert_region__": "{The region of the endpoint to which the alert is sent}",
        "__config_app__": "sls_pub_alert",
        "__pub_alert_service__": "{The ID of the alert ingestion service}",
        "__pub_alert_app__": "{The ID of the alert ingestion application}",
        "__pub_alert_protocol__": "loki",
        "severity": "page"
    },
    "severity": 2,
    "policy": {
        "alert_policy_id": "{The alert policy that is specified for the alert ingestion application}",
        "action_policy_id": "{The action policy that is specified for the alert ingestion application}",
        "repeat_interval": "{The cycle that is specified for the alert ingestion application}"
    },
    "drill_down_query": "http://127.0.0.1:9090/graph?g0.expr=go_threads%7Binstance%3D%22localhost%3A9090%22%2Cjob%3D%22prometheus%22%7D+%3E+0\\u0026g0.tab=1"
}
The following table describes the mappings between the alert fields of Log Service and the alert fields of Loki.
Alert field of Log Service Alert field of Loki Description
aliuid None The ID of the Alibaba Cloud account to which the alert ingestion application belongs.
alert_id alertname The ID of the alert monitoring rule.
alert_type None The type of the alert. Valid value: sls_pub.
alert_name alertname The ID of the alert monitoring rule.
status None The status of the alert. Valid values: firing and resolved.
  • If the value of the endsAt field in the Loki alert is greater than the value of the alert_time field, the value of the status field is firing.
  • If the value of the endsAt field in the Loki alert is less than the value of the alert_time field, the value of the status field is resolved.
next_eval_interval None The interval at which the alert is evaluated. If the current status of the alert is firing, the value of the next_eval_interval field is calculated by using the following formula: Evaluation interval = (endsAt - alert_time)/4.
alert_time None The time when Log Service receives the Loki alert.
fire_time startsAt The time when Log Service receives the Loki alert.
resolve_time None The time when the alert is cleared.
  • If the value of the endsAt field in the Loki alert is greater than the value of the alert_time field, the value of the resolve_time field is 0.
  • If the value of the endsAt field in the Loki alert is less than the value of the alert_time field, the value of the resolve_time field is a timestamp. The timestamp is the same as the timestamp of the endsAt field.
labels labels The labels of the alert.

If the severity field exists in the labels field of the Loki alert, the severity field is added to the annotations field after the alert is ingested into Log Service.

annotations annotations After the Loki alert is ingested into Log Service, the following fields are added to the annotations field of the corresponding Log Service alert:
  • __config_app__: "sls_pub_alert"
  • __pub_alert_service__: {The ID of the alert ingestion service}
  • __pub_alert_app__: {The ID of the alert ingestion application}
  • __pub_alert_protocol__: "loki"
  • __pub_alert_region__: {The region of the endpoint to which the alert is sent}
severity severity The severity of the alert. For more information, see Severity levels.
policy None The alert policy that is specified for the alert ingestion application. For more information, see Description of the policy variable.
project None The project to which Alert Center belongs. For more information, see Project.
drill_down_query generatorURL The value of the generatorURL field in the Loki alert is displayed.