Grafana provides a GUI that allows you to use the alerting feature. You can add a contact point in Grafana so that Grafana can send alerts to the alerting system of Log Service. Then, the alerting system processes the alerts, such as denoising the alerts and sending alert notifications. If your Grafana version is 8.0 or later, you can follow the instructions in this topic.
Prerequisites
An alert ingestion application is created. For more information, see Configure webhook URLs for alert ingestion.Configure Grafana
- Log on to the Grafana console.
- Create a contact point.
- In the left-side navigation pane, choose , and then click New contact point.
- On the Create contact point page, configure the following parameters.
Parameter Description Name The name of the contact point. Contact point type The type of the contact point. Select webhook. Url The URL of the contact point. In this example, enter the full URL of the webhook URL that is generated after you create an alert ingestion service and an alert ingestion application in the alert ingestion system of Log Service. For more information, see Obtain webhook URLs. - Click Save contact point.
- Add the contact point to a notification policy.
You can add the contact point to the root notification policy, or create a route and add the contact point to the route. In this example, add the contact point to the root notification policy.
- Click Notification policies.
- In the Root policy - default for all alerts section, click Edit.
- From the Default contact point list, select the contact point that you created in step 2.
- Click Save.
Grafana alerts
The following sample code provides an example of a Grafana alert:
{
"receiver": "http",
"status": "firing",
"alerts":
[
{
"status": "firing",
"labels":
{
"alertname": "test"
},
"annotations":
{
"description": "test alert",
},
"startsAt": "2022-02-17T21:44:07.106902+08:00",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://localhost:3000/alerting/abcd/edit",
"fingerprint": "65d7bbdb01ce6c93",
"silenceURL": "http://localhost:3000/alerting/silence/new?alertmanager=grafana&matchers=alertname%3Dtest",
"dashboardURL": "",
"panelURL": "",
"valueString": "[ metric='go_threads{instance=\"prometheus\", job=\"prometheus\"}' labels={__name__=go_threads, instance=prometheus, job=prometheus} value=15 ]"
}
],
"groupLabels":
{},
"commonLabels":
{
"alertname": "test"
},
"commonAnnotations":
{
"description": "test alert",
},
"externalURL": "http://localhost:3000/",
"version": "1",
"groupKey": "{}/{alertname=\"test\"}:{}",
"truncatedAlerts": 0,
"orgId": 1,
"title": "[FIRING:1] (test)",
"state": "alerting",
"message": "**Firing**\n\nValue: [ metric='go_threads{instance=\"prometheus\", job=\"prometheus\"}' labels={__name__=go_threads, instance=prometheus, job=prometheus} value=15 ]\nLabels:\n - alertname = test\nAnnotations:\n - description = test alert\nSource: http://localhost:3000/alerting/abcd/edit\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana&matchers=alertname%3Dtest\n"
}Field mappings
After a Grafana alert is ingested into Log Service, the alert is converted to a Log Service alert based on field mappings. The following sample code provides an example of a Log Service alert:
{
"aliuid": "{The ID of the Alibaba Cloud account to which the alert ingestion application belongs}",
"alert_instance_id": "{The alert instance ID that is automatically generated}",
"alert_id": "test",
"alert_type": "sls_pub",
"alert_name": "test",
"region": "{The region of the project to which Alert Center belongs}",
"project": "{The project to which Alert Center belongs}",
"project_id": 0,
"next_eval_interval": 0,
"alert_time": 1645607565,
"fire_time": 1645105447,
"fire_results": null,
"fire_results_count": 0,
"resolve_time": 0,
"status": "firing",
"results": null,
"labels":
{},
"annotations":
{
"__config_app__": "sls_pub_alert",
"__pub_alert_app__": "{The ID of the alert ingestion application}",
"__pub_alert_protocol__": "grafana8",
"__pub_alert_region__": "{The region of the endpoint to which the alert is sent}",
"__pub_alert_service__": "{The ID of the alert ingestion service}",
"desc": "test alert",
"fingerprint": "65d7bbdb01ce6c93",
"silenceURL": "http://localhost:3000/alerting/silence/new?alertmanager=grafana&matchers=alertname%3Dtest",
"valueString": "[ metric='go_threads{instance=\"prometheus\", job=\"prometheus\"}' labels={__name__=go_threads, instance=prometheus, job=prometheus} value=15 ]"
},
"severity": 6,
"policy":
{
"alert_policy_id": "{The alert policy that is specified for the alert ingestion application}",
"action_policy_id": "{The action policy that is specified for the alert ingestion application}",
"use_default": false,
"repeat_interval": "{The repeat interval that is specified for the alert ingestion application}"
},
"template": null,
"drill_down_query": "http://localhost:3000/alerting/abcd/edit",
}
| Log Service | Grafana | Description |
| aliuid | N/A | The Alibaba Cloud account ID associated with the alert ingestion application. |
| alert_id | alertname | The alert rule ID. If the labels field of a Grafana alert message contains the alertname field, the value of alert_id is the value of the alertname field. Otherwise, it is empty. |
| alert_type | N/A | The alert type. The value is fixed to sls_pub. |
| alert_name | alertname | The alert rule name. If the labels field of a Grafana alert message contains the alertname field, alert_name is set to the value of the alertname field, and the original field is deleted. Otherwise, it is empty. |
| status | status | The alert status. |
| next_eval_interval | N/A | The alert evaluation interval. The value is fixed to 0. |
| alert_time | N/A | The timestamp when Log Service received the Grafana alert. |
| fire_time | startsAt | The time when the alert was triggered. |
| resolve_time | endsAt | The time when the alert was resolved. If the status in a Grafana alert message is firing, the value of resolve_time is 0. Otherwise, the value is the timestamp that corresponds to endsAt. |
| labels | labels | The alert labels. If you add tag information in Information Processing when you create an alert ingestion application, this tag information is added to the labels field. |
| annotations | annotations | annotationsThe alert annotations. The original Grafana are retained, and the following fields are added:
If you add annotations in Information Processing when you create an alert ingestion application, the annotations will be added to the annotations field. |
| severity | severity | The alert severity. Note annotationslabelsseverityLog Service maps the value of the field from the Grafana or to a Log Service alert severity level. If a field is not found in either location, the alert severity defaults to Medium. For more information, see Alert severity. |
| policy | N/A | The alert policy configured in the alert ingestion application. For more information, see Policy structure. |
| project | N/A | The Log Service project that contains the Alert Center. For more information, see Project. |
| drill_down_query |
|
A URL to drill down into the alert in Grafana.
Clicking the URL takes you to the Grafana alert management page and automatically displays alert messages that match the content of the groupLabels field. |