All Products
Search
Document Center

Managed Service for Grafana:Push Grafana native alert events to ITSM

Last Updated:Jan 21, 2025

Information Technology Service Management (ITSM) of Application Real-Time Monitoring Service (ARMS) is an out-of-the-box intelligent alerting service designed to receive alert notifications from various sources, including Prometheus. The service processes these alerts by automatically deduplicating and grouping them, and applying algorithmic noise reduction techniques. Alerts are then routed to different notification targets based on their label content. This streamlined approach helps O&M teams efficiently manage alerts and significantly reduces project risks. ITSM allows you to send alert notifications through multiple notification methods, such as phone calls, SMS, emails, DingTalk, WeCom, Lark, and webhooks. Then, you can handle alerts in various scenarios with ease.

Alert parameters

Before you push Grafana native alert events to ITSM, you need to configure the alert parameters of the workspace on the Parameter Settings page. The parameters are as follows:

  • Set unified_alerting to true.

  • Set alerting to true.

Grafana 8.2.x and earlier versions

Step 1: Log on to Grafana

  1. Log on to the Managed Service for Grafana console. In the left-side navigation pane, click Workspace Management.

  2. On the Workspace Management page, find the workspace that you want to manage and click the URL in the URL column to go to Grafana.

    Note

    You can log on to Grafana with the administrator account of Grafana and the password that you configured when you created the workspace. You can also click Sign in with Alibaba Cloud to log on to Grafana with the current Alibaba Cloud account.

Step 2: Create a notification channel

First, search for a notification channel named ARMS_GRAFANA. If such a channel exists, skip this step. Otherwise, you need to create one.

  1. In the left-side navigation pane, choose image > Notification channels.

  2. On the Notification channels tab, click New channel. The following table describes the parameters.

    image.png

    Parameter

    Description

    Name

    Enter ARMS_GRAFANA.

    Type

    The type of the notification channel. In this example, select webhook.

    Url

    The endpoint where ARMS integrates Grafana alerts. For information about how to create a Grafana integration and obtain the endpoint, see Integrate Grafana alerts.

Step 3: Create an alert rule

  1. In the left-side navigation pane, choose image.png > Manage.

  2. On the Manage tab, click the dashboard for which you want to create an alert rule.

  3. In the dashboard, click 截屏2024-01-22 15.54.47_副本.png next to a panel name and click Edit.

  4. On the Alert tab of the Edit Panel page, click Create Alert.

  5. Configure other parameters based on your requirements. For more information, see Grafana documentation or Configure Grafana native alerting.

    In the Notifications section, click image next to Send to and select the ARMS_GRAFANA channel.

    image

  6. After the alert is created, click Save in the upper-right corner of the Edit Panel page.

    When an alert is triggered, the alert event is reported to the Alert Management > Alert Event History page of the ARMS console. For more information, see View historical alert events.

Step 4: Create a notification policy

Managed Service for Grafana creates a notification policy by default. To receive alert notifications from Grafana, you must configure the notification policy.

  1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Notification Policies.

  2. On the Notification Policy page, find the notification policy named ARMS_GRAFANA and click Edit in the Actions column. If no such a policy exists, create one. For more information, see Create a notification policy.

  3. In the Edit Notification Policy panel, click the Notification Objects tab. On the Notification Objects tab, configure the following settings. For more information, see Create and manage a notification policy.

    1. In the When an alert is triggered section, click Add Notification Object and select a notification endpoint and a notification method. The following table describes the parameters.

      Parameter

      Description

      Contacts

      Notification endpoints can be contacts, contact groups, schedules, DingTalk, Lark, WeCom, or webhooks. For information about how to create a notification endpoint, see Overview.

      Note

      Before a contact can receive alert notifications by using phone calls, the phone number of the contact must be verified. For information about how to verify a mobile number, see Contacts.

      Notification Methods

      Notification methods can be phone calls, SMS, or emails. Multiple notification methods can be selected at the same time.

    2. In the Notification Template section, set the notification formats of Email, Text Messages, Phone, and DingTalk/Lark/WeCom.

  4. Click Save.

Grafana versions later than 8.2.x

Note

The alerting interaction experiences of Grafana 9.0.x and Grafana 10.0.x are slightly different. This topic uses Grafana 9.0.x as an example.

Step 1: Disable the ARMS alerting plug-in

  1. Log on to the Managed Service for Grafana console. In the left-side navigation pane, click Workspace Management.

  2. On the Workspace Management page, click the ID of the workspace that you want to manage.

  3. In the left-side navigation pane, click Alert Management. On the page that appears, disable the ARMS alerting plug-in and enable Grafana native alerting.

Step 2: Create an alert rule

  1. On the Workspace Management page, find the workspace that you want to manage and click the URL in the URL column to go to Grafana.

    Note

    You can log on to Grafana with the administrator account of Grafana and the password that you configured when you created the workspace. You can also click Sign in with Alibaba Cloud to log on to Grafana with the current Alibaba Cloud account.

  2. In the left-side navigation pane, choose image.png > Browse.

  3. On the Browse tab, click the dashboard for which you want to create an alert rule.

  4. On the dashboard details page, move the pointer over the panel for which you want to create alerts, and click the arrow on the right side of the panel name. In the drop-down list that appears, select Edit.

  5. On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.

  6. Configure other parameters based on your business requirements. For more information, see Grafana documentation and Configure Grafana native alerting.

  7. After the alert is created, click Save in the upper-right corner.

    When an alert is triggered, the alert event is reported to the Alert Management > Alert Event History page of the ARMS console. For more information, see View historical alert events.

Step 3: Create a contact point

First search for a contact point named ARMS_GRAFANA. If such a contact point exists, skip this step. Otherwise, you need to create one.

  1. In the left-side navigation pane, choose image > Contact points.

  2. On the Contact points tab, click New contact Point or Add contact point. The following table describes the parameters.

    image

    Parameter

    Description

    Name

    Enter ARMS_GRAFANA.

    IContact point type

    The type of the contact point. In this example, select Webhook.

    Addresses

    The endpoint where ARMS integrates Grafana alerts. For information about how to create a Grafana integration and obtain the endpoint, see Integrate Grafana alerts.

Step 4: Create a notification policy

By creating a notification policy, you can link alert rules to the contact point.

  1. In the left-side navigation pane, choose image > Notification policies.

  2. On the Notification policies tab, click New nested policy or New specific policy. The following table describes the parameters.

    image.png

    Parameter

    Description

    Matching labels

    Select labels to specify the alert rules that apply.

    Contact point

    Specify the contact point to which alert events are sent.

    Configure the alert grouping and silencing options based on your needs.

Step 5: Create a notification policy

Managed Service for Grafana creates a notification policy by default. To receive alert notifications from Grafana, you must configure the notification policy.

  1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Notification Policies.

  2. On the Notification Policy page, find the notification policy named ARMS_GRAFANA and click Edit in the Actions column. If no such a policy exists, create one. For more information, see Create a notification policy.

  3. In the Edit Notification Policy panel, click the Notification Objects tab. On the Notification Objects tab, configure the following settings. For more information, see Create and manage a notification policy.

    1. In the When an alert is triggered section, click Add Notification Object and select a notification endpoint and a notification method. The following table describes the parameters.

      Parameter

      Description

      Contacts

      Notification endpoints can be contacts, contact groups, schedules, DingTalk, Lark, WeCom, or webhooks. For information about how to create a notification endpoint, see Overview.

      Note

      Before a contact can receive alert notifications by using phone calls, the phone number of the contact must be verified. For information about how to verify a mobile number, see Contacts.

      Notification Methods

      Notification methods can be phone calls, SMS, or emails. Multiple notification methods can be selected at the same time.

    2. In the Notification Template section, set the notification formats of Email, Text Messages, Phone, and DingTalk/Lark/WeCom.

  4. Click Save.

References