All Products
Search

This Product

    Managed Service for Grafana:Set up ITSM alert management in Managed Service for Grafana

    Document Center

    Managed Service for Grafana:Set up ITSM alert management in Managed Service for Grafana

    Last Updated:Mar 11, 2026

    When multiple Grafana alert rules fire simultaneously, raw alert volumes create noise that delays incident response. Application Real-Time Monitoring Service (ARMS) Information Technology Service Management (ITSM) receives alerts from Grafana and other sources such as Prometheus, automatically deduplicates, groups, and reduces noise, then routes actionable notifications to the right teams through channels such as email, SMS, DingTalk, or webhooks.

    How it works

    ITSM processes alerts through a four-stage pipeline:

    1. Alert rules evaluate conditions in your Grafana dashboards and fire alert events when thresholds are breached.

    2. Deduplication and noise reduction -- ITSM merges duplicate events and applies algorithmic noise reduction to suppress redundant notifications.

    3. Notification policies match alert events by label and route them to the appropriate notification targets.

    4. Notification targets deliver alerts through channels such as email, SMS, phone calls, DingTalk, Lark, WeCom, or webhooks (including PagerDuty and Microsoft Teams).

    The setup process follows this pipeline: enable alerting, create alert rules, configure notification targets, define notification policies, and verify delivery.

    Prerequisites

    Before you begin, ensure that you have:

    • A Managed Service for Grafana workspace running Grafana 9.0.x or later

    • Access to the ARMS console

    Note

    To check or update your Grafana version, see View or modify workspace information.

    Step 1: Enable alerting

    Grafana 9.0.x and later enables alerting by default. If alerting is turned off in your workspace, enable it manually.

    1. Log on to the ARMS console. In the left-side navigation pane, choose Managed Service for Grafana > Workspace Management.

      Alternatively, log on to the Managed Service for Grafana console and click Workspace Management in the left-side navigation pane.

    2. On the Workspace Management page, click the ID of the workspace that you want to manage.

    3. In the left-side navigation pane, click Alert Management. In the upper-left corner of the page, turn on the switch.

    4. In the confirmation dialog, click OK.

    5. In the upper-right corner of the Alert Management page, click Manage Alert Rules to open the Grafana ITSM page.

    Step 2: Create an alert rule

    Create a Grafana-managed alert rule from a dashboard panel, then add custom labels to route the alert through ITSM notification policies.

    Grafana 9.0.x

    1. In the left-side navigation pane, choose image > Browse.

    2. On the Browse tab, click the dashboard for which you want to create an alert rule.

    3. In the dashboard, click a panel name and select Edit.

    4. On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.

      Note

      Alternatively, click image in the left-side navigation pane. On the Alert Overview page, click Create Alert Rule.

    5. Configure the alert rule parameters. For parameter details, see the Grafana documentation.

      In the Notifications section, add custom labels as key-value pairs. For example: grafana_alertname=Pod disk usage _hangzhou. These labels match notification policies in Step 4.

      p569942

    6. Click Save in the upper-right corner.

    Grafana 10.0.x

    1. In the upper-left corner of Grafana, choose image > Dashboards.

    2. On the Dashboards page, select the dashboard for which you want to create an alert rule.

    3. In the dashboard, click the icon in the upper-right corner of a panel and click Edit.

    4. On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.

      Note

      Alternatively, choose image > ITSM in the left-side navigation pane. On the Alert Overview page, click Create Alert Rule.

    5. Configure the alert rule parameters. For parameter details, see the Grafana documentation.

      In the Notifications section, add custom labels as key-value pairs. For example: grafana_alertname=Pod disk usage _hangzhou. These labels match notification policies in Step 4.

      Grafana alert notification labels

    6. Click Save rule in the upper-right corner.

    Verify alert delivery

    When an alert fires, the event appears on the Alert Event History page of the ARMS console. For more information, see View historical alert events.

    Note

    On the Alert Event History page, set the Integration Name filter to ARMS_GRAFANA to show only events from Grafana alert rules.

    Step 3: Create notification targets

    Set up the channels through which ITSM delivers alert notifications.

    Notification target type Use case
    Contact and contact group Email, phone call, or SMS notifications to individuals or groups
    DingTalk, Lark, and WeCom Team group chat notifications for collaborative alert handling
    Webhook Third-party integrations such as PagerDuty, Microsoft Teams Incoming Webhook, or internal platforms
    On-call scheduling Rotating on-call assignments across team members

    Grafana 9.0.x

    In the left-side navigation pane, choose image > Notification Objects and create notification targets.

    Grafana 10.0.x

    In the upper-left corner of Grafana, click image, then choose ITSM > Notification Object and create notification targets.

    For setup instructions, see Notification targets.

    Step 4: Create a notification policy

    Notification policies route alert events to notification targets based on matching rules. Each policy matches alerts by the label key-value pairs attached to alert events, then forwards matching alerts to specified notification targets.

    Grafana 9.0.x

    1. In the left-side navigation pane, choose image > Notification Policies, then click Create Notification Policy.

    2. In the upper-left corner of the Create Notification Policy page, enter a notification policy name.

    3. In the Matching Rule step, configure a matching rule for alert events:

      1. Set Data Source to Grafana.

      2. Click Add Condition and enter the label keys and values from Step 2. For example: grafana_alertname equals Pod disk usage_hangzhou.

        Note

        As shown in the following figure, Condition 2 filters Grafana alert rules. If you do not add Condition 3, all alert rules in the current Grafana workspace use this notification policy.

        Notification policy matching rule

      3. Click Next.

    4. Configure the remaining parameters and click Save. For more information, see Create and manage a notification policy.

    Grafana 10.0.x

    1. In the upper-left corner of Grafana, click image, choose ITSM > Notification Policy, then click Create Notification Policy.

    2. In the upper-left corner of the Create Notification Policy page, enter a notification policy name.

    3. In the Matching Rule step, configure a matching rule for alert events:

      1. Set Data Source to Grafana.

      2. Click Add Condition and enter the label keys and values from Step 2. For example: grafana_alertname equals Pod disk usage_hangzhou.

        Note

        As shown in the following figure, Condition 2 filters Grafana alert rules. If you do not add Condition 3, all alert rules in the current Grafana workspace use this notification policy.

        Notification policy matching rule

      3. Click Next.

    4. Configure the remaining parameters and click Save. For more information, see Create and manage a notification policy.

    Step 5: Verify alert notifications

    When a notification policy is triggered, ITSM delivers alert notifications through the channels specified in the matched notification targets. Check your configured channels (email inbox, group chat, webhook endpoint) to confirm delivery.

    See also