ITSM is an out-of-the-box, intelligent alert management service. It receives alerts from various clients, such as Prometheus, and processes them using automated deduplication, grouping, and algorithmic noise reduction. Alerts are then routed to different notification targets based on label content to help O&M teams reduce project risks. This topic describes how to enable ITSM in Managed Service for Grafana.
Grafana version requirements
ITSM is supported in Grafana 9.0.x and later. To learn how to upgrade your Grafana version, see View workspace information.
Step 1: Enable ITSM
ITSM is enabled by default in Grafana 9.0.x and later. If you have previously disabled integrated ITSM on the Alert Management page, follow this step to re-enable it.
-
Log on to the Managed Service for Grafana console. In the left-side navigation pane, click Workspace Management.
-
On the Workspace Management page, click the ID of the workspace that you want to manage.
In the left-side navigation pane, click Alert Management, and then turn on the ITSM switch in the upper-left corner of the page.
In the dialog box that appears, click OK.
In the upper-right corner of the Alert Management page, click Manage Alert Rules to go to the ITSM page in Grafana.
Step 2: Create an alert rule
Grafana 9.0.x
In the left-side navigation pane of Grafana, choose
> Browse.On the Browse tab, select the dashboard for your alert rule.
On the dashboard page, select the panel for your alert rule, click the panel name, and then click Edit.
On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.
NoteAlternatively, choose
in the left-side navigation pane of Grafana. On the Alert Overview page, click Create Alert Monitoring Rule.Configure the alert parameters as needed. For more information, see the official Grafana documentation.
In the Add annotation and labels section, enter a key and value for a custom label, for example,
grafana_alertname=Pod_disk_usage_hangzhou. This label is used to match the alert rule when you create a notification policy in Step 4.After you create the alert rule, click save in the upper-right corner.
When a Grafana alert is triggered, the event appears on the Alert Event History page in ITSM. For more information, see View alert event history.
NoteOn the Alert Event History page, you can set the Integration Name to ARMS_GRAFANA to filter events reported by Grafana alert rules.
Grafana 10.0.x
In the upper-left corner of Grafana, click
and then select Dashboard.On the Dashboard page, select the dashboard for your alert rule.
On the dashboard page, select the panel for your alert rule, click the icon in the upper-right corner of the panel, and then click Edit.
On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.
NoteAlternatively, click
in the left-side navigation pane of Grafana and select ITSM. On the Alert Overview page, click Create Alert Rule.Configure the alert parameters as needed. For more information, see the official Grafana documentation.
In the Add annotation and labels section, enter a key and value for a custom label, for example,
grafana_alertname=Pod_disk_usage_hangzhou. This label is used to match the alert rule when you create a notification policy in Step 4.After you create the alert rule, click Save rule in the upper-right corner.
When a Grafana alert is triggered, the event appears on the Alert Event History page in ITSM. For more information, see View alert event history.
NoteOn the Alert Event History page, you can set the Integration Name to ARMS_GRAFANA to filter events reported by Grafana alert rules.
Step 3: Create notification targets
Grafana 9.0.x
In the left-side navigation pane of Grafana, choose
> Notification Objects, and then create notification targets as needed. For more information, see Notification targets.
Contact or contact group: Used for notifications via email, phone, and SMS.
DingTalk, Lark, or WeCom: For handling alerts in team collaboration chats.
Webhook: Pushes alerts to third-party notification targets such as PagerDuty, Teams Incoming Webhook, or internal enterprise platforms.
On-call scheduling: Used for scenarios where team members handle alerts on rotation.
Grafana 10.0.x
In the upper-left corner of Grafana, click
and choose ITSM > Notification Objects. Then, create notification targets as needed. For more information, see Notification targets.
Contact or contact group: Used for notifications via email, phone, and SMS.
DingTalk, Lark, or WeCom: For handling alerts in team collaboration chats.
Webhook: Pushes alerts to third-party notification targets such as PagerDuty, Teams Incoming Webhook, or internal enterprise platforms.
On-call scheduling: Used for scenarios where team members handle alerts on rotation.
Step 4: Create a notification policy
Grafana 9.0.x
In the left-side navigation pane of Grafana, choose
> Notification Policies, and then click Create Notification Policy.On the Create Notification Policy page, enter a name for the notification policy in the upper-left corner.
In the Matching Rule step, configure rules to match alert events.
Select Data Source as the Data source.
Click + Add Condition and enter the key and value of the label that you configured in Step 2. For example,
grafana_alertname equals Pod_disk_usage_hangzhou.NoteThe second matcher filters for alerts from the Grafana integration. If you do not add a third matcher, this notification policy applies to all alerts from the current Grafana workspace.
In the list of conditions, the
Integration Typefield of Condition 1 is automatically set toGrafana(cannot be edited), theIntegration Namefield of Condition 2 is equal toARMS_GRAFANA, and thegrafana_alertnamefield of Condition 3 is equal to a specific alert name, such asPod_disk_usage_hangzhou. The three conditions are connected by a logical AND.Click Next.
Configure other parameters as needed, and then click Save. For more information, see Notification policies.
Grafana 10.0.x
In the upper-left corner of Grafana, click
and choose ITSM > Notification Policies. Then, click Create Notification Policy.On the Create Notification Policy page, enter a name for the notification policy in the upper-left corner.
In the Matching Rule step, configure rules to match alert events.
Select Data Source as the Data source.
Click + Add Condition and enter the key and value of the label that you configured in Step 2. For example,
grafana_alertname equals Pod_disk_usage_hangzhou.NoteThe second matcher filters for alerts from the Grafana integration. If you do not add a third matcher, this notification policy applies to all alerts from the current Grafana workspace.
In the list of conditions, the
Integration Typefield for Condition 1 is automatically set toGrafana(cannot be edited), theIntegration Namefield for Condition 2 equalsARMS_GRAFANA, and thegrafana_alertnamefield for Condition 3 equals a specific alert name, such asPod disk usage_hangzhou. The three conditions are connected by a logical AND.Click Next.
Configure other parameters as needed, and then click Save. For more information, see Notification policies.
Step 5: View alert notifications
When a notification policy is triggered, ITSM sends an alert notification to the channel specified in the matching notification target. Check the channel you configured, such as an email inbox, group chat, or webhook endpoint, to confirm that the notification was delivered.
Related documents
ARMS alert management supports sending notifications to targets by phone, SMS, email, DingTalk, WeCom, Lark, and webhooks, which allows you to handle alerts in various scenarios. For more information, see Alert management overview.