When Grafana dashboards detect anomalies, alert events often stay siloed within Grafana and lack flexible notification routing. By connecting Grafana native alerting to Information Technology Service Management (ITSM) in Application Real-Time Monitoring Service (ARMS), alert events flow into ARMS -- an out-of-the-box intelligent alerting service that receives alerts from various sources, including Prometheus -- for automatic deduplication, grouping, and noise reduction. Alerts are then routed to different notification targets based on their label content through phone calls, SMS, email, DingTalk, WeCom, Lark, or webhooks.
How it works
Grafana alert rule fires
|
v
Webhook contact point (or notification channel) sends the event
|
v
ARMS ITSM endpoint receives the event
|
v
ITSM deduplicates, groups, and reduces noise
|
v
ARMS notification policy routes the alert to contactsThe setup involves three tasks:
Connect Grafana to ITSM -- Create a webhook-based notification channel (Grafana 8.2.x and earlier) or contact point (later than 8.2.x) that sends alert events to the ARMS ITSM endpoint.
Create alert rules -- Define alert rules on your Grafana dashboard panels and assign them to the ITSM notification channel or contact point.
Configure notification routing in ARMS -- Set up an ARMS notification policy to route incoming alert events to your contacts.
Before you begin
Before you begin, make sure that you have:
Alert parameters unified_alerting and alerting both set to
trueon the Parameter Settings page of your Grafana workspaceAn ARMS integration endpoint URL (create a Grafana integration in ARMS and copy the webhook endpoint URL; see Integrate Grafana alerts)
Your Grafana version identified -- the configuration steps differ between Grafana 8.2.x and earlier (notification channels) and later than 8.2.x (contact points). Check the version on your workspace details page
Grafana 8.2.x and earlier
Step 1: Log on to Grafana
Log on to the Managed Service for Grafana console. In the left-side navigation pane, click Workspace Management.
On the Workspace Management page, find your workspace and click the URL in the URL column to open Grafana.
Log on with the Grafana administrator account and the password you set when creating the workspace. Alternatively, click Sign in with Alibaba Cloud to use your current Alibaba Cloud account.
Step 2: Create a notification channel
If a notification channel named ARMS_GRAFANA already exists, skip this step.
In the left-side navigation pane, choose
> Notification channels.On the Notification channels tab, click New channel and configure the following parameters:
Configuration Item
Description
Name
Enter
ARMS_GRAFANA.Type
Select webhook.
Url
Enter the ARMS integration endpoint URL you obtained in Before you begin. See Integrate Grafana alerts.
Step 3: Create an alert rule
In the left-side navigation pane, choose
> Manage.On the Manage tab, click the dashboard for which to create an alert rule.
In the dashboard, click
next to a panel name and click Edit.On the Edit Panel page, click Alert, and then click Create Alert.
Configure alert parameters based on your requirements. See Grafana documentation or Configure Grafana native alerting.
In the Notifications section, click the
icon to the right of Send to, and then select the ARMS_GRAFANA alert notification channel.
Click Save in the upper-right corner of the Edit Panel page.
When this alert fires, the event appears on the page in the ARMS console. See View historical alert events.
Grafana versions later than 8.2.x
The alerting UI differs slightly between Grafana 9.0.x and 10.0.x. The following steps use Grafana 9.0.x as an example.
Step 1: Disable the ARMS alerting plug-in
Log on to the Managed Service for Grafana console. In the left-side navigation pane, click Workspace Management.
On the Workspace Management page, click the ID of your workspace.
In the left-side navigation pane, click Alert Management. On the page that appears, disable the ARMS alerting plug-in and enable Grafana native alerting.
Step 2: Create an alert rule
On the Workspace Management page, find your workspace and click the URL in the URL column to open Grafana.
NoteLog on with the Grafana administrator account and the password you set when creating the workspace. Alternatively, click Sign in with Alibaba Cloud to use your current Alibaba Cloud account.
In the left-side navigation pane, choose
> Browse.On the Browse tab, click the dashboard for which to create an alert rule.
Hover over the panel, click the arrow next to the panel name, and select Edit from the drop-down list.
On the Edit Panel page, click the Alert tab, and then click Create alert rule from this panel.
Configure alert parameters based on your requirements. See Grafana documentation and Configure Grafana native alerting.
Click Save in the upper-right corner.
When this alert fires, the event appears on the page in the ARMS console. See View historical alert events.
Step 3: Create a contact point
If a contact point named ARMS_GRAFANA already exists, skip this step.
In the left-side navigation pane, choose
> Contact points.On the Contact points tab, click New contact Point or Add contact point and configure the following parameters:
Configuration Item
Description
Name
Enter
ARMS_GRAFANA.Contact point type
Select Webhook.
Addresses
Enter the ARMS integration endpoint URL you obtained in Before you begin. See Integrate Grafana alerts.
Step 4: Create a notification policy in Grafana
Create a notification policy to route alert rules to the contact point.
In the left-side navigation pane, choose
> Notification policies.On the Notification policies tab, click New specific policy/New nested policy and configure the following parameters:
Configuration Item
Description
Matching labels
Select labels to specify which alert rules apply.
Contact point
Select the
ARMS_GRAFANAcontact point.Configure alert grouping and silencing options as needed.
Configure the ARMS notification policy
Managed Service for Grafana creates a notification policy by default. To receive alert notifications from Grafana, you must configure the notification policy. After Grafana sends alert events to the ITSM endpoint, configure a notification policy in ARMS to specify how alerts reach your team. This step applies to all Grafana versions.
Log on to the ARMS console. In the left-side navigation pane, choose .
On the Notification Policy page, find the policy named
ARMS_GRAFANAand click Edit in the Actions column. If no such policy exists, create one. See Create a notification policy.In the Edit Notification Policy panel, click the Notification Objects tab. On the Notification Objects tab, complete the following configuration. For more information, see Create and manage a notification policy.
In the When an alert is triggered section, click Add Notification Object and select a notification endpoint and method.
Configuration Item
Description
Contacts
Select a notification endpoint: contacts, contact groups, schedules, DingTalk, Lark, WeCom, or webhooks. To create endpoints, see Overview.
Notification Methods
Select one or more methods: phone calls, SMS, or email.
NoteTo receive phone call notifications, verify the contact's phone number first. See Contacts.
In the Notification Template section, configure the notification formats for Email, Text Messages, Phone, and DingTalk/Lark/WeCom.
Click Save.
Verify the integration
After completing the setup, verify the end-to-end flow:
Trigger a test alert in Grafana, or wait for an existing alert rule to fire.
Check the page in the ARMS console to confirm the event was received.
Verify that your contacts received notifications through the configured channels.
Related topics
Enable or disable Grafana alerts on the Alert rules tab. See Grafana documentation.