Application Real-Time Monitoring Service:Integrate Grafana alerts

Step 1: Create a Grafana integration

1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Integrations.
2. On the Alert Integration tab, click Grafana.
3. In the Create Grafana Integration dialog box, enter the integration name, set the automatic recovery time of alert events, and then click Save.
   Note If an alert event is not triggered again within the specified period of time, the alert event is automatically cleared.
4. On the Alert Integration tab, view the created Grafana integration and obtain the integration endpoint.
   

Step 2: Configure Grafana alerts

1. In the left-side navigation pane of the Grafana page, choose Alerting > Notification channels.
   
2. On the Notification channels tab, click New channel.
   Note If you are creating a channel for the first time, click Add channel.

   
3. On the New notification channel page, perform the following operations:
   1. Enter a custom Grafana alert name in the Name field.
   2. Set Type to webhook.
   3. Set URL to the integration endpoint that is obtained in Step 1.
   4. In the Notification settings section, select Default, Include image, and Send reminders. After you select Send reminders, you need to set the reporting frequency of alerts. The reporting frequency cannot exceed the automatic recovery time of alert events set in Step 1.
      Note If you do not select Send reminders, each alert is reported only once.
   5. Click Test to check whether Grafana is connected to Alert Management.
   6. After the connection test is passed, click OK.
   

(Optional) Step 3: Modify the integration

Alert Management provides field mappings between Grafana alert sources and ARMS alert events. You can also add or modify the mappings between fields on the Edit Integration page.

The following script provides a sample alert source:

{
    "@timestamp":"2021-09-09T12:25:09.556Z",
    "orgId":0,
    "host":"10.1.11.14",
    "ruleName":"Test notification",
    "ruleUrl":"http://localhost:8001/grafana/",
    "state":"alerting",
    "title":"[Alerting] Test notification",
    "panelId":1,
    "imageUrl":"https://grafana.com/assets/img/blog/mixed_styles.png",
    "evalMatches":[
        {
            "metric":"High value",
            "value":100,
            "tags":null
        },
        {
            "metric":"Higher Value",
            "value":200,
            "tags":null
        }
    ],
    "ruleId":0,
    "tags":{

    },
    "dashboardId":1,
    "message":"Someone is testing the alert notification within grafana.",
    "@version":"1"
}

1. On the Alert Integration tab, find the integration that you want to manage and click Edit in the Actions column.
2. In the Event Mapping section, click Send Test Data.
3. In the Send Test Data dialog box, enter the alert content of a third-party alert source in the JSON format and click Send.
   Note

   • If the message Uploaded. No events are generated. Configure mappings based on the original data. appears, the fields of the alert source are not mapped to the fields of ARMS alert events. The data that is sent is displayed in the left-side box. This allows you to select the source fields when you configure mappings.
   • If the message Uploaded. appears, the alert content is reported to the Alert Event History page. For more information, see View historical alert events.

   
4. In the Send Test Data dialog box, click Disable.
5. In the left part of the Event Mapping section, click the data records for which you want to configure mappings to view the details.
6. In the right part of the Event Mapping section, configure field mappings between the alert source and ARMS.
   1. Optional:In the Select Root Node section, specify whether to enable batch processing.
      If an array node exists in the alert data, you can specify the array node as the root node. The data that belongs to the root node is processed in a batch.

      After you select Use Batch Processing, select the array node to be processed as the root node.

      Note If multiple array nodes exist in the alert data, you can select only one of the array nodes for batch processing.

      For example, an evalMatches array node contains two value fields. If you select the evalMatches array node as the root node, the two value fields under the node are mapped to the specified alert fields of ARMS as the same field.

      • If you select the array node as the root node, you can set the $.evalMatches[*].value attribute values of all elements in the array to be iteratively mapped to the value fields of ARMS.
      • If you do not select the array node as the root node, you can iteratively map the $.evalMatches[0].value or $.evalMatches[1].value attribute values of specific subscript elements in the array to the value fields of ARMS.
   2. In the Map Source Fields to Target Fields section, map the fields of the alert source to the fields of ARMS alerts.
      Click the Map icon to change the field mapping method.

      • Direct: The specified field of the alert source is mapped to the specified alert field of ARMS.
      • Series: You can use delimiters to concatenate multiple fields of the alert source into one field, and then map this field to the specified alert field of ARMS. Only special characters can be used as delimiters.

        For example, you can use an underscore (_) to concatenate the $.host and $.title fields into $.host_$.title and then map the field to the alertname field of ARMS.

      • Condition: The specified fields of the alert source are mapped to the alert fields of ARMS only if the field values meet the specified conditions.
      • Mapping table: You must configure a mapping table that maps the severity levels of the alert source to the severity levels of ARMS alerts. You need to configure a mapping table only for the severity field.

      The following table describes the alert fields of ARMS.

      Alert field	Description
      alertname	The name of the custom alert.
      severity	The alert level. You must configure mappings for this field. The mapping method must be set to Direct.
      message	The description of the alert. The description is used as the content of the alert message. The description cannot exceed 15,000 characters in length.
      value	The sample value of a metric.
      imageUrl	The URL of the line chart that contains Grafana metrics. The URL is used to map the line chart.
      check	The check item of the alert. Examples: CPU, JVM, Application Crash, and Deployment.
      source	The source of the alert.
      class	The type of the object that triggers the alert event, for example, host.
      service	The source service of the alert. Example: Login Service.
      startat	The timestamp that represents the start time of the event.
      endat	The timestamp that represents the end time of the event.
      generatorUrl	The URL of the event details.

      
7. Configure field deduplication for alert events.
   To reduce duplicate data, the system uses relevant fields as the basis for deduplication. Alert Management allows you to preview the deduplication grouping results of historical event data that is displayed in the Event Mapping section. You can change the fields to be deduplicated.

   Note You can configure deduplication only for events that are not cleared.
   1. In the Event Deduplication section on the Integration Details page, select the fields that are used for deduplication.
      If multiple events have the same value for a specified field, the events are merged into one alert notification.

      For example, if you specify the metric and ruleName fields of ARMS as the deduplication key, events with the same metric and ruleName values in the Grafana alert source are merged into one event. Events with different metric and ruleName values remain unchanged.

   2. Click Deduplication Test to preview the alert group after deduplication.
      Note The deduplication test takes effect only on the latest 10 data records that are uploaded in the left part of the Event Mapping section.

      
8. Click Save.

View the details about an alert event

1. In the left-side navigation pane of the ARMS console, choose Alert Management > Alert Event History.
2. On the Alert Event History page, click the name of the alert event to view the event details. For more information, see View historical alert events.

Manage the integration

In the left-side navigation pane, choose Alert Management > Integrations. On the Alert Integration tab, you can perform the following operations on the integrations that you created:

• View the details of an integration: Find the integration and then click the row. On the Integration Details page, view the integration details.
• Update a key: Find the Grafana integration and then choose More > Update Key in the Actions column. In the message that appears, click OK.
  Important After you update the key of the Grafana integration, change the URL in Step 2 to the new integration endpoint.
• Modify an integration: Find the integration and then click Edit in the Actions column. On the Integration Details page, modify the integration information and then click Save.
• Enable or disable an integration: Find the integration and then click Disable or Enable in the Actions column.
• Delete an integration: Find the integration and then click Delete in the Actions column. In the message that appears, click OK.
• Add an event processing flow to an integration: Find the integration and click Add Event Processing Flow in the Actions column. For more information, see Work with event processing flows.
• Create a notification policy: Find the integration for which you want to create a notification policy, and click More in the Actions column. In the list that appears, click Create Notification Policy. For more information, see Create and manage a notification policy.