Application Real-Time Monitoring Service:Integrate Grafana alerts

Step 1: Create a Grafana integration

1. Log on to the ARMS console. In the left-side navigation pane, choose Alert Management > Integrations.

2. On the Alert Integration tab, click Grafana.

3. In the New Grafana Integration dialog box, enter an integration name and description, select the Grafana version, set the auto-resolve timeout for alert events, and click Save.

   Note

   Automatic recovery time after timeout: If an alert event is not triggered again within the specified period of time, the alert event is automatically cleared.

4. On the Alert Integration tab, find the new Grafana integration and get its integration endpoint.

   Click the Copy button next to the integration endpoint.

Step 2: Configure Grafana alerts

1. In the left-side navigation pane of Grafana, choose > Notification Channels.

2. On the Notification Channels tab, click New channel.

   Note

   If this is your first time creating a channel, click Add channel.

3. On the New Notification Channel page, perform the following steps.

   1. Enter a custom Grafana alert name in the Name field.

   2. Set Type to webhook.

   3. Set URL to the integration endpoint that is obtained in Step 1.

   4. In the Notification settings section, select Default, Include image, and Send reminders. If you select Send reminders, you must set the reporting frequency for the same alert. The reporting frequency cannot exceed the auto-resolve timeout that you set in Step 1.

      Note

      If you do not select Send reminders, the system sends only one notification for each alert.

   5. Click Test to verify the connection to ARMS Alert Management.

   6. Once the test succeeds, click Save.

(Optional) Step 3: Modify the integration

ARMS Alert Management provides default field mappings between Grafana and ARMS. You can also add or modify these mappings on the Edit Integration page as needed.

The following script provides a sample alert source:

{
    "@timestamp":"2021-09-09T12:25:09.556Z",
    "orgId":0,
    "host":"10.1.11.14",
    "ruleName":"Test notification",
    "ruleUrl":"http://localhost:8001/grafana/",
    "state":"alerting",
    "title":"[Alerting] Test notification",
    "panelId":1,
    "imageUrl":"https://grafana.com/assets/img/blog/mixed_styles.png",
    "evalMatches":[
        {
            "metric":"High value",
            "value":100,
            "tags":null
        },
        {
            "metric":"Higher Value",
            "value":200,
            "tags":null
        }
    ],
    "ruleId":0,
    "tags":{
    },
    "dashboardId":1,
    "message":"Someone is testing the alert notification within grafana.",
    "@version":"1"
}

1. On the Alert Integration tab, find the integration that you want to manage and click Edit in the Actions column.

2. In the Event Mapping section, click Send Test Data.

3. In the Send Test Data dialog box, enter the JSON content from your alert source, and click Send.

   Note

   • If the message Uploaded. No events are generated. Configure mappings based on the original data. appears, the fields of the alert source are not mapped to the fields of ARMS alert events. The data that is sent is displayed in the left-side box. This allows you to select the source fields when you configure mappings.

   • If the message Uploaded. appears, the alert content is reported to the Alert Event History page. For more information, see View alert event history.

   The text area of the Send Test Data dialog box contains sample notification data in JSON format, with fields such as @timestamp, orgId, host, ruleName, ruleUrl, state, title, and panelId. You can click Copy as cURL command to obtain the corresponding request.

4. In the Send Test Data dialog box, click Disable.

5. In the left part of the Event Mapping section, click the data records for which you want to configure mappings to view the details.

6. In the right pane of the Event Mapping section, configure the mappings between the source alert fields and the ARMS alert fields.

   1. Optional: In the Select Root Node section, specify whether to use batch processing.

      If the alert data contains an array, you can specify the array as a root node to process its elements in a batch.

      After you select Use Batch Processing, select the array node to be processed as the root node.

      Note

      You can select only one array for batch processing.

      For example, in the sample alert, the evalMatches array contains two value fields. If you select evalMatches as the root node, ARMS maps the two value fields under it to the same specified ARMS alert field.

      • If you select a root node, you can iteratively map the values of the $.evalMatches[*].value attribute for all elements in the array to the ARMS value field.

      • If you do not select a root node, you can map the attribute value of a specific element in the array, such as $.evalMatches[0].value or $.evalMatches[1].value, to the ARMS value field.

   2. In the Map Source Fields to Target Fields section, map the source fields to the target ARMS alert fields.

      Click the mapping icon to change the mapping method.

      • Direct: The source field is directly mapped to the corresponding ARMS alert field.

      • Series: Concatenates multiple source fields into a single field by using a specified separator, and maps this field to an ARMS alert field. You can use only special characters as separators.

        For example, you can use an underscore (_) to concatenate the $.host and $.title fields into $.host_$.title and then map it to the ARMS alertname field.

      • Condition: A field is mapped only if the alert field value meets a specified condition.

      • Mapping table: Configures the mapping between the severity levels of the alert source and ARMS. This mapping method is required only for the severity field.

      The following table describes the alert fields of ARMS.

      Alert field	Description
      alertname	The name of the custom alert.
      severity	The severity level of the alert. You must configure mappings for this field. The mapping method must be set to Direct.
      message	The description of the alert. The description is used as the content of the alert message. The description cannot exceed 15,000 characters in length.
      value	The sample value of a metric.
      imageUrl	The URL of the line chart that contains Grafana metrics. The URL is used to map the line chart.
      check	The check item of the alert. Examples: CPU, JVM, Application Crash, and Deployment.
      source	The source of the alert.
      class	The type of the object that triggers the alert event, for example, host.
      service	The source service of the alert. Example: Login Service.
      startat	The timestamp that represents the start time of the event.
      endat	The timestamp that represents the end time of the event.
      generatorUrl	The URL of the event details.

      On the Event Mapping page, click Send Test Data to verify data ingestion. The Latest 10 Data Entries section displays the received source data and its status. Expand a data entry to view its JSON structure, which includes fields such as @timestamp, orgId, host, and ruleName. In the right pane, you can map these source fields to the corresponding ARMS alert fields.

7. Configure field deduplication for alert events.

   To reduce duplicate data, the system uses related fields as de-duplication criteria. ARMS Alert Management allows you to preview the de-duplication grouping results of historical event data in the Event Mapping area. You can adjust the de-duplication fields as needed.

   Note

   Event deduplication applies only to unresolved events.

   1. In the Event Deduplication section of the Edit Integration page, select the fields for deduplication.

      When multiple events have the same values for the selected fields, they are merged into a single alert.

      For example, if you set the ARMS metric and ruleName fields as the deduplication key, events from the Grafana source that have the same metric and rule name are merged into a single event. Events with different values remain separate.

   2. Click Deduplication Test to preview the alert groupings after deduplication.

      Note

      The deduplication test runs on the last 10 test data entries uploaded in the left pane of the Event Mapping section.

      Select metric and ruleName as the deduplication fields and click Deduplication Test. The system groups events based on the selected fields. Events with the same metric and rule name are grouped together. The grouping results are displayed in a table that includes columns such as alertname, value, severity, startat, endat, and message.

8. After you complete the configuration, click Save.

View the details about an alert event

1. In the left-side navigation pane of the ARMS console, choose Alert Management > Alert Event History.

2. On the Alert Event History page, click the name of the alert event to view the event details. For more information, see View alert event history.

Manage the integration

In the left-side navigation pane, choose Alert Management > Integrations. On the Alert Integration tab, you can perform the following operations on the integrations that you created:

• View the details of an integration: Find the integration and then click the row. On the Integration Details page, view the integration details.

• Update key: To change the key for an integration, find the integration, choose More > Update Key in the Actions column, and then click Confirm in the dialog box that appears.

  Important

  After you update the key, you must also update the URL configured in Step 2 with the new integration endpoint.

• Modify an integration: Find the integration and then click Edit in the Actions column. On the Integration Details page, modify the integration information and then click Save.

• Enable or disable an integration: Find the integration and then click Disable or Enable in the Actions column.

• Delete an integration: Find the integration and then click Delete in the Actions column. In the message that appears, click OK.

• Add an event processing flow to an integration: Find the integration and click Add Event Processing Flow in the Actions column. For more information, see Event processing flow.

• Create a notification policy: Find the integration for which you want to create a notification policy, and click More in the Actions column. In the list that appears, click Create Notification Policy. For more information, see Notification policies.