All Products
Search
Document Center

:Configure webhook URLs for alert ingestion

Last Updated:Jul 06, 2023

You can configure webhook URLs for alert ingestion. To do this, you must create an alert ingestion service and an alert ingestion application in the Log Service console. Then, Log Service can receive alerts from external monitoring systems.

Step 1: Create an alert ingestion service and an alert ingestion application

  1. Log on to the Log Service console.
  2. Go to the Alert Ingestion page.

    1. In the Projects section, click a project.

    2. In the left-side navigation pane, click Alerts.

    3. Choose Alert Management > Alert Ingestion.

  3. Create an alert ingestion service.

    1. Click Create.

    2. In the Add Service dialog box, specify the ID and name of the service, and click Save.

  4. Create an alert ingestion application.

    1. Click Application in the Actions column of the service.

    2. In the Application Management dialog box, click Create.

    3. In the Add Application dialog box, set the parameters and click Save. The following table describes the parameters.

      Parameter

      Description

      ID

      The ID of the alert ingestion application.

      Name

      The name of the alert ingestion application.

      Protocol Verification

      Protocol

      The format of alerts. Valid values:

      • AlertManager: receives Alertmanager alerts.

      • Prometheus: receives Prometheus alerts.

      • Zabbix: receives Zabbix alerts.

      • Grafana: receives Grafana alerts.

      • Datadog: receives Datadog alerts.

      • CloudMonitor: receives CloudMonitor alerts.

      • Loki: receives Loki alerts.

      Alert Policy

      The alert policy that is used to merge, silence, and suppress alerts.

      • If you select Simple Mode or Standard Mode, you do not need to configure an alert policy. In this case, Log Service uses the built-in alert policy sls.builtin.dynamic to manage alerts by default.
      • If you select Advanced Mode, you can select a built-in or custom alert policy to manage alerts. For more information about how to create an alert policy, see Create an alert policy.

      Action Policy

      The action policy that is used to manage alert notification methods and the frequency at which alert notifications are sent.

      • If you set the Alert Policy parameter to Simple Mode, you need to only configure an action group for this parameter.

        After you configure an action group, Log Service automatically creates an action policy whose name is in the Rule name-Action policy format. Alert notifications are sent based on the action policy for all alerts that are triggered based on the alert monitoring rule. For more information about how to configure alert notification methods, see Notification methods.

        Important You can modify an action policy on the Action Policy tab. For more information, see Create an action policy. If you add conditions when you modify an action policy, the value of the Alert Policy parameter is automatically changed to Standard Mode.
      • If you set the Alert Policy parameter to Standard Mode or Advanced Mode, you can select a built-in or custom action policy to send alert notifications. For more information about how to create an action policy, see Create an action policy.

        If you set the Alert Policy parameter to Advanced Mode, you can turn on or turn off Custom Action Policy. For more information, see Dynamic action policy mechanism.

      Cycle

      If duplicate alerts are triggered in the specified cycle, the action policy that you select is executed only once, and only one alert notification is sent.

      Whitelist

      If you turn on the Whitelist switch, you must specify an AccessKey ID. Only the webhook URL that contains a valid AccessKey ID can be used to access alerts.

      For example, if you specify an AccessKey ID as AEDC****ERT and set the {ACCESS_KEY_ID} variable of the path_prefix parameter in Prometheus to AEDC****ERT, the related alerts can be ingested into the alerting system of Log Service.

      Important

      You must grant the AliyunLogPutOpenEventPolicy permission to the RAM user to which the AccessKey pair belongs. For more information, see Appendix: Obtain an AccessKey ID.

      Filter

      Filter by Keyword

      The alert ingestion system ingests only the alerts that contain one or more specified keywords.

      Enrichment

      Add Label

      Add labels to alerts. Labels are formatted in key-value pairs. For example, you can set the key of a label to Environment and set the value to Staging environment. For more information, see Labels.

      Add Annotation

      Add annotations to alerts. Annotations are formatted in key-value pairs. For example, you can set the key of an annotation to title and set the value to Prometheus alert. For more information, see Annotations.

      If you set the Protocol parameter to CloudMonitor, Log Service automatically adds the __user_language__ annotation based on the language of the console. The valid values of the annotation are en and cn. The value en indicates English and the value cn indicates Chinese.

      Important

      When you set the Add Annotation parameter, you cannot reference alert template variables.

Step 2: Obtain webhook URLs

When you configure parameters in Prometheus or Grafana, you must use the related webhook URL.

  1. In the Application Management dialog box, click Webhook URLs of the alert ingestion application.

  2. In the Webhook URLs dialog box, select the region where alerts are ingested.

    Note

    If your Prometheus or Grafana is deployed on an Elastic Compute Service (ECS) instance, we recommend that you select the region where the ECS instance resides and use an internal endpoint that can be accessed over a LAN or a virtual private cloud (VPC). You can also use the public endpoint that can be accessed over the Internet in a region.

  3. Move the pointer over the blank area next to the text box and copy the full URL, domain name, or subpath of the webhook URL.

    Replace the {ACCESS_KEY_ID} variable in a webhook URL with the AccessKey ID of the AccessKey pair that you want to use. For information about how to obtain an AccessKey ID, see Appendix: Obtain an AccessKey ID.

    Webhook URLs

Appendix: Obtain an AccessKey ID

To ensure the security of your Alibaba Cloud account, we recommend that you use a RAM user rather than an Alibaba Cloud account to ingest alerts. Before you can use a RAM user to ingest alerts, you must grant the AliyunLogPutOpenEventPolicy permission to the RAM user. To obtain an AccessKey ID, perform the following steps:

  1. Create a RAM user. For more information, see Create a RAM user.

  2. Grant the AliyunLogPutOpenEventPolicy permission to the RAM user. For more information, see Grant permissions to the RAM user.

  3. Create an AccessKey pair that includes an AccessKey ID for the RAM user. For more information, see Create an AccessKey pair.

What to do next