Managed Service for OpenTelemetry:Send custom alert notifications with a webhook

Prerequisites

Before you begin, make sure you have:

• A Managed Service for OpenTelemetry instance with at least one alert rule configured

• A Lark account (or another messaging platform that provides incoming webhook URLs)

• Lark desktop application installed and running

Obtain a webhook URL from Lark

Set up a custom bot in a Lark group chat. The bot generates a webhook URL that Managed Service for OpenTelemetry uses to deliver alert notifications.

1. Open Lark and log on.

2. Click the + icon, then select New group to create a group for receiving alerts.

3. Click the group settings icon and open the BOTs tab.

4. On the BOTs tab, click Add Bot.

   

5. In the Add Bot dialog box, select Custom Bot.

   

6. Enter a Bot name and Description, then click Add.

   

7. Click Copy next to Webhook URL to copy the URL, then click Save. Keep this URL -- you need it in the next step.

   

Create a webhook in the OpenTelemetry console

1. Log on to the Managed Service for OpenTelemetry console.

2. In the left-side navigation pane, choose Alert Management > Notification Objects. On the Webhook Integration tab, click Create Webhook.

3. In the Create Webhook panel, configure the following parameters. For more information about notification templates, see Configure a notification template and a webhook template.

   Parameter	Required	Description	Constraints
   Webhook Name	Yes	A descriptive name for the webhook, for example, lark-alerts.	--
   Post or Get	Yes	The HTTP request method. For this example, select Post and paste the Lark webhook URL you copied earlier.	URL: 256 characters max
   Header or Param	No	Custom HTTP headers or query parameters. Click + Create to add entries. The default header is Content-Type: text/plain; charset=UTF-8. For this example, add two headers: Arms-Content-Type: json and Content-Type: application/json.	Each value: 200 characters max. Combined total of headers and params: 6 max
   Notification Template	No	The JSON payload sent when an alert fires. Only available when Post is selected. Use template variables to include dynamic alert data. For a ready-to-use Lark template, see the Lark notification template section below.	500 characters max
   Template to Clear Alerts	No	The JSON payload sent when an alert resolves. Only available when Post is selected. For a ready-to-use Lark template, see the Lark alert resolution template section below.	500 characters max

4. Click Send Test to verify the configuration. If the webhook is set up correctly, a test message appears in your Lark group.

   Note

   If you do not have a receiving endpoint ready yet, you can use a free HTTP debugging service such as Beeceptor or Webhook.site to capture and inspect the webhook payload before connecting to your production endpoint.

5. Click OK.

Link the webhook to a notification policy

After creating the webhook, associate it with a notification policy so that alerts are delivered to the webhook endpoint.

1. In the left-side navigation pane, choose Alert Management > Notification Objects.

2. Create or modify a notification policy. Set Notification Object to Universal Webhook and select the webhook you created.

3. Save the notification policy.

Template variables

The notification template supports the following variables. Wrap each variable in double braces, for example, {{ .commonLabels.alertname }}.

Conditional variables may be empty depending on the alert context. Use {{if ... }} blocks to include them only when they have values. See the Lark templates below for examples.

Lark notification template

Paste the following JSON into the Notification Template field. This template uses the Lark text message format with conditional blocks so that cluster and application names appear only when applicable.

{
  "msg_type": "text",
  "content": {
    "text": "Alert name: {{ .commonLabels.alertname }}\n{{if .commonLabels.clustername }}Cluster name: {{ .commonLabels.clustername }}\n{{ end }}{{if eq \"app\" .commonLabels._aliyun_arms_involvedObject_kind }}Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}\n{{ end }}Notification policy: {{ .dispatchRuleName }} \nAlert time: {{ .startTime }} \nNotification content: {{ for .alerts }} {{ .annotations.message }}\n {{ end }}"
  }
}

Lark alert resolution template

Paste the following JSON into the Template to Clear Alerts field. This template follows the same structure as the notification template but includes the resolution timestamp instead of the alert time.

{
  "msg_type": "text",
  "content": {
    "text": "Alert name: {{ .commonLabels.alertname }}\n{{if .commonLabels.clustername }}Cluster name: {{ .commonLabels.clustername }}\n{{ end }}{{if eq \"app\" .commonLabels._aliyun_arms_involvedObject_kind }}Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}\n{{ end }}Time when the alert was resolved: {{ .startTime }} \nNotification policy: {{ .dispatchRuleName }} \nNotification content: {{ for .alerts }} {{ .annotations.message }}\n {{ end }}"
  }
}

Generic notification template

For platforms that do not require a specific message envelope (such as Lark's msg_type wrapper), use this generic JSON structure as a starting point.

{
  "Alert name": "{{ .commonLabels.alertname }}{{if .commonLabels.clustername }}",
  "Cluster name": "{{ .commonLabels.clustername }} {{ end }}{{if eq \"app\" .commonLabels._aliyun_arms_involvedObject_kind }}",
  "Application name": "{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
  "Notification policy": "{{ .dispatchRuleName }}",
  "Alert time": "{{ .startTime }}",
  "Notification content": "{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}

Generic alert resolution template

For platforms that do not require a specific message envelope, use this generic JSON structure for alert resolution notifications. This template uses .endTime to indicate when the alert was resolved.

{
  "Alert name": "{{ .commonLabels.alertname }}{{if .commonLabels.clustername }}",
  "Cluster name": "{{ .commonLabels.clustername }} {{ end }}{{if eq \"app\" .commonLabels._aliyun_arms_involvedObject_kind }}",
  "Application name": "{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
  "Notification policy": "{{ .dispatchRuleName }}",
  "Time when the alert was resolved": "{{ .endTime }}",
  "Notification content": "{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}

Verify and troubleshoot

After you link the webhook to a notification policy, verify that alerts are delivered correctly:

• Trigger a test alert and confirm that the message appears in your Lark group (or your chosen endpoint).

• Check the payload format. If the message appears but the content is malformed, review your notification template for syntax errors in the conditional blocks or missing closing {{ end }} tags.

• Inspect raw requests. To debug payload issues without modifying your production bot, send test notifications to a service such as Beeceptor or Webhook.site and compare the raw JSON against your template.

• Handle timeouts. The webhook request times out after 5 seconds. If your endpoint consistently exceeds this limit, consider placing a lightweight proxy or queue (such as an API gateway or function compute endpoint) in front of the slower service.

What to do next

• To customize notification content further, see Configure a notification template and a webhook template.

• To set up notifications through other channels (such as email or SMS), configure additional notification objects within the same notification policy.