Configuring a webhook's notification template allows Application Real-Time Monitoring Service (ARMS) to send alert content in the open-source Alertmanager format.
Limitations
Note the following limitations when using a webhook for custom alert notifications:
-
The Content-Type must be application/json.
-
Cannot be used with the
$contentfield. If you use$alertmanager_content,$contentdoes not take effect. -
The webhook test feature in the ARMS console cannot test the replacement of
$alertmanager_content.
For setup instructions, see Customize alert notifications by using a webhook.
Alert payload format
ARMS sends alert payloads to your webhook in the open-source Alertmanager format, as shown in the following example. For more information, see the official Prometheus documentation.
{
"alerts": [
{
"annotations": {
"_aliyun_arms_alert_value": "15.521240234375",
"_aliyun_arms_alert_message": "null\nMessage ID: ac10c42a16124966960554200d2450-7996494\n",
"_aliyun_arms_alert_now_value": "15.521240234375",
"message": "Alert: Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} Memory usage exceeds 80%. Current value: {{ printf \"%.2f\" $value }}%",
"value": "15.521240234375",
"_aliyun_arms_alert_past_value": "15.521240234375"
},
"endsAt": "2021-02-22T07:27:15.404000000Z",
"fingerprint": "bec72890cc2c7b4a027e008df0cd1013",
"labels": {
"container": "kube-state-metrics",
"severity": "warning",
"_aliyun_arms_alert_level": "ERROR",
"instance": "10.0.80.186:10255",
"clustername": "klyz1688-kubernetes-1",
"_aliyun_arms_alert_type": "101",
"_aliyun_arms_integration_name": "test-integration-prometheus",
"alertname": "Container memory usage > 80%",
"_aliyun_arms_userid": "1131971649496228",
"_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
"pod_name": "kube-state-metrics-ccb59dbff-jljg4",
"_aliyun_arms_involvedObject_id": "cb36dcafb9b9340498fad2e1f40b9a254",
"_aliyun_arms_region_id": "cn-hangzhou",
"_aliyun_arms_involvedObject_kind": "cluster",
"_aliyun_arms_product_type": "PROMETHEUS",
"name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
"namespace": "arms-prom",
"_aliyun_arms_integration_id": "80",
"_aliyun_arms_involvedObject_type": "ManagedKubernetes",
"_aliyun_arms_alert_rule_id": "3927051"
},
"startsAt": "2021-02-22T07:18:15.578000000Z",
"status": "firing"
}
],
"commonAnnotations": {
"_aliyun_arms_alert_value": "15.521240234375",
"_aliyun_arms_alert_message": "null\nMessage ID: ac10c42a16124966960554200d2450-7996494\n",
"_aliyun_arms_alert_now_value": "15.521240234375",
"message": "Alert: Namespace: {{$labels.namespace}} / Pod: {{$labels.pod_name}} / Container: {{$labels.container}} Memory usage exceeds 80%. Current value: {{ printf \"%.2f\" $value }}%",
"value": "15.521240234375",
"_aliyun_arms_alert_past_value": "15.521240234375"
},
"commonLabels": {
"container": "kube-state-metrics",
"severity": "warning",
"_aliyun_arms_alert_level": "ERROR",
"instance": "10.0.80.186:10255",
"clustername": "klyz1688-kubernetes-1",
"_aliyun_arms_alert_type": "101",
"_aliyun_arms_integration_name": "test-integration-prometheus",
"alertname": "Container memory usage > 80%",
"_aliyun_arms_userid": "1131971649496228",
"_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
"pod_name": "kube-state-metrics-ccb59dbff-jljg4",
"_aliyun_arms_involvedObject_id": "cb36dcafb9b9340498fad2e1f40b9a254",
"_aliyun_arms_region_id": "cn-hangzhou",
"_aliyun_arms_involvedObject_kind": "cluster",
"_aliyun_arms_product_type": "PROMETHEUS",
"name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
"namespace": "arms-prom",
"_aliyun_arms_integration_id": "80",
"_aliyun_arms_involvedObject_type": "ManagedKubernetes",
"_aliyun_arms_alert_rule_id": "3927051"
},
"externalURL": "https://alerts.console.aliyun.com/#/alarm/alert/detail/2848",
"groupLabels": {
"alertname": "Container memory usage > 80%"
},
"receiver": "testjiubian",
"status": "firing"
}
Receive Alertmanager-formatted alerts via a webhook
-
Log on to the ARMS console. In the left-side navigation pane, choose .
-
Click the Webhook Integration tab, and then click Create Webhook.
-
In the dialog box, configure the parameters. For more information, see Step 2: Create a webhook contact. In the Create Webhook dialog box, enter a Webhook Name such as
my-webhook, select Post for the Method, and enter the destination URL. Set the Content-Type in the Header toapplication/json. Enter$alertmanager_contentfor both the Notification Template and Resolved Template. Then, click Send Test to verify the configuration.