If the alert rules of the specified notification policy are triggered, ARMS sends alert notifications by DingTalk, Lark, WeCom, text message, phone call, email, or webhook. ARMS provide preset notification templates for each notification method. If a preset notification template cannot meet your requirements, you can modify the notification template. This topic describes the Go template syntax, preset notification templates, and variables in the templates.
Go template syntax
ARMS uses the Go template syntax to render notification templates. When the Go template syntax is used to render a template, the template is filled based on the context.
{
"alerts": [
{
"annotations": {
"message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.",
"value": "15.521240234375"
},
"endsAt": "2023-02-22T07:27:15.404000000Z",
"fingerprint": "bec72890cc2c7b4a027e008df0cd1013",
"labels": {
"container": "kube-state-metrics",
"severity": "warning",
"instance": "10.0.80.186:10255",
"clustername": "klyz1688-kubernetes-1",
"alertname": "The memory usage of the container has exceeded 80%.",
"_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
"pod_name": "kube-state-metrics-ccb59dbff-jljg4",
"_aliyun_arms_involvedObject_kind": "cluster",
"name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
"namespace": "arms-prom"
},
"startsAt": "2023-02-22T07:18:15.578000000Z",
"status": "firing"
}
],
"commonAnnotations": {
"message": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.",
"value": "15.521240234375"
},
"commonLabels": {
"container": "kube-state-metrics",
"severity": "warning",
"instance": "10.0.80.186:10255",
"clustername": "klyz1688-kubernetes-1",
"alertname": "The memory usage of the container has exceeded 80%.",
"_aliyun_arms_involvedObject_name": "klyz1688-kubernetes-1",
"pod_name": "kube-state-metrics-ccb59dbff-jljg4",
"_aliyun_arms_involvedObject_kind": "cluster",
"name": "k8s_kube-state-metrics_kube-state-metrics-ccb59dbff-jljg4_arms-prom_359508f3-7e76-4740-b915-41ea48849641_0",
"namespace": "arms-prom"
},
"groupLabels": {
"alertname": "The memory usage of the container has exceeded 80%."
},
"status": "firing",
"startTime":"2023-02-22 07:18:15",
"endTime":"The time when the alert was cleared",
"level":"error",
"dispatchRuleName":"Notification policy name"
"alarmId":"123456"
}
Dot (.
)
A dot is used to indicate the context in the current scope.
{{ . }}
Output: The context in the preceding sample code is returned.
level
field in the context: {{ .level }}
error
commonLabels.alertname
field in the context: {{ .commonLabels.alertname }}
The memory usage of the container has exceeded 80%.
Variables
The data passed to a Go template can be stored in the variables of the template and accessed. For example, if you use the $context
variable in the { {$context := .}}
action and save passed data in the variable, you can use the {{$context}}
action to access the variable.
$alertname
variable. Then, the value of the $alertname
variable is returned: {{ $alertname := .commonLabels.alertname }} {{$alertname}}
The memory usage of the container has exceeded 80%.
alerts
list to the $alert0
variable. Then, the alertname
of the first alert (element) is returned: {{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}
The memory usage of the container has exceeded 80%.
if and else actions
You can execute an if
action to check data. If the if action cannot meet your business requirements, you can execute an else
action. For null values, 0, nil pointer, empty strings, or zero-length strings, false
is returned.
severity
tag in an alert event is warning
, a P3 alert
is generated: {{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}
P3 alert
severity
tag in an alert event is critical
, a P1 alert
is generated. Otherwise, a P2 alert
is generated: {{if eq "critical" .commonLabels.severity }} P1 alert {{ else }} P2 alert {{ end }}
P2 alert
for action
The for
action is used to repeatedly execute code.
alerts
list and obtain the alertname
of each alert: {{ for .alerts}} {{.labels.alertname}} \n {{end}}
The memory usage of the container has exceeded 80%.
range action
The range
action is used to traverse the specified content.
{{ $ran := range 0 10 }} {{ for $ran }} {{.}} {{end}}
0 1 2 3 4 5 6 7 8 9
index function
The index
function is used to obtain the elements in an array.
alerts
list and assign a value to the $alert0
variable: {{ $alert0 := index .alerts 0 }} {{$alert0.labels.alertname}}
The memory usage of the container has exceeded 80%.
Comparison functions
Common comparison functions:
eq
: equal tone
: not equal tolt
: less thanle
: less than or equal togt
: greater thange
: greater than or equal to
severity
tag in the commonLabels
section is equal to warning
, a P3 alert
is generated: {{if eq "warning" .commonLabels.severity }} P3 alert {{ end }}
P3 alert
Logical functions and, or, and not
severity
tag in the commonLabels
section is equal to warning
or warnning
, a P3 alert
is generated: {{if eq "warning" .commonLabels.severity or eq "warning" .commonLabels.severity }} P3 alert {{ end }}
P3 alert
Common built-in functions
printf: formats the output data.
value
of the first alert in the alerts
list is rounded to two decimal places and returned: {{ $alert0 := index .alerts 0 }} {{ printf "%.2f" $alert0.annotations.value }}
15.52
humanizePercentage: formats percentages.
{{ "0.95332123123124" | humanizePercentage }}
95.33%
humanizeDate: formats a date by converting a millisecond timestamp to a date. The default date format is yyyy-MM-dd HH:mm:ss
.
{{ "1671074058234" | humanizeDate }}
2022-12-15 11:14:18
len: obtains the length of an array.
alerts
list: {{ len .alerts }}
1
Preset notification templates
Email notification template
Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert triggered at: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event) {{ end }} {{ for .alerts }} {{.annotations.message}} {{ if .generatorURL }} <a href="{{.generatorURL}}" >Details URL</a> {{end}} {{end}}
Template content | Description |
---|---|
| The name of the alert. By default, notification policies are grouped by alertname . Therefore, all events related to an alert have the same alert name. |
| If the clustername label exists in the event, the value of the clustername label is rendered as the cluster name field. |
| If the type of the object that triggers the alert is app , the name of the corresponding application is returned. |
| The name of the notification policy that matches the alert. |
| The time when the alert was triggered. |
| The alert event list is traversed to render Note The HTML syntax <a></a> is used in the preset template to render URLs.
|
Alert name: The memory usage of the container has exceeded 80%.
Cluster name: klyz1688-kubernetes-1
Notification policy: Notification policy name
Alert triggered at: 2023-02-22 07:18:15
Alert content: Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%.
Text message and phone call notification templates
Text message and phone call notification templates have the same content as the email template except that they do not contain the URL of alert details.
{{ .level }} alert occurred
Alert name: {{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}
Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
Notification policy: {{ .dispatchRuleName }}
Alert triggered at: {{ .startTime }}
Alert content: {{ if .newIncidentIn }} (New event)
{{end}} {{ for .alerts }} {{ .annotations.message }} {{ end }}
Chatbot notification template
{{ if .newIncidentIn }} (New event) {{end}}
{{if .commonLabels.clustername }}
> Cluster name: {{ .commonLabels.clustername }}
{{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}
> Application name: {{ .commonLabels._aliyun_arms_involvedObject_name }}
{{ end }}
{{ for .alerts }} > {{ .annotations.message }} {{if .generatorURL }} [Details URL]({{.generatorURL}}) {{end}}{{if .annotations._aliyun_arms_insights_analyze_link }}[<font color='#ff0000'>Root cause</font>]({{ .annotations._aliyun_arms_insights_analyze_link}}){{ end }}{{if eq "1" .labels._aliyun_arms_denoise_code }} (Important: {{.labels._aliyun_arms_important_reason }}) {{end}}
{{end}}
- Reference:
>
- Font color:
<font color='#ff0000'>Root cause</font>
Note Only DingTalk and WeCom allow you to change font colors. Lark does not allow you to change font colors. - Line break:
- DingTalk line breaks: Press Enter twice in the template.
- Lark line breaks: Press Enter only once in the template.
- WeCom line breaks: Press Enter only once in the template.
Webhook template
{
"Alert name":"{{ .commonLabels.alertname }}
{{if .commonLabels.clustername }}", "Cluster name":"{{ .commonLabels.clustername }} {{ end }}
{{if eq "app" .commonLabels._aliyun_arms_involvedObject_kind }}", "Application name":"{{ .commonLabels._aliyun_arms_involvedObject_name }} {{ end }}",
"Notification Policy":"{{ .dispatchRuleName }}",
"Alert triggered at":"{{ .startTime }}",
"Alert content":"{{ for .alerts }} {{ .annotations.message }} {{ end }}"
}
{
"Alert name": "The memory usage of the container has exceeded 80%.",
"Cluster name": "klyz1688-kubernetes-1",
"Notification policy": "Notification policy name",
"Alert triggered at":"{{ .startTime }}",
"Alert content": "Namespace: arms-prom / Pod: kube-state-metrics-ccb59dbff-jljg4 / Container: kube-state-metrics. The memory usage of the container has exceeded 80%. Current value: 15.52%."
}
Variables
The following table lists the variables that can be used in a notification template.
Variable | Description |
---|---|
alarmId | The ID of the original alert. |
alerts | The original alert event. The data type is List<Alert>. |
alerts.annotations | The annotations of the original alert. The data type is Map<String, String>. |
alert.endsAt | The time when the original alert was cleared. The time when the alert was cleared is the time when the alert was triggered plus the period of time consumed to clear the alert. |
alert.startsAt | The time when the original alert was triggered. |
alert.fingerprint | The fingerprint of the original alert. Alerts of one label group have the same fingerprint. |
alert.labels | The labels of the original alert. |
alert.status | The status of the original alert. Valid values:
|
commonAnnotations | The same annotations for a group of original alerts. |
commonLabels | The same labels for a group of original alerts. |
groupLabels | The group labels. Alerts are grouped based on the labels that are specified in the group condition of the notification policy. |
status | The status of a group of alerts. Valid values:
|
startTime | The time when the alert group was created. |
endTime | The time when the group of alerts was cleared, which is the time when the last original alert in the group was cleared. |
level | The severity level of the alert. Valid values:
|
dispatchRuleName | The name of the notification policy for the alert group. |