You can use the labels, annotations, fire_results, and results variables in an alert template to reference log content in alert notifications.

Log Service can display the details of an alert in alert notifications. This way, you can view alert details when you receive an alert notification. To include alert details in an alert notification, you must configure the required parameters when you create an alert monitoring rule and add the labels, annotations, fire_results, and results variables to the alert template that is used. For more information, see Variables in new alert templates.
  • labels: the labels of an alert
  • annotations: the annotations of an alert
  • fire_results: the data records for which an alert is triggered
  • results: the query parameters and intermediate results

For example, if you monitor access to NGINX, you can use the following query statement to obtain information such as the status code that is returned and indicates an access error, the IP address of the machine on which the error occurred, and the number of times that the error occurred. Then, you can create an alert monitoring rule based on the information and reference the information in the alert template. If a status code is 400 or a larger number, Log Service sends you an alert notification that includes the status code, IP address of the machine on which the error occurred, and the number of times that the error occurred.

  • Query statement
    status >=400 | SELECT status, __source__ AS ip, count(*) AS cnt GROUP BY status,ip
  • Query resultsQuery results

Use the labels variable

You can use the labels variable in an alert template to reference log content in alert notifications.

  • Alert monitoring rule
    You can add labels or configure group evaluation when you create an alert monitoring rule. After an alert is triggered, information about labels is generated and included in the alert message.
    • Configure group evaluation: After you configure group evaluation, the fields that you select for group evaluation are automatically added to the information about labels. For more information, see Use the group evaluation feature.
      Notice When you select fields for group evaluation, we recommend that you select a field that can identify a monitored entity and whose values can be enumerated. If a field cannot identify a monitored entity, we recommend that you do not select the field. If you do not select a suitable field, an excessive number of groups are generated. If the trigger condition in a group is met, an alert is triggered. If a large number of groups exist, alert storms may occur. In this case, you may miss out on important alert information.

      For example, if you select the status field for group evaluation, different status codes trigger different alerts when error log are recorded.

    • Add labels: Labels are identifying attributes. For more information, see Labels.

    For example, if you want to add information such as the log source, environment, and status code to alert messages, you can add the app:nginx and env:prod labels and select the status field for group evaluation.

    labels
  • Alert message

    After an alert is triggered, Log Service generates an alert message that contains the following information:

    {
        "labels": {
            "app": "nginx",
            "env": "prod",
            "status": "404"
        }
    }
  • Alert template

    You can add the labels variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

    Alert template

Use the annotations variable

You can use the annotations variable in an alert template to reference log content in alert notifications.

  • Alert monitoring rule
    You can manually add information about annotations or turn on Auto-Add Annotations to automatically add information about annotations when you create an alert monitoring rule. After an alert is triggered, information about annotations is generated and included in the alert message. For more information, see Annotations.
    Note Log Service provides two annotations by default.

    For example, if you want to add information such as the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred to alert messages, you can add annotations such as status:${status}, ip:${ip}, and cnt:${cnt}.

    • Manually add information about annotationsAlert monitoring rule
    • Turn on Auto-Add Annotations to automatically add information about annotationsAlert monitoring rule
  • Alert message

    After an alert is triggered, Log Service generates an alert message that contains the following information:

    {
        "annotations": {
            "title": "Alert is triggered by an NGINX access error",
            "desc": "The error that is indicated by the status code 400 occurred 15 times"
            "status": "400"
            "ip": "127.0.0.1"
            "cnt": "15"
            "__count__": "1"
        }
    }
  • Alert template

    You can add the annotations variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

    Alert template

Use the fire_results variable

You can use the fire_results variable in an alert template to reference log content in alert notifications.

  • Alert monitoring rule

    Log Service generates alert messages based on the alert monitoring rule that you create.

    Alert monitoring rule
  • Alert message

    After an alert is triggered, Log Service generates an alert message that contains the following information. The information includes the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred.

    {
        "fire_results": [
            { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
            { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
            { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
            { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
            { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
            null
        ]
    }
  • Alert template

    You can add the fire_results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

    Alert template

Use the results variable

You can use the results variable in an alert template to reference log content in alert notifications.

  • Alert monitoring rule

    Log Service generates alert messages based on the alert monitoring rule that you create.

    Alert monitoring rule
  • Alert message
    After an alert is triggered, Log Service generates an alert message that contains the following information. The information includes the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred.
    Note If an alert monitoring rule contains multiple query statements, the results array contains multiple items. Each item corresponds to a query statement.
    {
        "results": [{
            "store_type": "log",
            "region": "cn-hangzhou",
            "project": "test-alert",
            "store": "nginx-access-log",
            "query": "status >= 400 | select status, __source__ as ip, count(*) as cnt group by status, ip",
            "start_time": 1640006894,
            "end_time": 1640007014,
            "dashboard_id": "",
            "raw_results": [
                { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
                { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
                { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
                { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
                { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
                null
            ],
            "raw_result_count": 6,
            "fire_result": {
                "status": "401",
                "ip": "127.0.0.1",
                "cnt": "3"
            },
            "has_sql": true,
            "truncated": false,
            "role_arn": ""
        }]
    }
  • Alert template

    You can add the results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

    Alert template

Template variables

If you reference a variable in the JSON format in an alert template, such as {{ alert.results[0].raw_results | to_json }}, the content of the variable is displayed as a JSON string in alert notifications. You can also use a cycle method when you configure the alert template to display the content in alert notifications by line. This way, the content is easy to read. For more information, see Syntax for new alert templates. Example:
{%- for result in alert.fire_results %}
- status: {{ alert.status }}, count: {{ alert.cnt }}
{%- endfor %}

Log Service allows you to use built-in functions to format the content of alert notifications. For more information, see Syntax for new alert templates and Built-in functions in alert templates.