Simple Log Service:Reference log content in alert notifications

For example, if you monitor access to NGINX, you can use the following query statement to obtain information such as the status code that is returned and indicates an access error, the IP address of the machine on which the error occurred, and the number of times that the error occurred. Then, you can create an alert monitoring rule based on the information and reference the information in the alert template. If a status code is 400 or a larger number, Simple Log Service sends you an alert notification that includes the status code, IP address of the machine on which the error occurred, and the number of times that the error occurred.

• Query statement

  status >=400 | SELECT status, __source__ AS ip, count(*) AS cnt GROUP BY status,ip

• Query results

Use the labels variable

You can use the labels variable in an alert template to reference log content in alert notifications.

• Alert monitoring rule

  You can add labels or configure group evaluation when you create an alert monitoring rule. After an alert is triggered, information about labels is generated and included in the alert message.

  • Configure group evaluation: After you configure group evaluation, the fields that you select for group evaluation are automatically added to the information about labels. For more information, see Use the group evaluation feature.

    Important

    When you select fields for group evaluation, we recommend that you select a field that can identify a monitored entity and whose values can be enumerated. If a field cannot identify a monitored entity, we recommend that you do not select the field. If you do not select a suitable field, an excessive number of groups are generated. If the trigger condition in a group is met, an alert is triggered. If a large number of groups exist, alert storms may occur. In this case, you may miss out on important alert information.

    For example, if you select the status field for group evaluation, different status codes trigger different alerts when error logs are recorded.

  • Add labels: Labels are identifying attributes. For more information, see Labels.

  For example, if you want to add information such as the log source, environment, and status code to alert messages, you can add the app:nginx and env:prod labels and select the status field for group evaluation.

  

• Alert message

  After an alert is triggered, Simple Log Service generates an alert message that contains the following information:

  {
      "labels": {
          "app": "nginx",
          "env": "prod",
          "status": "404"
      }
  }

• Alert template

  You can add the labels variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

  

Use the annotations variable

You can use the annotations variable in an alert template to reference log content in alert notifications.

• Alert monitoring rule

  You can manually add information about annotations or turn on Auto-Add Annotations to automatically add information about annotations when you create an alert monitoring rule. After an alert is triggered, information about annotations is generated and included in the alert message. For more information, see Annotations.

  Note

  Simple Log Service provides two annotations by default.

  For example, if you want to add information such as the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred to alert messages, you can add annotations such as status:${status}, ip:${ip}, and cnt:${cnt}.

  • Manually add information about annotations

  • Turn on Auto-Add Annotations to automatically add information about annotations

• Alert message

  After an alert is triggered, Simple Log Service generates an alert message that contains the following information:

  {
      "annotations": {
          "title": "Alert is triggered by an NGINX access error",
          "desc": "The error that is indicated by the status code 400 occurred 15 times"
          "status": "400"
          "ip": "127.0.0.1"
          "cnt": "15"
          "__count__": "1"
      }
  }

• Alert template

  You can add the annotations variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

  

Use the fire_results variable

You can use the fire_results variable in an alert template to reference log content in alert notifications.

• Alert monitoring rule

  Simple Log Service generates alert messages based on the alert monitoring rule that you create.

  

• Alert message

  After an alert is triggered, Simple Log Service generates an alert message that contains the following information. The information includes the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred.

  {
      "fire_results": [
          { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
          { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
          { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
          { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
          { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
          null
      ]
  }

• Alert template

  You can add the fire_results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

  

Use the results variable

You can use the results variable in an alert template to reference log content in alert notifications.

• Alert monitoring rule

  Simple Log Service generates alert messages based on the alert monitoring rule that you create.

  

• Alert message

  After an alert is triggered, Simple Log Service generates an alert message that contains the following information. The information includes the status code that is returned and indicates an access error, IP address of the machine on which the error occurred, and number of times that the error occurred.

  Note

  If an alert monitoring rule contains multiple query statements, the results array contains multiple items. Each item corresponds to a query statement.

  {
      "results": [{
          "store_type": "log",
          "region": "cn-hangzhou",
          "project": "test-alert",
          "store": "nginx-access-log",
          "query": "status >= 400 | select status, __source__ as ip, count(*) as cnt group by status, ip",
          "start_time": 1640006894,
          "end_time": 1640007014,
          "dashboard_id": "",
          "raw_results": [
              { "status": "401", "ip": "127.0.0.1", "cnt": "3" },
              { "status": "400", "ip": "127.0.0.1", "cnt": "7" },
              { "status": "501", "ip": "127.0.0.1", "cnt": "4" },
              { "status": "404", "ip": "127.0.0.1", "cnt": "4" },
              { "status": "402", "ip": "127.0.0.1", "cnt": "6" },
              null
          ],
          "raw_result_count": 6,
          "fire_result": {
              "status": "401",
              "ip": "127.0.0.1",
              "cnt": "3"
          },
          "has_sql": true,
          "truncated": false,
          "role_arn": ""
      }]
  }

• Alert template

  You can add the results variable to the alert template to reference specified log content in alert notifications. For more information about how to create an alert template, see Create an alert template.

  

Template variables

{%- for result in alert.fire_results %}
- status: {{ alert.status }}, count: {{ alert.cnt }}
{%- endfor %}

Simple Log Service allows you to use built-in functions to format the content of alert notifications. For more information, see Syntax for new alert templates and Built-in functions in alert templates.