Prometheus Service provides preset alert rules for connected Alibaba Cloud services. If the preset alert rules do not meet your business requirements, you can create custom alert rules. This topic describes how to view preset alert rules and create custom alert rules.

Prerequisites

View preset alert rules

  1. Log on to the ARMS console .
  2. In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.
  3. In the top navigation bar of the Prometheus Monitoring page, select the region where your Prometheus instance resides.
    On the Prometheus Monitoring page, all the clusters connected to Prometheus Service are displayed. Prometheus for Cloud Services is displayed in the Instance Type column of a Prometheus instance that monitors Alibaba Cloud services.
  4. Click the name of the Prometheus instance that monitors Alibaba Cloud services.
  5. In the left-side navigation pane, click Alert Configurations.
    The Alert Configurations page displays preset alert rules for Alibaba Cloud services. Alert Configurations page

Create an alert rule

If the preset alert rules do not meet your business requirements, you can create custom alert rules.

  1. Log on to the ARMS console .
  2. In the left-side navigation pane, choose Prometheus Monitoring > Prometheus Instances.
  3. In the top navigation bar of the Prometheus Monitoring page, select the region where the Prometheus instance resides.
    On the Prometheus Monitoring page, all the clusters connected to Prometheus Service are displayed. Prometheus for Cloud Services is displayed in the Instance Type column of a Prometheus instance that monitors Alibaba Cloud services.
  4. Click the name of the Prometheus instance that monitors Alibaba Cloud services.
  5. In the left-side navigation pane, click Alert Configurations.
  6. On the Alert Configurations page, click Create Alert in the upper-right corner.
  7. In the Create Alert panel, set the following parameters:
    1. Optional:Select a template from the Alert Template drop-down list.
    2. Enter a rule name in the Rule Name field. Example: Alert for inbound traffic.
    3. Enter a PromQL statement as the expression in the Alert Expression field. Example: (sum(rate(kube_state_metrics_list_total{job="kube-state-metrics",result="error"}[5m])) / sum(rate(kube_state_metrics_list_total{job="kube-state-metrics"}[5m]))) > 0.01.
      Notice
      • If a PromQL statement contains a dollar sign ($), an error occurs. You must delete the equal sign (=) and the parameters on both sides of the equal sign (=) in the statement that contains the dollar sign ($). For example, change sum (rate (container_network_receive_bytes_total{instance=~"^$HostIp.*"}[1m])) to sum (rate (container_network_receive_bytes_total[1m])).
      • Prometheus Service supports various metrics of Alibaba Cloud services. For more information, see View the metrics of Alibaba Cloud services.
    4. Enter a number in the Duration field. An alert notification is sent only when the alert condition persists for the specified duration. Unit: minutes. For example, if you enter 1, an alert notification is sent only when the alert condition persists for 1 minute.
      Note The alert condition refers to the condition specified by the PromQL statement. By default, Prometheus Service collects data at intervals of 15 seconds. An alert notification is sent only when 4N data records that are consecutively collected meet the alert condition. The threshold 4N is calculated by using the following formula: N × 60/15 = 4N. In this example, N is set to 1. Therefore, an alert notification is sent only when four consecutively collected data records meet the alert condition. To ensure that an alert notification can be sent each time a collected data record meets the alert condition, set N to 0.
    5. Enter the notification content in the Alert Message field.
    6. Optional:Click Create Tag in the Tags field of the Advanced Settings section to add one or more tags to the alert rule. The specified tags can be used as options for a dispatch rule in a notification policy.
    7. Optional:Click Create Annotation in the Annotations field of the Advanced Settings section. Then, enter message in the Key field and {{variable name}} alert message in the Value field. The specified annotation is in the format of message:{{variable name}} alert message. Example: message:{{$labels.pod_name}} restart.

      You can use a custom variable name or select an existing tag as the variable name. The following content describes the existing tags:

      • The tags that are carried in the metrics of an alert rule expression.
      • The tags that are created when you create an alert rule. For more information, see Create an alert rule.
      • The default tags provided by ARMS. The following table describes the default tags.
        Tag Description
        alertname The name of the alert. The format is <Alert name>_<Cluster name>.
        _aliyun_arms_alert_level The level of the alert.
        _aliyun_arms_alert_type The type of the alert.
        _aliyun_arms_alert_rule_id The ID of the alert rule.
        _aliyun_arms_region_id The ID of the region.
        _aliyun_arms_userid The ID of the user.
        _aliyun_arms_involvedObject_type The subtype of the associated object, for example, ManagedKubernetes or ServerlessKubernetes.
        _aliyun_arms_involvedObject_kind The type of the associated object, for example, app or cluster.
        _aliyun_arms_involvedObject_id The ID of the associated object.
        _aliyun_arms_involvedObject_name The name of the associated object.
    8. Click OK.
    After you create an alert rule, the Alert Configurations page displays the created alert rule.