Alibaba Cloud Prometheus Monitoring provides out-of-the-box alert rules. You can also customize alert rules for specific monitored objects. If a rule is triggered, the system sends alert messages to the specified contact group by using a specified notification method. This way, contacts can resolve issues at the earliest opportunity.

Prerequisites

Procedure

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, click Prometheus Monitoring.
  3. In the top navigation bar, select a region. Then, click the name of the required Kubernetes cluster.
  4. In the left-side navigation pane, choose Alarm configuration beta. Then, click Create Alert in the upper-right corner.
  5. In the Create Alert dialog box, configure the following parameters, and then click OK.
    Note The Time parameter is not supported.
    1. Enter a name in the Rule Name field. Example: alerts for inbound traffic.
    2. Enter an expression that uses a PromQL statement. Example: (sum(rate(kube_state_metrics_list_total{job="kube-state-metrics",result="error"}[5m])) / sum(rate(kube_state_metrics_list_total{job="kube-state-metrics"}[5m]))) > 0.01.
      Notice An error may be reported if a PromQL statement contains a dollar sign ($). You must remove the equal sign (=) and the parameters on both sides of the equal sign (=) from the statement that contains the dollar sign ($). For example, change sum (rate (container_network_receive_bytes_total{instance=~"^$HostIp.*"}[1m])) to sum (rate (container_network_receive_bytes_total[1m])).
    3. In the Labels section, click Create Tag to specify alert tags. The specified tags can be used as options for a dispatch rule.
    4. In the Annotations section, specify a template for alert messages. Click Create Annotation. Set Key to message and Value to {{variable name}} alert message. The specified annotation is in the format of message:{{variable name}} alert notification. Example: message:{{$labels.pod_name}} restart.

      You can customize a variable name or select an existing tag as the variable name. Existing tags:

      • The tags that are carried in the metrics of an alert rule expression.
      • The tags that are created when you create an alert rule. For more information, see Create an alert.
      • The default tags provided by ARMS. The following table describes the default tags.
        Tag Description
        alertname The name of the alert. The format is <Alert name>_<Cluster name>.
        _aliyun_arms_alert_level The level of the alert.
        _aliyun_arms_alert_type The type of the alert.
        _aliyun_arms_alert_rule_id The ID of the alert rule.
        _aliyun_arms_region_id The ID of the region.
        _aliyun_arms_userid The ID of the user.
        _aliyun_arms_involvedObject_type The subtype of the associated object, for example, ManagedKubernetes or ServerlessKubernetes.
        _aliyun_arms_involvedObject_kind The type of the associated object, for example, app or cluster.
        _aliyun_arms_involvedObject_id The ID of the associated object.
        _aliyun_arms_involvedObject_name The name of the associated object.
    Prometheus-Create alarm