Alibaba Cloud Service Mesh:Import the generated Prometheus rule to activate SLOs

How Prometheus Operator selects rules

The Prometheus Operator uses custom resource definitions (CRDs) to manage Prometheus configurations. To add recording and alerting rules, create a PrometheusRule custom resource (CR). The Prometheus Operator determines which PrometheusRule objects to load based on the ruleSelector field in the Prometheus CR:

• If ruleSelector specifies matchLabels, the PrometheusRule must carry the same labels.

• If ruleSelector is left empty, labels are optional.

Step 1: Get the ruleSelector labels

Check the ruleSelector field in the Prometheus CR to determine which labels the Prometheus Operator requires on PrometheusRule objects.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Custom Resources in the left-side navigation pane.

3. On the CRDs tab, click PrometheusRule.

4. On the Resource Objects tab, select monitoring from the Namespace drop-down list. Find ack-prometheus-operator-prometheus and click Edit YAML in the Actions column.

5. Locate the ruleSelector field. A typical configuration looks like this: In this example, any PrometheusRule must include both labels (app: ack-prometheus-operator and release: ack-prometheus-operator) to be selected.

      ruleSelector:
        matchLabels:
          app: ack-prometheus-operator
          release: ack-prometheus-operator

Step 2: Deploy the PrometheusRule

1. Create a file named prometheusrule.yaml with the following content:

   • Set the labels field to match the matchLabels values from Step 1.

   • Replace the spec content with the generated Prometheus rule from the ASM console.

      apiVersion: monitoring.coreos.com/v1
      kind: PrometheusRule
      metadata:
        labels:
          app: ack-prometheus-operator
          release: ack-prometheus-operator
        name: asm-rules
        namespace: monitoring
      spec:
        # Replace with the generated Prometheus rule.

2. Apply the PrometheusRule to the ACK cluster:

      kubectl apply -f prometheusrule.yaml

Step 3: Verify that the rule is loaded

After the PrometheusRule is applied, the Prometheus Operator controller automatically writes the rule configurations into the Prometheus ConfigMap. Verify this through either the ACK console or the command line.

Option A: ACK console

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of the cluster that you want to manage and choose Configurations > ConfigMaps in the left-side navigation pane.

3. On the ConfigMap page, select monitoring from the Namespace drop-down list, find the Prometheus ConfigMap, and click Edit YAML in the Actions column.

4. Confirm that the ASM SLO rules appear in the ConfigMap. The following figure shows a successful import.

   

Option B: Command line

Run the following command to list PrometheusRule resources:

kubectl get prometheusrules -n monitoring

The output should include the asm-rules PrometheusRule.

Step 4: Verify that SLOs are active

Simulate traffic to validate SLO metrics

Use the following scenarios to confirm that metrics and alerts respond correctly.

Scenario 1: Normal traffic (99.5% success rate)

This script sends 200 requests where only 1 returns a 500 error, simulating a 99.5% success rate:

#!/bin/bash
for i in $(seq 200)
do
  if (( $i == 100 ))
  then
    curl -I http://<ingress-gateway-ip>/status/500;
  else
    curl -I http://<ingress-gateway-ip>/;
  fi
  echo "OK"
  sleep 0.01;
done;

Replace <ingress-gateway-ip> with the IP address of the ingress gateway. For details, see Deploy an ingress gateway service.

After the script completes, query slo:period_error_budget_remaining:ratio in the Prometheus console to check the remaining error budget. A value close to 1 indicates most of the budget remains intact.

Scenario 2: Error traffic (50% failure rate)

This script sends alternating successful and failing requests to simulate a 50% error rate and trigger burn rate alerts:

#!/bin/bash
for i in $(seq 200)
do
  curl -I http://<ingress-gateway-ip>/
  curl -I http://<ingress-gateway-ip>/status/500;
  echo "OK"
  sleep 0.01;
done;

After the script completes, open the Alerts page in the Prometheus console. If the burn rate exceeds the threshold, alerts fire.

What to do next

• Configure SLOs for applications in ASM to add or modify SLO targets

• SLO overview for a detailed explanation of SLO concepts, metrics, and multi-window burn rate alerting