Alibaba Cloud Service Mesh:Import the generated Prometheus rule to the Prometheus system for the SLOs to take effect

Step 1: Import the generated Prometheus rule to the Prometheus system

In this example, Prometheus is deployed by using the Prometheus Operator. The Prometheus configurations are based on the custom resource definitions (CRDs). To define recording and alerting rules, you can create a Prometheus custom resource (CR) that includes a PrometheusRule object with app: ack-prometheus-operator and release: ack-prometheus-operator labels.

1. Obtain settings of the ruleSelector field in the ACK console.

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

   2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Custom Resources in the left-side navigation pane.

   3. On the CRDs tab, click the PrometheusRule.

   4. On the Resource Objects tab, select monitoring from the Namespace drop-down list. Find ack-prometheus-operator-prometheus and click Edit YAML in the Actions column.

   5. Obtain settings of the ruleSelector field.

      The following code shows a sample ruleSelector field. If you want the Prometheus Operator to select a Prometheus rule, the PrometheusRule must have the same set of labels as that in matchLabels.

      ruleSelector:
        matchLabels:
          app: ack-prometheus-operator
          release: ack-prometheus-operator

2. Deploy a PrometheusRule.

   1. Create a prometheusrule.yaml file that contains the following content:

      In the YAML file, the content in the labels field is the same as that in the matchLabels field in the previous step. The content in the spec field is the generated Prometheus rule.

      apiVersion: monitoring.coreos.com/v1
      kind: PrometheusRule
      metadata:
        labels:
          app: ack-prometheus-operator
          release: ack-prometheus-operator
        name: asm-rules
        namespace: monitoring
      spec:
        # Replace with the generated Prometheus rule. 

   2. Run the following command in the ACK cluster to deploy the PrometheusRule:

      kubectl apply -f prometheusrule.yaml

3. Check whether the PrometheusRule is deployed.

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

   2. On the Clusters page, click the name of the cluster that you want to manage and choose Configurations > ConfigMaps in the left-side navigation pane.

   3. In the upper part of the ConfigMap page, select monitoring from the Namespace drop-down list, find the Prometheus ConfigMap, and click Edit YAML in the Actions column.

      If the controller that corresponds to the PrometheusRule automatically writes the configurations to the ConfigMap for Prometheus to read, the PrometheusRule is deployed. The following figure shows a successful example.

Step 2: Verify the effect of SLOs

View Prometheus monitoring data and alerts

1. Use kubectl to connect to the ACK cluster. Run the following command on the CLI to forward all traffic from a local port to the ack-prometheus-operator-prometheus service:

   kubectl --namespace monitoring port-forward svc/ack-prometheus-operator-prometheus 9090

2. Click https://localhost:9090 to access the Prometheus console.

3. On the Prometheus page, enter asm_slo_info in the text box, and click Execute to view the SLO configurations.

   The following figure shows that the Prometheus recording rule is configured.

4. In the top navigation bar, click Alerts to view the alerting rule.

   If information similar to that in the following figure appears, the Prometheus alerting rule is configured.

Scenario 1: Simulate normal requests

1. Run the following script on the CLI to simulate the situation that 99.5% of requests are successfully responded to.

   Replace {IP address of the ingress gateway} in the script with the actual IP address of the ingress gateway. For more information about how to obtain the IP address of an ingress gateway, see Deploy an ingress gateway service.

   #!/bin/bash
   for i in `seq 200`
   do
   if (( $i == 100 ))
   then
       curl -I http://{IP address of the ingress gateway}/status/500;
   else
       curl -I http://{IP address of the ingress gateway}/;
   fi
   echo "OK"
   sleep 0.01;
   done;

2. Return to the Prometheus page of the Prometheus console, enter slo:period_error_budget_remaining:ratio in the text box, and then click Execute. View the changes of the remaining error budget.

   Sample result:

   The following table describes the main metrics of the SLO. For more information, see SLO overview.

   Metric	Description
   slo:period_error_budget_remaining:ratio	The remaining error budget during the 30-day compliance period of the SLO.
   slo:sli_error:ratio_rate30d	The average error rate during the 30-day compliance period of the SLO.
   slo:period_burn_rate:ratio	The burn rate for the 30-day compliance period of the SLO.
   slo:current_burn_rate:ratio	The current burn rate.

Scenario 2: Simulate error requests

Manually simulate faults to trigger alerts.

1. Run the following script on the CLI to simulate the situation that 50% of requests fail and the burn rate is 50%.

   Replace {IP address of the ingress gateway} in the script with the actual IP address of the ingress gateway.

   #!/bin/bash
   for i in `seq 200`
   do
   curl -I http://{IP address of the ingress gateway}/
   curl -I http://{IP address of the ingress gateway}/status/500;
   echo "OK"
   sleep 0.01;
   done;

2. Return to the Alerts page of the Prometheus console to view alerts.

   Sample result:

View alerts in the Alertmanager console

The Alertmanager component collects alerts generated by the Prometheus server and sends the alerts to the specified contacts.

1. Run the following command to forward all traffic from the local port to the ack-prometheus-operator-alertmanager service:

   kubectl --namespace monitoring port-forward svc/ack-prometheus-operator-alertmanager 9093

2. Click https://localhost:9093 to access to the Alertmanager console.

3. On the Alertmanager page, click the icon to view alerts.

   The following figure shows that custom alert information is collected.