Managed Service for Prometheus:Edit RecordingRule.yaml

Configure recording rules

1. Log on to the Cloud Monitor console.

2. In the navigation pane on the left, choose Managed Service for Prometheus > Instances to open the instance list for Managed Service for Prometheus.

3. In the upper-left corner of the page, select the target region, and then click the name of the desired Prometheus instance.

4. In the left-side navigation pane, click Settings. On the Settings tab, click Edit RecordingRule.yaml.

5. In the dialog box that appears, delete the default template, enter your recording rule configuration, and then click Save.

   Note

   Use only one RecordingRule.yaml file per cluster. The names of the rule groups must be unique.

   Example:

   groups:
     - name: apiserver_request_total
       interval: 60s
       rules:
         - record: job_instance_mode:apiserver_request_total:avg_rate5m
           expr: avg by (job, instance, mode) (rate(apiserver_request_total[5m]))
           labels:
             team: operations
         - record: job:apiserver_request_total:sum_rate10m
           expr: sum by (job)(rate(apiserver_request_total[10m]))
           labels:
             team: operations

   Parameter	Description
   groups	A list of rule groups. You can configure multiple rule groups in one RecordingRule.yaml file.
   name	The name of the rule group. The name must be unique within a file.
   interval	(Optional) The evaluation interval for rules in the group. Default: 60s.
   rules	The list of rules in the group.
   record	The name of the new time series to record. The name must comply with the Prometheus metric naming conventions. Note The recommended format for a recording rule name is level:metric:operations. level: The aggregation level and the output labels of the rule. metric: The name of the metric. operations: The list of operations applied to the metric, with the most recent operation listed first.
   expr	The expression that Alibaba Cloud Managed Service for Prometheus uses to calculate recording rule metrics. The expression must be a valid PromQL query.
   labels	(Optional) Labels to add to the new metric.

View recording rule metrics

Method 1: View metrics in Grafana

1. Log on to your Grafana system as an administrator.

2. In the left-side navigation pane, click Explore.

3. On the Explore page, select the data source that corresponds to your Kubernetes cluster. In the metric query field, enter the name of the recording rule metric, such as job_instance_mode:apiserver_request_total:avg_rate5m. Set Query type to Range and then click Run Query. The query returns a line chart that shows the trend of the apiserver request rate.

Method 2: View metrics from the Prometheus console

1. Log on to the Cloud Monitor console.

2. In the navigation pane on the left, choose Managed Service for Prometheus > Instances to open the instance list for Managed Service for Prometheus.

3. In the upper-left corner of the page, select the target region, and then click the name of the desired Prometheus instance.

4. In the left-side navigation pane, choose Service Discovery and click the Metrics tab. Then, enter the name of the recording rule metric in the search box to view its details.

Recording rules and remote write

Recording rules in Alibaba Cloud Managed Service for Prometheus support remote write, the same as open-source Prometheus.

1. Log on to the Cloud Monitor console.

2. In the navigation pane on the left, choose Managed Service for Prometheus > Instances to open the instance list for Managed Service for Prometheus.

3. In the upper-left corner of the page, select the target region, and then click the name of the desired Prometheus instance.

4. In the left-side navigation pane, click Settings. On the Settings tab, click Edit Prometheus.yaml.

5. In the Edit Prometheus.yaml dialog box that appears, configure remote write.

   remote_write:
       - url: "https://cn-beijing.arms.aliyuncs.com/prometheusxxx/cn-beijing/api/v3/write"
         remote_timeout: 60s
         name: xxx
         basic_auth:
             username: xxx
             password: xxx
         write_relabel_configs:
           - action: keep
             source_labels: ["instance"]
             regex: "xxx"
       - url: "https://cn-hangzhou.arms.aliyuncs.com/prometheuxxxxx/cn-hangzhou/api/v3/write"
         remote_timeout: 60s
         name: xxx
         basic_auth:
             username: xxx
             password: xxx
         write_relabel_configs:
           - action: replace
             source_labels: ["job"]
             regex: "(.*)"
             target_label: __xxx_job__
             replacement: ${1}:xxx

   Note

   The method for configuring remote write is the same as in open source Prometheus. Multiple remote write configurations and write_relabel_configs are supported.

   To remotely write only the metrics generated by a recording rule to another storage, configure a write_relabel_configs section as shown in the following example:

   remote_write:
     - url: "https://xxxx/api/v1/prom/write?db=dbname&u=username&p=password"    // The endpoint of the remote storage
       write_relabel_configs:
         - source_labels: [__name__]
           regex: job_instance_mode:apiserver_request_total:avg_rate5m
           action: keep

   Note

   This configuration remotely writes only the specified recording rule metric (job_instance_mode:apiserver_request_total:avg_rate5m) to another storage.

Related operations

Remote write requires a public endpoint URL. If you need to add the recording rule component's IP address to a network whitelist, use the appropriate IP address for your region.

 China (Beijing): 101.200.XX.XX
 China (Hangzhou): 118.31.XX.XX
 China (Shanghai): 106.14.XX.XX
 China (Shenzhen): 8.129.XX.XX
 China (Zhangjiakou): 39.103.XX.XX
 China (Qingdao): 139.129.XX.XX
 China (Chengdu): 47.108.XX.XX
 China (Hong Kong): 47.242.XX.XX
 Singapore: 47.241.XX.XX

To obtain the full IP addresses, you can submit a ticket or contact a product expert by searching for the Prometheus on-call account in DingTalk (ID: aliprometheus).