Access open source Prometheus data with Prometheus Service using an address - Managed Service for Prometheus

Managed Service for Prometheus provides a standard Remote Read endpoint. You can add this endpoint to your self-managed Prometheus configuration to query metrics from your instance in the cloud. This topic describes how to configure a self-managed Prometheus server to read metrics from a Managed Service for Prometheus instance.

Important

The Prometheus Remote Read mode reads a large amount of raw time series data to a local Prometheus engine for computation and has many usage limitations. Therefore, we recommend that you directly use the HTTP API query protocol provided by Managed Service for Prometheus.

Managed Service for Prometheus provides a complete Prometheus query API. It processes queries on the server and returns only the final results, delivering performance many times faster than using Remote Read. The API allows you to send queries directly over HTTP and connect tools like Grafana without a local Prometheus engine. For configuration details, see Connect Grafana or self-managed applications to Prometheus data by using an HTTP API endpoint.

Limitations of the Remote Read approach include:

High infrastructure costs of maintaining a local Prometheus compute engine.
High end-to-end latency because each query requires transferring large volumes of raw data over the network.
Strict service limits that are easy to exceed when querying large datasets.

Limitations

The Remote Read endpoint does not support HTTP/2.

Prerequisites

You have created a Managed Service for Prometheus instance. For more information, see Create an instance.

(Optional) Step 1: Grant CMS read and write permissions to a RAM user

If your Alibaba Cloud Prometheus instance was created by an Alibaba Cloud account and you need to use the AccessKey ID and AccessKey Secret of a RAM user for remote read and write operations, you must first grant the RAM user read and write permissions for CMS.

Log on to the RAM console using an Alibaba Cloud account or as a RAM administrator. In the navigation pane on the left, choose Permission Management > Authorize.

On the Authorize page, click Create Authorization. On the Create Authorization page, configure the parameters as follows.

Parameter	Description
Resource Range	Select a resource scope as needed.
Principal	Specify the principal. The principal is the RAM user to whom you want to grant permissions.
Policy	Add the following permissions to the RAM user as needed: `AliyunCloudMonitorFullAccess`: Full permissions for CMS. A RAM user with this permission has full permissions to view, edit, and delete instances of all sub-products. `AliyunCloudMonitorReadOnlyAccess`: Read-only permissions for CMS. A RAM user with this permission can only view instance information for all sub-products. The user cannot modify or delete instances.

Click Confirm, and then click Delete.

Step 2: Get the Remote Read endpoint

Log on to the CloudMonitor console. In the navigation pane on the left, choose Prometheus MonitoringInstance List to go to the Instance List page.
In the top menu bar, select the region of the instance. In the Actions column for the target cluster, click Modify.
On the Modify tab, copy the public or internal Remote Read endpoint as needed.

The public endpoint format is http://{region}.arms.aliyuncs.com:9090/api/v1/prometheus/xxx. The internal endpoint format is http://{region}-intranet.arms.aliyuncs.com:9090/api/v1/prometheus/xxx. The xxx part is the system-generated authentication credential.

Step 3: Configure self-managed Prometheus

Install Prometheus. For more information, see the Prometheus download page.

Edit the Prometheus.yml configuration file. Add the following content to the end of the file. Replace the remote_read URL with the endpoint from Step 2. Then, save the file.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
remote_read:
  # Replace this with your Remote Read endpoint.
  - url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_read"
    read_recent: true

Restart the self-managed Prometheus service.

Step 4: View Alibaba Cloud Prometheus data on your self-managed Prometheus

Log on to your self-managed Prometheus in a browser.
```
http://localhost:9090
```
On the Prometheus page, click Graph in the menu bar at the top of the page.
On the Graph tab, enter a query expression in the search box, and then click Execute to the right of the search box.
For example, query the CPU utilization of a specific IP address.
```
100 - (avg by (instance) (rate(node_cpu_seconds_total{instance="your_IP_address:9100", mode="idle"}[5m])) * 100)
```

FAQ

Queries return no data and report no errors after Remote Read is configured

If queries return no data after the configuration, check whether your local Prometheus configuration contains external_labels. This setting can prevent Remote Read from querying the expected data. Run the following command to view the external_labels configuration in the Prometheus configuration file and confirm that the cloud and cluster labels are set correctly.

[xxx prometheus]# head prometheus.yml
global:
  scrape_interval: 60s
  evaluation_interval: 15s
  external_labels:
    cloud: xxx
    cluster: xxx