Use remote write to deliver data from a Managed Service for Prometheus instance to a self-managed one - Application Real-Time Monitoring Service

You can use remote write to deliver data from a Prometheus instance created in Alibaba Cloud Managed Service for Prometheus to a self-managed Prometheus instance. This way, you can query or store the metric data of Managed Service for Prometheus instances and self-managed ones in a centralized manner.

Prerequisites

A Prometheus instance is created. For more information, see the following:
EventBridge is activated. Authorization is completed in the EventBridge console. For more information, see Activate EventBridge and grant permissions to a RAM user.

Limits

Prometheus instance data cannot be delivered to a free Prometheus instance.

The following table lists the instances that support data delivery.

Instance	Description
Prometheus for Alibaba Cloud services	The free instance, except for those starting with the name cloud-product-prometheus.
Prometheus for container services	N/A
Prometheus for application monitoring	N/A
Prometheus for Flink Serverless	N/A
Prometheus for Kubernetes	N/A
General-purpose Prometheus instance	The general-purpose instance, except for those whose data is reported through OpenTelemetry endpoints.

When you deliver the data in the virtual private cloud (VPC), if the VPC where the Prometheus instance resides is not the same as the target VPC, ensure that the IP address of the vSwitch in the target VPC has been added to the whitelist of the Prometheus instance. Otherwise, network connection may fail.
On the vSwitch page in the VPC console, you can obtain the CIDR block of the vSwitch.

Step 1: Create a delivery task

Log on to the ARMS console.
In the left-side navigation pane, choose Managed Service for Prometheus > Data Delivery.
On the Data Delivery page, select a region in the top navigation bar and click Create Task.
In the dialog box that appears, set the Task Name and Task Description parameters, and click OK.

On the Edit Task page, configure the data source and event target.

Click + Add Data Source, set the parameters, and then click OK. The following table lists the parameters.

Parameter

Description

Example

Prometheus Instance

Select the Prometheus instance whose data you want to deliver.

c78cb8273c02*****

Data Filtering

Use labels to specify the metrics that you want to include or exclude.

Regular expressions are supported. Use line breaks to separate multiple conditions. The data can be delivered only when the relationship among the conditions is Logical AND.

__name__=AliyunEcs_CPUUtilization|AliyunEcs_memory_usedutilization
regionId=cn-hangzhou
id=i-2ze0mxp.*

Click Add Target. In the panel that appears, set the Destination Type parameter to Prometheus Remote Write, set other parameter as prompted (you can also set the external_labels parameter), and then click OK.

On the Edit Task page, click OK and Save.

Step 2: Configure the remote write endpoint and authentication

Prometheus instance type	Remote write endpoint	Requirements
Managed Service for Prometheus instance	For more information, see Configure open source Prometheus to read monitoring data from Managed Service for Prometheus by using a remote read URL.	You must specify an authentication method and an AccessKey pair that has the required permissions. For more information, see View the information about AccessKey pairs of a RAM user.
Self-managed Prometheus instance	For more information, see Open source Prometheus documentation.	The version of the Prometheus instance is 2.39 or later. You must configure the `out_of_order_time_window` setting. For more information, see PromLabs documentation. You must configure the `--web.enable-remote-write-receiver` startup parameter. For more information, see Prometheus documentation.

Step 3: Configure the network

Prometheus instance type	Network type	Network requirements
Managed Service for Prometheus instance	Internet	None.
Managed Service for Prometheus instance	VPC	Select a VPC in the same region, and configure the Remote Write URL as the internal endpoint of Service for Prometheus.
Self-managed Prometheus instance	Internet	None.
Self-managed Prometheus instance	VPC	Select the VPC where the Prometheus instance resides. Make sure that the remote write endpoint is accessible through the VPC, vSwitch, and security group. Note Prometheus uses Server Load Balancer (SLB) to expose services. For more information, see Getting started.

Step 4: Verify the result

Log on to the ARMS console.
In the left-side navigation pane, choose Managed Service for Prometheus > Instances.
Click the name of the Prometheus instance. In the left-side navigation pane of the page that appears, click Service Discovery.
On the Metrics tab, check whether the metric data is delivered as expected.