Deliver Prometheus metrics via remote write to another instance - Managed Service for Prometheus

The data delivery feature uses Remote Write to deliver metrics from Managed Service for Prometheus to a self-managed Prometheus instance. Use data delivery to import data from Alibaba Cloud into an existing instance, so that you can centrally manage, query, and store metrics across cloud and on-premises environments. Confirm the Remote Write endpoint, authentication method, and network mode of the delivery target before you create a delivery task.

Prerequisites

Ensure that a Prometheus instance is connected. For more information, see the following topics:
EventBridge has been activated. For more information, see Activate EventBridge and grant permissions.

If the delivery target is a self-managed Prometheus instance, make sure that the following requirements are met:
- Use self-managed Prometheus version 2.39 or later.
- Configure out_of_order_time_window. For more information, see What's new in Prometheus 2.39.
- Configure the --web.enable-remote-write-receiver startup parameter. For more information, see Feature flags.

Limits

When you select Managed Service for Prometheus as the delivery target, free instance types are not supported.

Supported data sources:

Instance type	Description
Prometheus for Cloud Services	Excludes free instances whose names start with cloud-product-prometheus.
Prometheus for Container Service	None
Prometheus for Flink Serverless	None
Prometheus for Kubernetes	None
General-purpose	Excludes data reported through OpenTelemetry endpoints.

If the Prometheus instance and the destination reside in different VPCs, add the destination vSwitch CIDR block to the Prometheus instance allowlist to avoid network connection failures.
Only real-time data generated after task creation is shipped; historical data is not supported.

Step 1: Prepare the delivery target

A delivery task can deliver data to a self-managed Prometheus instance or to an Alibaba Cloud Prometheus instance. This topic focuses on the self-managed Prometheus target and lists the Alibaba Cloud Prometheus settings for comparison.

Remote Write endpoint and authentication

Prometheus type	Remote Write endpoint	Requirements
Alibaba Cloud Prometheus	See Use remote read with Managed Service for Prometheus.	Select Basic Auth and enter an AccessKey with the required permissions. For more information, see View RAM user AccessKey information.
Self-managed Prometheus	See the Prometheus Remote Write specification.	The self-managed Prometheus instance meets the requirements in Prerequisites.

Network requirements

Prometheus type	Network mode	Network requirements
Alibaba Cloud Prometheus	Internet	None
Alibaba Cloud Prometheus	VPC	Select a VPC network in the same region, and configure the Prometheus Remote Write endpoint as the internal endpoint of Alibaba Cloud Prometheus.
Self-managed Prometheus	Internet	None
Self-managed Prometheus	VPC	Select a VPC and vSwitch that can access the self-managed Prometheus instance. The data delivery service automatically selects an IP address on the selected vSwitch as the outbound IP address. Prometheus exposes services by using load balancing. For more information, see Service management. For the whitelist requirement that applies to the vSwitch IP address, see Limits.

Step 2: Create a delivery task

Log on to the Cloud Monitor console.
In the left-side navigation pane, click Prometheus Monitoring > Data Shipping.
On the Data Shipping page, select a destination region in the top menu bar and click Create Task.
In the dialog box that appears, enter a Task Name and Task description, and click OK.
On the Edit Task page, configure the data source and destination.

Click + Add Data Source, configure the following parameters, and then click OK.

Parameter	Description	Example
Prometheus Instance	The source Prometheus instance.	c78cb8273c02*****
Data Filtering	Filter data by metric labels. Supports regular expressions. Separate multiple conditions with line breaks (AND logic).	`__name__=AliyunEcs_CPUUtilization\|AliyunEcs_memory_usedutilization regionId=cn-hangzhou id=i-2ze0mxp.*`
Data Labeling	Additional labels for shipped metric data. Separate with line breaks.	`deliver_test_key1=ssss deliver_test_key2=yyyy`

Click Add Target and set Destination Type to Prometheus Remote Write. Configure the remaining parameters in the form, including external_labels, and then click OK. For the values of the Remote Write endpoint, authentication method, and network mode, see Step 1: Prepare the delivery target.
On the Edit Task page, click OK, and then click Save to create the data shipping task.

Step 3: Verify the result

If the delivery target is an Alibaba Cloud Prometheus instance, verify the delivered data in the console:

Log on to the Cloud Monitor console.
In the left navigation pane, choose Managed Service for Prometheus > Instances.
Click the name of the Managed Service for Prometheus instance that you want to manage, and then click Metric Management in the left-side navigation pane.
On the Metrics Explorer tab, search for the metrics of the source instance and check whether the metrics are synchronized to the target Prometheus instance.

If the delivery target is a self-managed Prometheus instance, query the delivered metrics on the self-managed Prometheus instance and check whether the metrics of the source instance are synchronized.

If the metrics of the source instance are not synchronized to the target Prometheus instance, check whether the network requirements in Limits are met.