Ship Prometheus data to MaxCompute - Cloud Monitor - Alibaba Cloud Documentation Center

To export data from a Prometheus instance for custom processing, you can use the data shipping feature. This feature ships Prometheus instance data to MaxCompute. MaxCompute provides big data computing services and lets you combine data from other data sources. You can then use the processed data for tasks such as report analysis.

Prerequisites

A Prometheus instance is connected. For more information, see the following topics:
An Apsara MQ for Kafka instance is created and a topic is created. For more information, see Overview.
Function Compute (FC) is activated. For more information, see Step 1: Activate Function Compute.
EventBridge is activated. For more information, see Activate EventBridge and grant permissions.
Note
The data shipping feature depends on EventBridge. EventBridge will be commercially available on June 3, 2025. For information about EventBridge billing, see Billing overview.

Limitations

When you deliver the data in the virtual private cloud (VPC), if the VPC where the Prometheus instance resides is not the same as the target VPC, ensure that the IP address of the vSwitch in the target VPC has been added to the whitelist of the Prometheus instance. Otherwise, network connection may fail.
On the vSwitch page in the VPC console, you can obtain the CIDR block of the vSwitch.

The following table lists the instances that support data delivery.

Instance	Description
Prometheus for Alibaba Cloud services	The free instance, except for those starting with the name cloud-product-prometheus.
Prometheus for container services	N/A
Prometheus for Flink Serverless	N/A
Prometheus for Kubernetes	N/A
General-purpose Prometheus instance	The general-purpose instance, except for those whose data is reported through OpenTelemetry endpoints.

Only real-time data generated after you create a delivery task can be exported. Delivering historical data is not supported.

Billing

When data is shipped to MaxCompute, a Function Compute (FC) service is created. The service name prefix is {task_name}_SINK_FC_TO_MAXCOMPUTE. Function Compute (FC) is used to write data to MaxCompute, and this process incurs FC service fees. Data is sent to Function Compute (FC) in batches. Fewer calls to Function Compute (FC) result in lower costs. You can log on to the Function Compute console to view the status of your current FC resources.

Step 1: Prepare MaxCompute resources

Connect to a MaxCompute project. For more information, see Select a connection tool.
Define the table schema. Set the column names in the table schema to match the Prometheus data tag names. If a column name does not have a corresponding tag value, NULL is written to that column. If there are multiple types of metric data, the column names must be a union of all metric tag names.

For example, you need to export the following two types of metrics:

Metric example 1

{
	"__name__": "metric1",
	"cluster_id": "***",
	"host": "***",
	"node_id": "***",
	"job": "***",
	"value": 347.51300048828125,
	"timestamp": 1703211584056
}

Metric example 2

{
	"__name__": "metric2",
	"cluster_id": "***",
	"job": "***",
  "instance": "****",
  "cpu": "1",
	"value": 1.0,
	"timestamp": 1703211584056
}

Create a SQL table in MaxCompute based on the following example. For more information, see Use SQL for analysis.

CREATE TABLE metric (
  __name__ 		VARCHAR(50) NOT NULL,
  cluster_id 	VARCHAR(50),
  host 				VARCHAR(50),
  node_id 		VARCHAR(50),
  job 				VARCHAR(50),
  instance 		VARCHAR(50),
  cpu 				VARCHAR(50),
  __value__ 			DOUBLE,
  timestamp		BIGINT
);

Note

The table must contain at least the __name__, __value__, and timestamp fields. These are the basic tags for Prometheus data. We recommend that you design the table schema based on your business needs and define enough fields for tag mapping. If key tags are missing, the table will receive a large amount of duplicate data.

Step 2: Create a data shipping task

Log on to the Managed Service for Prometheus console.
In the left navigation pane, click Data Delivery.
On the Data Delivery page, select a region in the top navigation bar and click Create Task.
In the dialog box that appears, set the Task Name and Task Description parameters, and click OK.

On the Edit Task page, configure the data source and event target.

Click + Add Data Source, set the parameters, and then click OK. The following table lists the parameters.

Parameter	Description	Example
Prometheus Instance	The Prometheus instance whose data you want to deliver.	c78cb8273c02*****
Data Filtering	The label of the metric to be filtered. Regular expressions are supported. Use line breaks to separate multiple conditions. The data can be delivered only when the relationship among the conditions is Logical AND.	`__name__=AliyunEcs_CPUUtilization\|AliyunEcs_memory_usedutilization regionId=cn-hangzhou id=i-2ze0mxp.*`
Data Labeling	The label you add to metric data to be delivered. Use line breaks to separate multiple labels.	`deliver_test_key1=ssss deliver_test_key2=yyyy`

Click Add Target, set Destination Type to MaxCompute, configure the other parameters, and then click OK.

On the Edit Task page, click OK and Save.

Step 3: Verify that data is shipped to MaxCompute

In the SQL editor, enter the following SQL statement to query data in the `metric` table. Click Run and view the result on the Result tab.

select * from metric limit 20;