Managed Service for Prometheus:Deliver Prometheus data to MaxCompute

Prerequisites

• Ensure that a Prometheus instance is connected. For more information, see the following topics:

  • Container observability

  • Cloud service observability

  • Host observability

• Deploy an ApsaraMQ for Kafka instance and create the necessary resources, such as a topic. For more information, see Overview.

• Function Compute (FC) has been activated. For more information, see Activate Function Compute.

• EventBridge has been activated. For more information, see Activate EventBridge and grant permissions.

Limits

• If the Prometheus instance and the destination reside in different VPCs, add the destination vSwitch CIDR block to the Prometheus instance allowlist to avoid network connection failures.

• Supported data sources:

  Instance type	Description
  Prometheus for Cloud Services	Excludes free instances whose names start with cloud-product-prometheus.
  Prometheus for Container Service	None
  Prometheus for Flink Serverless	None
  Prometheus for Kubernetes	None
  General-purpose	Excludes data reported through OpenTelemetry endpoints.

• Only real-time data generated after task creation is shipped; historical data is not supported.

Billing

When you deliver data to MaxCompute, a Function Compute (FC) service is created. The service name is prefixed with {task_name}_SINK_FC_TO_MAXCOMPUTE. Function Compute (FC) writes the data to MaxCompute, which incurs FC service fees. Data is batched before being sent to Function Compute (FC). Therefore, the fewer the invocations of Function Compute (FC), the lower the related costs. You can log on to the Function Compute console to view the status of your FC resources.

Step 1: Prepare MaxCompute resources

1. Connect to your MaxCompute project by using one of the methods described in the MaxCompute documentation. For more information, see Select a connection tool.

2. Define the table schema. The column names in the table schema must match the Prometheus data label names. If a column name does not have a corresponding label, a NULL value is written to that column. If you have multiple types of metrics, the column names must be the union of all metric label names.

{
	"__name__": "metric1",
	"cluster_id": "***",
	"host": "***",
	"node_id": "***",
	"job": "***",
	"value": 347.51300048828125,
	"timestamp": 1703211584056
}

{
	"__name__": "metric2",
	"cluster_id": "***",
	"job": "***",
  "instance": "****",
  "cpu": "1",
	"value": 1.0,
	"timestamp": 1703211584056
}

CREATE TABLE metric (
  __name__ 		VARCHAR(50) NOT NULL,
  cluster_id 	VARCHAR(50),
  host 				VARCHAR(50),
  node_id 		VARCHAR(50),
  job 				VARCHAR(50),
  instance 		VARCHAR(50),
  cpu 				VARCHAR(50),
  __value__ 			DOUBLE,
  timestamp		BIGINT
);

Step 2: Create a data delivery task

1. Log on to the Cloud Monitor console.

2. In the left-side navigation pane, click Prometheus Monitoring > Data Shipping.

3. On the Data Shipping page, select a destination region in the top menu bar and click Create Task.

4. In the dialog box that appears, enter a Task Name and Task description, and click OK.

5. On the Edit Task page, configure the data source and destination.

   1. Click + Add Data Source, configure the following parameters, and then click OK.

      Parameter	Description	Example
      Prometheus Instance	The source Prometheus instance.	c78cb8273c02*****
      Data Filtering	Filter data by metric labels. Supports regular expressions. Separate multiple conditions with line breaks (AND logic).	__name__=AliyunEcs_CPUUtilization|AliyunEcs_memory_usedutilization regionId=cn-hangzhou id=i-2ze0mxp.*
      Data Labeling	Additional labels for shipped metric data. Separate with line breaks.	deliver_test_key1=ssss deliver_test_key2=yyyy

   2. Click Add Target, set Target Type to MaxCompute, configure the remaining fields as required, and then click OK.

6. On the Edit Task page, click OK, and then click Save to create the data shipping task.

Step 3: Verify data delivery

In the SQL editor, enter the following SQL statement to query data in the metric table. Click Run and view the execution result on the Result tab.

select * from metric limit 20;