Push monitoring data to Managed Service for Prometheus by using Pushgateway, which is ideal for data sources that cannot be periodically scraped.
Solution overview
If your data source cannot or should not be periodically scraped by Prometheus Server, for example in environments without stable network connectivity, use Pushgateway instead. The data source sends monitoring data to Pushgateway, and Prometheus Server periodically retrieves it. Follow these steps:
-
Get the Pushgateway endpoint: Obtain the Pushgateway endpoint from the Managed Service for Prometheus console.
-
Report data: Use the curl command or an open source SDK to push data. Ensure metrics are collected and monitored by Prometheus promptly and reliably.
-
Add data protection configuration (optional): The standard Pushgateway protocol does not include data protection features. The Pushgateway SDK supports only basic Auth and lacks advanced, standardized authentication. Any client that obtains the Pushgateway endpoint can push data. To secure your data, obtain a token from the Managed Service for Prometheus console and implement standard JWT-based authentication.
Prerequisites
You have created a Prometheus instance. For more information, see:
Step 1: Get the Push Gateway endpoint
Log on to the Cloud Monitor console. In the left navigation pane, choose . The Instances page appears.
-
In the menu bar at the top of the page, select the region where your Prometheus instance resides. In the Actions column of the destination cluster, click Settings.
-
On the Settings tab, find the Pushgateway URL section and copy the public URL.
This section also provides a private network Push Gateway URL. Choose the appropriate URL based on your actual network environment.
Step 2: Report data
V1 version
Push data using an open source SDK
-
The data protocol currently supports Text Format and Protobuf Delimited. It does not support Protobuf Text, Protobuf Compact-Text, or OpenMetrics. SDKs typically default to Protobuf Delimited.
-
Metric HELP metadata does not support Chinese characters. If HELP contains Chinese characters, data reporting fails.
The following Go and Java examples show how to push metrics using an open source SDK.
Go example:
completionTime := prometheus.NewGauge(prometheus.GaugeOpts{
Name: "db_backup_last_completion_timestamp_seconds",
Help: "The timestamp of the last successful completion of a DB backup.",
})
completionTime.SetToCurrentTime()
url : = "https://cn-hangzhou.arms.aliyuncs.com/prometheus/52b12ea9cf4bb9e35****/16727530178****/1df8lj***/cn-hangzhou/api/v2"
pusher := push.New(url, "test").
Collector(completionTime).Client(http.DefaultClient).
Grouping("key1", "test1").Grouping("key2", "dfdf/sdsd/").
Format(expfmt.FmtProtoDelim)
if err := pusher.Push(); err != nil {
fmt.Println("Could not push completion time to PushGateway: ", err)
}
Java example:
CollectorRegistry registry = new CollectorRegistry();
Gauge duration = Gauge.build()
.name("my_batch_job_duration_seconds").help("Duration of my batch job in seconds.").register(registry);
Gauge.Timer durationTimer = duration.startTimer();
try {
// Your code here.
// This is only added to the registry after success,
// so that a previous success in the Pushgateway isn't overwritten on failure.
Gauge lastSuccess = Gauge.build()
.name("my_batch_job_last_success").help("Last time my batch job succeeded, in unixtime.").register(registry);
lastSuccess.setToCurrentTime();
} finally {
durationTimer.setDuration();
PushGateway pg = new PushGateway(new URL("https://cn-hangzhou.arms.aliyuncs.com/prometheus/52b12ea9cf4bb9e35****/16727530178****/1df8lj***/cn-hangzhou/api/v2"));
pg.pushAdd(registry, "my_batch_job");
}
-
When you use an open source SDK and enter the Pushgateway endpoint obtained from the Prometheus Monitoring console, the system automatically appends a suffix such as
/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}. If you do not use an open source SDK, you must append this suffix yourself. Otherwise, you receive a 404 error. -
If you push data to a shared tenant cluster of Managed Service for Prometheus, all metrics must include the label
tenant_userid=****. The tag value must be the Alibaba Cloud account ID (that is, the root account ID) to which the metric belongs. This distinguishes metric ownership.
Push data using the curl command
The request type application/x-www-form-urlencoded is not supported. In the curl command, add a header that specifies Content-Type: text/plain; version=0.0.4; charset=utf-8.
echo "some_metric 3.14" | curl -H "Content-Type: text/plain; version=0.0.4; charset=utf-8" --data-binary @- https://cn-hangzhou.arms.aliyuncs.com/prometheus/51bbea9ck41b9e35****/16727530178****/1df8lj***/cn-hangzhou/api/v2/metrics/job/job_name/label_key_1/label_value_1/label_key_2/label_value_2
-
You can append multiple labels to the URL, but keep the total URL length within limits.
-
Replace “job_name” in the URL with your actual job name.
Add data protection configuration (optional)
-
Get a token: In the navigation pane on the left, choose Settings. On the Settings tab, in the Token section, click Generate Token.
-
Pass the token: After generating the token, use one of the following methods to pass it.
-
Method 1: Set the token in the client request header to push data normally. Otherwise, the system rejects the write request. Use the following header format:
Authorization: Bearer <token> -
Method 2: The Pushgateway SDK supports only basic Auth and does not support JWT. To use the SDK with authentication, use the BasicAuth interface and set the password to the token. The server side supports this authentication method. Method 1 requires additional development effort. The following example uses the Go SDK.
pusher := push.New(url, "test"). Collector(completionTime).Client(http.DefaultClient). Grouping("key1", "test1").Grouping("key2", "dfdf/sdsd/"). .BasicAuth("admin", "actual token value"). Format(expfmt.FmtProtoDelim)
-
V2 version
1. Grant CMS read and write permission to a RAM user
If your Alibaba Cloud Prometheus instance was created by an Alibaba Cloud account and you need to use a RAM user’s AccessKey ID and AccessKey secret for remote read and write operations, first grant the RAM user CMS read and write permission.
-
Log on to the Resource Access Management (RAM) console using an Alibaba Cloud account or a RAM administrator. In the navigation pane on the left, choose .
-
On the Authorization page, click Create Authorization. On the Create Authorization page, configure the following parameters.
Parameter
Description
Resource Scope
Select a resource scope as needed.
Principal
Select the RAM user that you want to authorize.
Policy
Select AliyunPrometheusMetricWriteAccess or AliyunCloudMonitorFullAccess.
-
Click OK, and then click Close.
2. Push data
Push data using an open source SDK
-
The data protocol currently supports Text Format and Protobuf Delimited. It does not support Protobuf Text, Protobuf Compact-Text, or OpenMetrics. SDKs typically default to Protobuf Delimited.
-
Metric HELP metadata does not support Chinese characters. If HELP contains Chinese characters, data reporting fails.
The following Go and Java examples show how to push metrics using an open source SDK.
Go example:
completionTime := prometheus.NewGauge(prometheus.GaugeOpts{
Name: "db_backup_last_completion_timestamp_seconds",
Help: "The timestamp of the last successful completion of a DB backup.",
})
completionTime.SetToCurrentTime()
url : = "https://k8s-log-c8b5d0a212ffa41cxxxx.cn-hangzhou.log.aliyuncs.com/prometheus/k8s-log-c8b5d0a212ffa41c0a5xxxx/aliyun-prom-c8b5d0a212ffa41c0xxxx/api/v1/pushgateway"
pusher := push.New(url, "test").
Collector(completionTime).Client(http.DefaultClient).
Grouping("key1", "test1").Grouping("key2", "dfdf/sdsd/").
BasicAuth("ak", "sk").
Format(expfmt.FmtProtoDelim)
if err := pusher.Push(); err != nil {
fmt.Println("Could not push completion time to PushGateway: ", err)
}
Java example:
CollectorRegistry registry = new CollectorRegistry();
Gauge duration = Gauge.build()
.name("my_batch_job_duration_seconds").help("Duration of my batch job in seconds.").register(registry);
Gauge.Timer durationTimer = duration.startTimer();
try {
// Your code here.
// This is only added to the registry after success,
// so that a previous success in the Pushgateway isn't overwritten on failure.
Gauge lastSuccess = Gauge.build()
.name("my_batch_job_last_success").help("Last time my batch job succeeded, in unixtime.").register(registry);
lastSuccess.setToCurrentTime();
} finally {
durationTimer.setDuration();
PushGateway pg = new PushGateway(new URL("https://k8s-log-c8b5d0a212ffa41cxxxx.cn-hangzhou.log.aliyuncs.com/prometheus/k8s-log-c8b5d0a212ffa41c0a5xxxx/aliyun-prom-c8b5d0a212ffa41c0xxxx/api/v1/pushgateway"));
pg.setConnectionFactory(new BasicAuthHttpConnectionFactory("ak", "sk"));
pg.pushAdd(registry, "my_batch_job");
}
-
When you use an open source SDK and enter the Pushgateway endpoint obtained from the Prometheus Monitoring console, the system automatically appends a suffix such as
/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}. If you do not use an open source SDK, you must append this suffix yourself. Otherwise, you receive a 404 error. -
If you push data to a shared tenant cluster of Managed Service for Prometheus, all metrics must include the label
tenant_userid=****. The tag value must be the Alibaba Cloud account ID (that is, the root account ID) to which the metric belongs. This distinguishes metric ownership. -
The API operation supports Security Token Service (STS) authentication. In this case, the password in BasicAuth uses the format
{AccessKey Secret}${STS Token}.
Push data using the curl command
The request type application/x-www-form-urlencoded is not supported. In the curl command, add a header that specifies Content-Type: text/plain; version=0.0.4; charset=utf-8.
echo "some_metric 3.14" | curl -u 'ak:sk' -H "Content-Type: text/plain; version=0.0.4; charset=utf-8" --data-binary @- https://k8s-log-c8b5d0a212ffa41cxxxx.cn-hangzhou.log.aliyuncs.com/prometheus/k8s-log-c8b5d0a212ffa41c0a5xxxx/aliyun-prom-c8b5d0a212ffa41c0xxxx/api/v1/pushgateway/metrics/job/job_name/label_key_1/label_value_1/label_key_2/label_value_2
-
You can append multiple labels to the URL, but keep the total URL length within limits.
-
Replace “job_name” in the URL with your actual job name.
Verify results
Use Grafana to verify that data was pushed successfully.
-
Go to the ApiServer dashboard: In the navigation pane on the left, select Dashboards. Then click the ApiServer dashboard hyperlink. The system redirects you to the dashboard page.
-
Select Explore to view data: On the dashboard page, hover over the
icon in the navigation pane on the left. In the pop-up box, click Explore. On the right side of the Explore page, use the dropdown menu to select the corresponding Explore instance and check whether data was pushed successfully.In the Metrics field, enter the name of the metric you pushed (such as
db_backup_last_completion_timestamp_seconds). Set Query type to Range. After clicking Run, if a curve appears in the Graph area below, data was pushed successfully.