Migrate historical data from a self-managed open-source Prometheus to Managed Service for Prometheus - Managed Service for Prometheus

This topic describes how to migrate historical data from a self-managed open-source Prometheus to Managed Service for Prometheus.

Prerequisites

Data source: Your self-managed open-source Prometheus instance must use Time Series Database (TSDB), which is the default storage engine.
Managed Service for Prometheus instance: Ensure that the destination Managed Service for Prometheus instance is running.
Network connectivity: Your self-managed Prometheus server must be able to access the Remote Write endpoint of Managed Service for Prometheus.
Tool dependency: You must install the promtool tool provided by Alibaba Cloud.

Migration steps

Step 1: Create or obtain a Managed Service for Prometheus instance

Log on to the Alibaba Cloud Management Console.
Log on to the Cloud Monitor console or the Managed Service for Prometheus console. Select an existing Prometheus instance or create a new one.
When you create the instance, select a storage duration that is longer than the historical data retention period of your self-managed Prometheus.
Important
First, evaluate your local data volume and scale out the Simple Log Service MetricStore shards for your Prometheus instance.
- Data volume > 1 GB: Scale out to 4 shards.
- Data volume > 10 GB: Scale out to 16 shards.
- Data volume > 100 GB: Scale out to 64 shards.
- Data volume < 1 TB: Scale out to 256 shards.
If the data volume is large, submit a ticket for assistance.
Obtain the write endpoint of the instance.
- Click the name of the destination instance. This opens the Settings page, where you can obtain the public Remote Write URL. If your self-managed service runs on Alibaba Cloud, you can also obtain the internal URL.
- For authentication, you can assign the AliyunPrometheusMetricWriteAccess role to a RAM user to grant write permissions. Then, you can generate an AccessKey pair to use as the username and password. Alternatively, you can go to the instance settings page to enable password-free write access from 0.0.0.0.

Step 2: Use `promtool migrate` to perform the migration

Note

The promtool migrate is an official tool provided by Managed Service for Prometheus. This tool is used to migrate local TSDB data to Managed Service for Prometheus.

Prepare for the migration.
- Back up data: Before the migration, back up the data/ directory of your self-managed Prometheus. This directory is the TSDB storage path.
- Install promtool.
```
curl -fsSL "https://o11y-addon-hangzhou-public.oss-cn-hangzhou.aliyuncs.com/share/promtool/install.sh" | bash
```
- View the help information for the promtool command.
```
# View all supported tool commands
promtool --help

# View data migration parameters
promtool migrate --help
```

Run the migration command.

promtool migrate \
  --db_dir=/path/to/local/prometheus/data \  # The TSDB data directory of your self-managed Prometheus
  --output=rw    # Write data using the Remote Write method
  --rw_url=<Alibaba Cloud Remote Write URL> \       # Example: https://xxx.region.aliyuncs.com/api/v1/write
  --rw_username=<Access Key ID> \            # The AccessKey ID for your Alibaba Cloud instance
  --rw_password=<Access Key Secret> \        # The AccessKey Secret for your Alibaba Cloud instance
  --min_time=START_TIMESTAMP \               # Optional: The start time for the migration (UNIX timestamp in milliseconds)
  --max_time=END_TIMESTAMP \                 # Optional: The end time for the migration (UNIX timestamp in milliseconds)

Parameter	Description	Example
`--db_dir`	The path of the TSDB data directory for your self-managed Prometheus. This parameter is required.	`/opt/prometheus/data`
`--rw_url`	The Remote Write address of your Managed Service for Prometheus instance. This parameter is required.	`https://xxx.region.aliyuncs.com/api/v1/write`
`--output`	The data output method.	Valid values: `rw` and `stdout`
`--rw_username`	The AccessKey ID for your Alibaba Cloud instance. This parameter is required if password-free access is not enabled.	`LTAI5tXXXXXXX`
`--rw_password`	The AccessKey Secret for your Alibaba Cloud instance. This parameter is required if password-free access is not enabled.	`Co4XXXXXXXXX`
`--min_time`	The start time for data migration (UNIX timestamp in milliseconds). The default value is 0, which means the migration starts from the earliest data.	`1609433200000`
`--max_time`	The end time for data migration (UNIX timestamp in milliseconds). The default value is the time when the migration starts.	`1640972400000`
`--match`	Filters the metrics to be migrated. Regular expressions are supported.	`'__name__=~"http_*"'`
`--rw_batch_size`	The number of samples sent in each batch. You can adjust this parameter to optimize performance.	`1000`

Verify the migration result.
1. Check the Managed Service for Prometheus instance.
  1. Log on to the Cloud Monitor console or the Managed Service for Prometheus console. On the Instances page, click the name of the destination instance.
  2. In the navigation pane on the left, click Metric Management. On the Metric Explorer tab, run a Prometheus Query Language (PromQL) query.
  3. Confirm that the time range of the historical data matches that of your self-managed Prometheus.
2. Compare data integrity.
  Select key metrics, such as up and node_cpu_seconds_total, and compare the data before and after the migration to verify consistency.

Notes

Data filtering
To migrate data for a specific time period or for specific metrics, use the --min_time, --max_time, and --match parameters to precisely control the migration and avoid redundant data transfer.
Performance optimization
- Adjust --rw_batch_size: A larger value can increase the migration speed. However, you must adjust the value based on your network bandwidth and the load on the Managed Service for Prometheus instance. The recommended value is between 500 and 2000.
- Migrate in batches: If the data volume is large, perform the migration in batches based on time periods, such as weekly or monthly.
Security
Avoid exposing the --rw_password directly in the command line. Use environment variables or configuration files to pass sensitive information.
Network and stability
Ensure a stable network connection during the migration. If the migration fails, you can resume it from the point of interruption by specifying the --min_time parameter.

FAQs

What do I do if authentication fails during migration?

Verify that the --rw_username and --rw_password parameters match the AccessKey pair of your Managed Service for Prometheus instance.
Confirm that the Remote Write URL is correct and that the Remote Write endpoint is enabled for your Managed Service for Prometheus instance.

What do I do if the migration is too slow?

Increase the value of --rw_batch_size, for example, --rw_batch_size=1000.
Ensure that there is sufficient network bandwidth between your self-managed Prometheus server and the Managed Service for Prometheus instance.

What do I do if data is not displayed after migration?

Wait a few minutes for the Managed Service for Prometheus instance to synchronize the data.
Verify that the retention period of the Prometheus instance is long enough.

Timestamp conversion tool

Use the following command to convert a date to a UNIX timestamp in milliseconds.

date -d "2020-01-01 00:00:00" +%s%3N
# Example output: 1577836800000 (corresponds to 2020-01-01 00:00:00 UTC)