Migrate historical data from self-managed Prometheus to Managed Service for Prometheus - Managed Service for Prometheus

Prerequisites

Data source: Your self-managed open-source Prometheus instance must use TSDB, which is the default storage engine.
Managed Service for Prometheus instance: Ensure the destination Managed Service for Prometheus instance is running.
Network connectivity: Your self-managed open-source Prometheus server must be able to access the Remote Write endpoint of your Managed Service for Prometheus instance.
Tool dependency: You must install the promtool provided by Alibaba Cloud.

Migration steps

Step 1: Get a Managed Service for Prometheus instance

Log on to the Alibaba Cloud Management Console.
Log on to the Cloud Monitor console or the Managed Service for Prometheus console, select an existing Prometheus instance, or create a new one.

When you create the instance, select a retention period that is longer than the historical retention period of your self-managed open-source Prometheus.
Important
Evaluate your local data volume and scale out the Log Service MetricStore shards for your Prometheus instance accordingly.
- Data volume > 1 GB: Scale out to 4 shards.
- Data volume > 10 GB: Scale out to 16 shards.
- Data volume > 100 GB: Scale out to 64 shards.
- Data volume < 1 TB: Scale out to 256 shards.
If you have a large volume of data, submit a ticket for assistance.
Obtain the write endpoint of the instance.
- Click the name of the destination instance to open the Settings page and obtain the public Remote Write URL. If your self-managed service runs on Alibaba Cloud, you can also obtain the internal URL.
- For authentication, you can assign the AliyunPrometheusMetricWriteAccess role to a RAM user and generate an AccessKey pair (AK/SK) to use as credentials. Alternatively, on the instance settings page, you can enable password-free write access for 0.0.0.0. In the Access Address section at the bottom of the page, copy the Remote Write URL for the public or internal network from the Network column.

Step 2: Migrate with `promtool migrate`

Note

promtool migrate is an official Managed Service for Prometheus tool that migrates local TSDB data to the cloud.

Prepare for the migration.

Data backup: Back up the data/ directory (the TSDB storage path) of your self-managed Prometheus before migration.

Install promtool.

curl -fsSL "https://o11y-addon-hangzhou-public.oss-cn-hangzhou.aliyuncs.com/share/promtool/install.sh" | bash

View the help information for the promtool command.

# View all supported tool commands.
promtool --help
# View data migration parameters.
promtool migrate --help

Run the migration command.

promtool migrate \
  --db_dir=/path/to/local/prometheus/data \  # The TSDB data directory of your self-managed open-source Prometheus.
  --output=rw    # Write data by using Remote Write.
  --rw_url=<Remote_Write_URL> \       # Example: https://xxx.region.aliyuncs.com/api/v1/write
  --rw_username=<AccessKey_ID> \            # The AccessKey ID for your Managed Service for Prometheus instance.
  --rw_password=<AccessKey_Secret> \        # The AccessKey Secret for your Managed Service for Prometheus instance.
  --min_time=START_TIMESTAMP \               # Optional: The start time for the migration (UNIX timestamp in milliseconds).
  --max_time=END_TIMESTAMP \                 # Optional: The end time for the migration (UNIX timestamp in milliseconds).

Parameter	Description	Example
`--db_dir`	The path to the TSDB data directory for your self-managed open-source Prometheus. This parameter is required.	`/opt/prometheus/data`
`--rw_url`	The Remote Write endpoint of your Managed Service for Prometheus instance. This parameter is required.	`https://xxx.region.aliyuncs.com/api/v1/write`
`--output`	The data output method.	Valid values: `rw` or `stdout`
`--rw_username`	The AccessKeyID of your Managed Service for Prometheus instance. This parameter is required if password-free write access is not enabled.	`LTAI5tXXXXXXX`
`--rw_password`	The AccessKeySecret of your Managed Service for Prometheus instance. This parameter is required if password-free write access is not enabled.	`Co4XXXXXXXXX`
`--min_time`	The start time for data migration (UNIX timestamp in milliseconds). Defaults to 0, which starts the migration from the earliest data.	`1609433200000`
`--max_time`	The end time for data migration (UNIX timestamp in milliseconds). Defaults to the time the migration starts.	`1640972400000`
`--match`	Filters the metrics to be migrated. Regular expressions are supported.	`'__name__=~"http_*"'`
`--rw_batch_size`	The number of samples sent in each batch. You can adjust this parameter to optimize performance.	`1000`

Verify the migration result.
1. Check the Managed Service for Prometheus instance.
  1. Log on to the Cloud Monitor console or the Managed Service for Prometheus console. On the Instance List page, click the name of the destination instance.
  2. In the navigation pane on the left, click Metric Management. On the Metrics Explorer tab, run a PromQL query.
  3. Confirm that the time range of the historical data matches that of your self-managed open-source Prometheus.
2. Verify data integrity.
  
  Select key metrics, such as up and node_cpu_seconds_total, to verify that the data volume is consistent before and after the migration.

Usage notes

Data filtering

Use the --min_time, --max_time, and --match parameters to limit the migration to a specific time range or set of metrics and avoid redundant data transfer.
Performance optimization
- Adjust --rw_batch_size to increase migration speed. Set the value based on network bandwidth and Alibaba Cloud instance load. The recommended range is 500 to 2000.
- For large data volumes, split the migration into time-based batches, such as weekly or monthly.
Security

Avoid exposing --rw_password in the command line. Use environment variables or configuration files to pass sensitive information instead.
Network and stability

Ensure network stability during migration. If the migration fails, use the --min_time parameter to resume from the point of interruption.

FAQ

Q1: What do I do if an authentication failure occurs during the migration?

Check that --rw_username and --rw_password match the Access Key of your Alibaba Cloud instance.
Confirm that the Remote Write URL is correct and that the Remote Write endpoint is enabled for your Managed Service for Prometheus instance.

Q2: What do I do if the migration is too slow?

Increase the --rw_batch_size value, such as --rw_batch_size=1000.
Ensure that there is sufficient network bandwidth between your self-managed open-source Prometheus server and the Managed Service for Prometheus instance.

Q3: What do I do if data is not displayed after the migration?

Wait a few minutes for the Managed Service for Prometheus instance to process and index the migrated data.
Check whether the retention period of the Prometheus instance is sufficient.

Timestamp conversion tool

Use the following command to convert a date to a UNIX timestamp in milliseconds.

date -d "2020-01-01 00:00:00" +%s%3N
# Example output: 1577836800000 (corresponds to 2020-01-01 00:00:00 UTC)