Time Series Database:Data migration solution for cloud InfluxDB

View the shard list

Before performing a backup, run the following InfluxQL statement to view the information for all shards and determine the shards to migrate:

SHOW SHARDS

From the result, obtain the id, database, and retention_policy fields for each shard. These fields are used as parameters in subsequent backup commands. You must perform the backup and restore operations on each shard individually.

backup

Run the backup command on the ECS instance hosting the self-hosted InfluxDB to back up data shard by shard. You must specify the database name, retention policy name, and shard ID for each backup.

• Syntax

  influxd backup -portable \
    -host <source instance VPC address:8088> \
    -db <database name> \
    -rp <retention policy name> \
    -shard <shard ID> \
    <backup directory>

• Parameters

  Parameter	Description
  -portable	Uses the portable backup format.
  -host	The VPC endpoint and backup port of the cloud TSDB for InfluxDB instance, in the format ts-xxx:8088.
  -db	The name of the database to back up.
  -rp	The name of the retention policy to back up.
  -shard	The ID of the shard to back up. Run SHOW SHARDS to obtain shard IDs.
  <backup directory>	The directory where backup files are saved, for example, /root/tmp/influx_backup.

• Example

  influxd backup -portable \
    -host ts-xxx.influxdata.tsdb.aliyuncs.com:8088 \
    -db example_db \
    -rp example_rp \
    -shard 123 \
    /root/tmp/influx_backup

  Note

  In this example, replace ts-xxx.influxdata.tsdb.aliyuncs.com:8088, example_db, example_rp, 123, and /root/tmp/influx_backup with values appropriate for your actual migration scope.

restore

Refer to the official documentation to restore data to an existing database.

1. On the ECS instance hosting the self-hosted InfluxDB, run the restore command to restore data to a temporary database.

   • Syntax

     influxd restore -portable \
       -db <backed-up database name> \
       -rp <backed-up retention policy name> \
       -shard <shard ID> \
       -newdb <temporary database name> \
       <backup directory>
     

   • Parameters

     Parameter	Description
     -portable	Reads the portable backup format.
     -db	The name of the backed-up database.
     -rp	The name of the backed-up retention policy.
     -shard	The ID of the backed-up shard.
     -newdb	The name of the temporary database to restore data into.
     <backup directory>	The directory where backup data is stored, for example, /root/tmp/influx_backup.

   • Example

     influxd restore -portable \
       -db example_db \
       -rp example_rp \
       -shard 123 \
       -newdb example_tmp_db \
       /root/tmp/influx_backup

     In this example, replace example_db, example_rp, 123, example_tmp_db, and /root/tmp/influx_backup with values appropriate for your actual backup content.

2. Use InfluxQL to query data from the temporary database and write it to the destination database.

   For large data volumes, SELECT INTO may produce incomplete data due to query timeouts. Confirm the following before execution:

   • If a query timeout is enabled (that is, INFLUXDB_COORDINATOR_QUERY_TIMEOUT is set), increase it to a sufficiently large value. InfluxDB does not enforce a query timeout by default.

   • For scenarios with large data volumes, run SELECT INTO in batches by time range to avoid timeouts on a single query.

     SELECT *
     INTO "example_db"."example_rp".:MEASUREMENT
     FROM "example_tmp_db"."example_rp"/.*/
     GROUP BY *

   /.*/ is InfluxQL regular expression syntax that matches all measurements.

3. Verify data integrity. Query the row counts in both the temporary database and the destination database. Confirm that the data is consistent before proceeding to the next step.

   SELECT COUNT(*) FROM "example_tmp_db"."example_rp"/.*/
   SELECT COUNT(*) FROM "example_db"."example_rp"/.*/

4. Drop the temporary database.

   DROP DATABASE "example_tmp_db";