Lindorm:Migrate full data from InfluxDB to LindormTSDB

Prerequisites

• The version of InfluxDB is 1.8.13 or later.

• LindormTSDB is activated.

• LTS is activated. For more information, see Purchase the LTS service and log on to the LTS web UI.

Usage notes

• We recommend that your InfluxDB instance, Lindorm instance, and LTS instance belong to the same virtual private cloud (VPC) to ensure network connectivity. For more information, see Network connection.

• During the migration, data is read from the InfluxDB instance and is written to LindormTSDB. Large values of the following items may lead to heavy workloads and affect the system stability. Therefore, before data migration, evaluate the impact of data migration based on the specifications of the InfluxDB instance, workloads, and values of the following items.

  Note

  If the instance is overloaded, we recommend that you upgrade the instance configurations before data migration.

  • The number of time series in the InfluxDB instance

  • The total size of data that you want to migrate

  • The average frequency at which data in each time series is reported

  • The time range of the data that you want to migrate

  • The interval at which each data migration task is split

• By default, the timestamps used in InfluxDB are 19 bits in length, which indicate time values in nanoseconds. The timestamps used in LindormTSDB are 13 bits in length, which indicate time values in milliseconds. After data is migrated from InfluxDB to LindormTSDB, the timestamps of the data are truncated to 13 bits in length, which indicate time values in milliseconds.

• LTS allows you to migrate historical data within a specific time range to a Lindorm instance. However, LTS cannot guarantee the effect of incremental data migration. Therefore, we recommend that you write incremental data to both InfluxDB and Lindorm before you migrate historical data.

• You can configure the data retention policy to physically isolate data in InfluxDB. You can configure the hot and cold data separation policy and time-to-live (TTL) to physically isolate data in LindormTSDB. This way, the retention policy remains unchanged for a InfluxDB database from which data is migrated to LindormTSDB.

  Note

  For more information, see Configure data retention policies and Create data models.

• A data migration task does not run concurrent subtasks. You can create multiple data migration tasks based on the database retention policy and time range. This way, you can initiate multiple data migration tasks at the same time to improve migration concurrency.

• If you want to migrate data from a self-managed InfluxDB instance to LindormTSDB, contact Lindorm technical support (DingTalk ID: s0s3eg3).

Configure the data source

Configure and start a data migration task

1. Create a destination database and a destination table in LindormTSDB. The destination database corresponds to the source database in InfluxDB, and the destination table corresponds to the measurement in InfluxDB. For more information, see CREATE DATABASE and CREATE TABLE.

   Note

   Data is migrated only to the destination database and table that you create. If you set the schemaPolicy parameter to WEAK when you configure the LindormTSDB data source, the system automatically creates a destination table.

2. In the left-side navigation pane of the LTS web UI, choose Lindorm TSDB > History Data Sync.

3. On the LINDORM TSDB History Data Sync page, click create new job and specify the following parameters:

   • Source Cluster: Select the InfluxDB data source.

   • Target Cluster: Select the LindormTSDB data source.

   • table mapping: Specify the range of data that you want to migrate. Format:

     {
       "type": "shard",
       "source_database": "test.autogen",
       "sink_database": "test",
       "startTime": 1,
       "endTime": 0
     }

     Parameter	Description
     source_database	The source database and data retention policy in InfluxDB. Format: <Database name>.<Data retention policy name>. Example: test.autogen. Note If you specify only the database name, the system only migrates data whose retention policy is autogen. The effect is equivalent to <Database name>.autogen.
     sink_database	Set the parameter to the destination database that you create in LindormTSDB.
     startTime	The start of the time range within which you want to migrate data. Specify the time in the 13-digit UNIX timestamp format. Unit: milliseconds.
     endTime	The end of the time range within which you want to migrate data. The default value is 0, which specifies that you want to migrate data that is generated before the current timestamp. Specify the time in the 13-digit UNIX timestamp format. Unit: milliseconds.
     type	N/A

View the migration progress

1. In the left-side navigation pane, choose Lindorm TSDB > History Data Sync.

2. On the page that appears, click the task name to view the execution status of the task.