Lindorm:Migrate full data from TSDB to LindormTSDB

Prerequisites

Before you begin, make sure that you have:

• A Linux or macOS client with the following installed:

  • Java Development Kit (JDK) 1.8 or later

  • Python 2.x or 3.x

• A TSDB instance running version 2.7.4 or later

• A Lindorm instance with LindormTSDB activated. For details, see Create an instance

How the migration tool works

The tool processes data in the following sequence:

1. Reads all time series data from the TSDB instance and saves it to a local file.

2. Splits the migration task into time groups based on beginDateTime, endDateTime, and splitIntervalMs. Within each time group, splits the workload into read subtasks based on oidBatch. Each read subtask reads a batch of time series within the specified time range and sends the data to the write component.

3. After all read subtasks in a time group complete, records the time group ID, migration task ID, and task status in a list named internal_datax_job<jobName>.

4. The write component receives data from each read subtask and writes it to LindormTSDB using the multi-value data model.

Before you start

Configure the migration task

Save your migration configuration as a JSON file (for example, job.json). The configuration has three sections: task settings, reader (TSDB source), and writer (LindormTSDB destination).

100 × 604800000 / 3600000 = 16,800 data points per subtask

{
    "job": {
        "setting": {
            "speed": {
                "channel": 1
            },
            "errorLimit": {
                "record": 0,
                "percentage": 0.00
            }
        },
        "content": [
            {
                "reader": {
                    "name": "tsdbreader",
                    "parameter": {
                        "sinkDbType": "LINDORM-MIGRATION",
                        "endpoint": "ts-xxxx:3242",
                        "beginDateTime": "2022-5-2 00:00:00",
                        "endDateTime": "2022-7-2 00:00:00",
                        "splitIntervalMs": 86400000,
                        "jobName": "myjob",
                        "selfId": 1,
                        "jobIds": [1],
                        "oidPath": "{$myworkplace}/oidfile",
                        "oidBatch": 100,
                        "oidCache": true
                    }
                },
                "writer": {
                    "name": "tsdbwriter",
                    "parameter": {
                        "endpoint": "ld-xxxx:8242",
                        "multiField": true,
                        "batchSize": 500
                    }
                }
            }
        ]
    }
}

Run the migration

1. Download the migration tool.

2. Decompress the package:

   tar -zxvf tsdb2lindorm.tar.gz

3. Start the migration task:

   python datax/bin/datax.py --jvm="-Xms8G -Xmx8G" job.json > job.result

   Replace job with the name of your configuration file.

4. After the command finishes, check job.result for errors. If the file contains no error output, the migration completed successfully.

5. (Optional) If the task fails, query the task status list to identify which time groups succeeded and where to resume:

   curl -u <username>:<password> <tsdb-endpoint>:3242/api/mquery -XPOST -d '{
       "start": 1,
       "queries": [
           {
               "metric": "internal_datax_job<jobName>",
               "fields": [
                   {
                       "field": "*",
                       "aggregator": "none"
                   }
               ]
           }
       ]
   }'

   Replace the placeholders:

   Placeholder	Description	Example
   <username>:<password>	TSDB account credentials. See Manage accounts.	admin:mypassword
   <tsdb-endpoint>	TSDB instance ID	ts-****
   <jobName>	Migration task name	internal_datax_jobmyjob

   The response lists each time group with its completion status:

   Timestamp (endtime)	jobId (tag)	state (field)
   1651795199999 (2022-05-05 23:59:59.999)	3	ok
   1651795199999 (2022-05-05 23:59:59.999)	2	ok
   1651795199999 (2022-05-05 23:59:59.999)	1	ok
   1651881599999 (2022-05-06 23:59:59.999)	2	ok

   To resume without re-migrating completed time groups, update beginDateTime in job.json to the start of the first incomplete time group before restarting. In this example, set beginDateTime to 2022-05-06 00:00:00.

Query migrated data

TSDB data originally written with the single-value model is stored in LindormTSDB under the multi-value data model. Use the /api/mquery endpoint and the multi-value query format when reading this data.

The following example shows the query difference for a metric named test_metric:

Query in TSDB:

curl -u username:password ts-xxxxx:3242/api/query -XPOST -d '{
    "start": 1657004460,
    "queries": [
        {
            "aggregator": "none",
            "metric": "test_metric"
        }
    ]
}'

Response from TSDB:

[
    {
        "aggregateTags": [],
        "dps": {
            "1657004460": 1.0
        },
        "fieldName": "",
        "metric": "test_metric",
        "tags": {
            "tagkey1": "1"
        }
    }
]

Query in LindormTSDB:

curl -u username:password ld-xxxxx:8242/api/mquery -XPOST -d '{
    "start": 1657004460,
    "queries": [
        {
            "metric": "test_metric",
            "fields": [
                {
                    "field": "*",
                    "aggregator": "none"
                }
            ],
            "aggregator": "none"
        }
    ]
}'

Response from LindormTSDB:

[
  {
    "aggregatedTags": [],
    "columns": [
      "timestamp",
      "value"
    ],
    "metric": "test_metric",
    "tags": {
      "tagkey1": "1"
    },
    "values": [
      [
        1657004460000,
        1.0
      ]
    ]
  }
]

Key differences:

• Use /api/mquery instead of /api/query.

• Specify fields explicitly using the fields array.

• Timestamps in LindormTSDB responses are in milliseconds (13 digits), compared to seconds (10 digits) in TSDB.

Performance test results

Use the following results as a reference when planning your migration. Actual performance depends on your instance specification, data volume, and reporting frequency.

TSDB Basic Edition II (4 CPU cores, 8 GB memory) — 2 instances

TSDB Standard Edition I (8 CPU cores, 16 GB memory) — 2 instances

What's next

• To query migrated data using SQL, create a time series table in LindormTSDB before migrating. See the LindormTSDB documentation for instructions on creating time series tables.

• After migration, validate a sample of your data in LindormTSDB using the query examples in Query migrated data.