Documentation
Home Page Data Management User Guide Solution Data archiving Archive data to Lindorm

Archive data to Lindorm

Updated at:
Copy as MD

Data Management (DMS) lets you archive historical records from relational databases to a Lindorm instance, reducing storage costs on production databases while keeping data accessible for queries.

The data archiving feature is available only in the Singapore and Indonesia (Jakarta) regions.

Prerequisites

Before you begin, make sure you have:

  • A supported source database:

    • MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, or AnalyticDB for MySQL V3.0

    • PostgreSQL: ApsaraDB RDS for PostgreSQL or PolarDB for PostgreSQL

    • PolarDB-X

  • The MySQL database account has the REPLICATION CLIENT permission (MySQL sources only).

  • Source tables have a primary key or a unique key.

  • A Lindorm instance running LindormTable V2.2.17 or later. To create one, see Create an instance.

  • The Lindorm instance is registered with DMS using Lindorm SQL as the database type. See Register an Alibaba Cloud database instance.

Tip: Add a timestamp column (such as gmt_modified) to source tables. This lets you use time-based filter conditions to select only the records you want to archive.

Usage notes

  • Periodic tasks require both the source and destination databases to be managed in Security Collaboration or Stable Change mode. One-time tasks support any control mode.

  • If you configure a periodic task and either database instance is in Stable Change mode, enable security hosting or switch to Security Collaboration mode to prevent task failures caused by instance logon expiration. See Security hosting and Change the control mode of an instance.

  • To run a recurring data archiving task, both the source and destination databases must be in Security Collaboration or Stable Change control mode. To run a data archiving task only once, no limit is imposed on the control mode.

    Note

    If your instances are in Stable Change control mode, enable security hosting or change the control mode to Security Collaboration before you configure a recurring data archiving task. This prevents task failures that are caused by instance logon expiration. For more information, see Enable security hosting and Change the control mode of an instance.

  • The data archiving feature is available only in the Singapore and Indonesia (Jakarta) regions.

Billing

You are charged for the destination Lindorm instance.

Archive data to a Lindorm instance

Step 1: Open the data archiving ticket form

  1. Log on to the DMS console V5.0.

  2. In the top navigation bar, choose Solution > Data Archiving.

    In simple mode, move the pointer over the 2023-01-28_15-57-17.png icon in the upper-left corner, then choose All functions > Solution > Data Archiving.

  3. On the Data Archiving Tickets page, click Data Archiving in the upper-right corner.

Step 2: Configure the archiving ticket

On the Ticket Application page, fill in the following parameters.

Basic settings

Parameter Required Description
Task Name Yes A descriptive name for the archiving task.
Archiving Destination Yes Select Lindorm.
Lindorm Instance Yes The destination Lindorm instance to archive data to.
Source Database Yes The source database to archive data from.
Operation Mode Yes How the task runs after the ticket is approved: Single execution runs the task once; Cyclic scheduling runs it on a recurring schedule. If you select Cyclic scheduling, configure the schedule in the Periodic scheduling section below.

Archiving configuration

In the Archiving Configuration section, select the source tables to archive.

For each table, you can optionally add a filter condition to limit which rows are archived. For example, gmt_modified<='${6_month_ago}' archives only rows modified more than six months ago.

To use time-based variables like ${6_month_ago} in filter conditions, define them first in the Variable Configuration section (see below), then reference them here.

Click Add to include more source tables.

Variable configuration (optional)

Define time variables to use as filter conditions. For example, create a variable named 6_month_ago in yyyy-MM-dd format with an offset of -6 months. If today is August 12, 2021, the variable resolves to 2021-02-11. For details, see Configure time variables.

Archive table mapping (optional)

By default, DMS creates destination tables with the same names and structure as the source tables, and automatically adds four metadata columns to each archived table:

Column Description
Archiving info The ticket number and the time when the data was archived
Database name The name of the source database
Table name The name of the source table
Instance ID The ID used when the source instance was registered with DMS

These extra columns do not affect queries on the original data.

To customize a destination table, click Edit in the Actions column and specify the table name, columns, shard keys, and partition keys.

Post-behavior (optional)

Choose what happens to the source data after archiving:

  • Clean up the archived data of the original table (delete-No Lock): DMS deletes the archived rows from the source table. Before deletion, DMS creates a temporary backup table in the source database — make sure the source database has enough storage space to hold this backup. After you verify the archived data is correct, create a regular data change ticket to drop the temporary backup tables.

  • If you do not select Clean up the archived data of the original table (delete-No Lock), the archived data is not deleted from the source tables. To free up space, manually delete the rows using a regular data change ticket and reclaim storage with a lock-free DDL ticket.

  • Verify Archived Data (This operation will be time-consuming.): DMS verifies the archived data during archiving, checking the table name, source database, and SQL statement count. View the results on the ticket details page.

Step 3: Submit and monitor the task

  1. Click Submit to submit the ticket.

  2. After the ticket is approved, DMS automatically starts the data archiving task.

  3. If the task fails, click Details in the Execute step to view the logs. For network or database connection failures, click Retry Breakpoint to resume the task.

Query archived data

Method 1: Use DMS

  1. On the Ticket Details page, go to Basic Information and click View next to Target Database. The SQL Console tab opens.

  2. In the left panel, go to the Table tab, double-click the target table, and click Execute to view the archived data.

The destination database and tables use the same names as the source database and tables.

Method 2: Use Lindorm directly

Query the archived data in LindormTable. See Data Query.

Periodic scheduling

When Operation Mode is set to Cyclic scheduling, configure the following parameters.

Parameter Description
Scheduling Cycle The recurrence frequency: Hour, Day, Week, or Month. If you select Hour, configure the Timed Scheduling parameter. If you select Day, configure the Specific Point in Time parameter. If you select Week or Month, configure the Specified Time and Specific Point in Time parameters.
Timed Scheduling Applies when Scheduling Cycle is Hour. Choose one of two methods: run at a fixed interval (set Starting Time, Intervals in hours, and End Time), or run at specific hours. Example: starting at 00:00, interval of 6 hours, ending at 20:59 — the task runs at 00:00, 06:00, 12:00, and 18:00.
Specified Time Applies when Scheduling Cycle is Week or Month. Select the days of the week or month on which the task runs.
Specific Point in Time The time of day the task runs on the specified days. Example: 02:55.
Cron Expression Auto-generated based on your scheduling settings.

What's next

Previous：Data archivingNext：Archive data to an AnalyticDB for MySQL V3.0 cluster