All Products
Search
Document Center

Data Management:Overview

Last Updated:Jun 27, 2024

In the use of an online database, an increasing amount of data is stored in the database. The data that is infrequently accessed occupies much storage space and affects the query performance and business operation. In this case, you can use the data archiving feature provided by Data Management (DMS) to periodically archive data of specific tables in the database to other databases or storage space. You can also perform other operations to reduce the online storage cost. For example, you can delete the archived data from the source tables and optimize the storage usage of the source tables.

Background information

On e-commerce platforms, a large amount of order data is generated every day, and the need for access to these order data diminishes over time. To improve database performance and free up storage space, you can use the data archiving feature of DMS to periodically archive data.

Usage notes

  • The business that is running may be affected when you use the data archiving feature.

    Important

    To minimize the impact, we recommend that you use the data archiving feature during off-peak hours.

  • The data archiving feature is available only in the Singapore and Indonesia (Jakarta) regions.

  • You cannot archive data from tables that contain virtual columns.

  • You cannot archive data from a database hosted on a third-party cloud service or a self-managed database in the DMS console or by using API operations.

Comparison between data archiving of DMS and data migration of DTS

  • Features

    • Data archiving of DMS allows you to configure filter conditions to archive infrequently accessed data to other storage space. This reduces the load on the primary database and improves database performance.

    • Data migration of Data Transmission Service (DTS) allows you to migrate data between homogeneous and heterogeneous data sources. For more information, see Overview of data migration scenarios.

  • Use scenarios

    • Data archiving of DMS allows you to archive infrequently accessed data to other databases or storage space on a regular basis or at a time.

    • Data migration of DTS is applicable to scenarios such as data migration to Alibaba Cloud, data migration between databases within Alibaba Cloud, and database splitting or scale-out.

Comparison among different categories of archiving destinations

Databases

Item

Lindorm

AnalyticDB for MySQL V3.0

AnalyticDB for PostgreSQL

ApsaraDB RDS for MySQL

PolarDB for MySQL

Supported source database types

  • MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, and AnalyticDB for MySQL V3.0

  • PostgreSQL: ApsaraDB RDS for PostgreSQL and PolarDB for PostgreSQL

  • PolarDB-X

Note

The account of a MySQL database must have the REPLICATION CLIENT permission.

Billing

You can use the data archiving feature free of charge. However, you may be charged when you purchase and use the destination instance.

Deletion of archived data from sources

After you archive data to a destination database, DMS can automatically delete the archived data from the source tables based on your settings. You no longer need to delete the source data after the data is archived. This reduces data risks.

  • The source table data is deleted without the lock of source tables. The process is stable and fast and does not involve large transactions.

  • The source data is cached in a local temporary table. You can restore the cached data if necessary.

Query of archived data

  • You can query the archived data in the DMS console.

  • You can also query the archived data in the corresponding database engine.

Configuration of the data archiving destination

  • You must specify a destination instance.

  • You do not need to specify a destination database or destination table. DMS automatically creates a destination database and a destination table based on the names of the source database and source table.

Change of the table schema

A column is added to the archived tables in the destination to store information such as the ticket ID and archiving time. Data in the tables can be used as usual.

Usage

  • This engine type supports the analysis of archived data.

  • This engine type can reduce online storage costs.

  • These engine types support the analysis of archived data and deliver better performance in the analysis.

  • These engine types can reduce online storage costs in an efficient manner.

Data control

These engine types provide the strongest data control. DMS can archive data to a database instance that you manage. You can flexibly process data by interacting with the databases.

Storage space

Item

Dedicated storage space (recommended)

Custom OSS bucket

OSS bucket connected to DBS

Supported source database types

  • MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, and AnalyticDB for MySQL V3.0

  • PostgreSQL: ApsaraDB RDS for PostgreSQL and PolarDB for PostgreSQL

  • MongoDB 3.6 and later

Note

The account of a MySQL database must have the REPLICATION CLIENT permission.

  • MySQL: ApsaraDB RDS for MySQL, PolarDB for MySQL, and AnalyticDB for MySQL V3.0

  • PostgreSQL: ApsaraDB RDS for PostgreSQL and PolarDB for PostgreSQL

  • MongoDB 3.6 and later

  • PolarDB-X

  • MySQL: ApsaraDB RDS for MySQL and PolarDB for MySQL

  • PolarDB for PostgreSQL

  • PolarDB-X

Billing

You are not charged for archiving data to dedicated storage space during public preview.

You can use the data archiving feature free of charge. However, you may be charged when you purchase and use Object Storage Service (OSS) buckets.

  • DMS creates a backup schedule in Database Backup (DBS). You are charged for data backup and storage based on the amount of data that is backed up. For more information about the billing, see Billing overview.

    Note

    By default, an xlarge backup schedule is created in DBS.

  • If you want to view archived data, you must activate Data Lake Analytics (DLA). For more information about the billing, see Billing overview.

Deletion of archived data from sources

After you archive data to a destination database, DMS can automatically delete the archived data from the source tables based on your settings. You no longer need to delete the source data after the data is archived. This reduces data risks.

  • The source table data is deleted without the lock of source tables. The process is stable and fast and does not involve large transactions.

  • The source data is cached in a local temporary table. You can restore the cached data if necessary.

The archived data cannot be automatically deleted from the source database. You must create a regular data change ticket to manually delete the archived data from the source database.

Query of archived data

You can query the archived data by using the logical data warehouses in DMS. You can also query the archived data by using the file merging feature to achieve higher efficiency.

You can query the archived data by using the logical data warehouses in DMS.

To query the archived data, you must activate DLA in DBS. Then, you can query the archived data in DLA.

Configuration of the data archiving destination

You do not need to focus on the archiving destination.

You must specify an OSS bucket.

You do not need to specify an OSS bucket. DMS automatically creates a backup schedule in DBS to archive data to OSS.

Change of the table schema

The table schema is not changed.

The table schema is not changed.

Note

If you query the archived data in DLA, the sequences of columns in the tables may be changed.

Usage

  • This type of storage space supports the analysis of archived data.

  • This type of storage space can reduce some online storage costs.

This type of storage space can reduce the most online storage costs.

This type of storage space can reduce many online storage costs.

Data control

This type of storage space provides strong data control. Data is archived and stored in DMS. You do not need to focus on the archiving destination.

This type of storage space provides strong data control. DMS can archive data to an OSS bucket that you purchase.

This type of storage space provides moderate data control. DMS can archive data to an OSS bucket connected to DBS.

References

FAQ

Q: If I select Clean up the archived data of the original table (delete-No Lock) when I configure parameters for data archiving, is the historical data still available for retrieval?

A: Yes, the historical data is still available for retrieval even if you select Clean up the archived data of the original table (delete-No Lock) when you configure parameters for data archiving. The data archiving feature regularly archives the data of large tables to other databases. You can query archived historical data in the destination instances to which the data is archived.