Cold data refers to data in database tables that is rarely updated and infrequently accessed. If you want to reduce costs, you can use the cold data archiving feature to move this data to low-cost Object Storage Service (OSS) and reduce storage expenses.
How it works
PolarDB for MySQL supports archiving data in four formats: CSV, ORC, IBD, and X-Engine. The archiving principles differ for each format, as described below:
|
CSV/ORC format |
IBD format |
|
Archive data manually or automatically. Data is converted to CSV or ORC format and stored as multiple files in OSS, then automatically deleted from PolarDB storage to reduce costs. Cluster nodes access archived data over the Alibaba Cloud internal network. For more information, see Manually archive cold data and Automatically archive cold data. Note
When you Archive a partitioned table, the following conditions must be met:
|
This solution integrates OSS with InnoDB, so no storage engine change is needed. You can manually migrate IBD files from PolarDB storage to OSS. Archived IBD files retain DML support and their original index structure. Source data is then automatically deleted from PolarDB storage to reduce costs. For more information, see Archive in IBD Format. Note
Archiving in IBD format is available in two editions, Basic Edition and Advanced Edition, and the underlying implementations differ: the Basic Edition encapsulates OSS access within InnoDB, while the Advanced Edition provides OSS hybrid storage capabilities based on shared storage PolarStore. The archiving syntax, supported archiving targets, and post-archiving operations differ between the two editions. Statements from one edition cannot be used with the other. |
Archiving format comparison
When archiving cold data, you can evaluate and select the appropriate format based on the following comparison.
-
Archiving standard tables, OSS external tables, and partitioned tables has specific limitations. Review these limitations before archiving to avoid impact on your business.
-
After cold data is archived, the data is stored in the default system-provided Object Storage Service (OSS) bucket, not in your own OSS. Currently, you can view the list of archived data only in the PolarDB console.
-
Partitioned table archiving methods:
-
Archive a partitioned table: Archives a partition in place. The data remains in the original table, but the storage medium for that partition changes from PolarDB (hot storage) to OSS (cold storage). The table becomes a hybrid partitioned table that contains both hot and cold partitions.
-
Archive a partition to an OSS external table: Moves the data of a partition out of the table and stores it in a new, independent OSS external table. The original partition is deleted from the source table.
-
|
Comparison item |
CSV |
ORC |
IBD |
X-Engine |
|
Open-source format |
Yes |
Yes |
No |
No |
|
Archiving method |
|
Manual archiving: |
Manual archiving:
|
|
|
Archiving speed |
Medium Note
Only single-threaded archiving is supported. |
Slow Note
Only single-threaded archiving is supported. |
Relatively fast Note
Both single-threaded and multi-threaded parallel archiving are supported. |
Fast Note
Data is archived to PolarStore storage. |
|
Query speed |
Note
Both single-threaded and multi-threaded reads are supported. |
Note
Only single-threaded reads are supported. |
Relatively fast Note
Both single-threaded and multi-threaded reads are supported. |
|
|
Transaction support |
No |
No |
Yes |
Yes |
|
Indexing capability |
No |
No |
Yes |
Yes |
|
How to modify archived data |
Archived tables in OSS are read-only. To modify the data, you must import it back from OSS to PolarDB storage. |
Archived tables support DML operations. |
Archived tables support DML operations. |
|
|
Storage space usage |
Same as an InnoDB table without indexes. |
For the same amount of data, ORC uses about 45% of the storage space that CSV uses. |
Same as an InnoDB IBD file. |
Compresses data to 10%–50% of its InnoDB size. The actual compression ratio depends on data characteristics. |
|
Backup and recovery |
Not supported. Note
|
Supported. |
||
|
Impact after archiving |
|
|
|
After archiving, you can query archived data without changing the access method. |
Applicable scope
-
Archiving to CSV format
-
The cluster edition is Cluster Edition:
-
Archive a standard table:
-
MySQL 8.0.1 with a revision of 8.0.1.1.47 or later.
-
MySQL 8.0.2 with a revision of 8.0.2.2.10 or later.
-
-
Archive a partitioned table:
-
MySQL 8.0.2 with a revision of 8.0.2.2.34.1 or later.
-
-
-
The cluster edition is Multi-master Cluster (Limitless) Edition:
-
MySQL 8.0.1 with a revision of 8.0.1.0.13 or later.
-
-
-
Archiving to ORC format
-
The cluster edition is Cluster Edition:
-
Archive a standard table: MySQL 8.0.2 with a revision of 8.0.2.2.30 or later.
-
Archive a partitioned table: MySQL 8.0.2 with a revision of 8.0.2.2.34.1 or later.
-
-
The cluster edition is Multi-master Cluster (Limitless) Edition: MySQL 8.0.2 with a revision of 8.0.2.2.30 or later.
-
-
Archiving to IBD format
-
Basic Edition: MySQL 8.0.1 with a revision of 8.0.1.1.51.2 or later.
-
Advanced Edition: MySQL 8.0.2 with a revision of 8.0.2.2.37 or later. The storage type must be PSL4 or PSL5. ESSD cloud disks are not supported.
-
-
Archiving to X-Engine format
-
Archive a standard table:
-
MySQL 8.0.1 with a revision of 8.0.1.1.31 or later.
-
MySQL 8.0.2 with a revision of 8.0.2.2.12 or later.
-
-
Archive a partitioned table: MySQL 8.0.2 with a revision of 8.0.2.2.12 or later.
-
Archive to an X-Engine column-store table: MySQL 8.0.2 with a revision of 8.0.2.2.34.1 or later.
-
Billing
Cold data is billed based on the storage capacity of cold data in OSS. The specific charges are as follows.
The Chinese mainland | China (Hong Kong) and other regions |
USD 0.0000325 per GB-hour | USD 0.0000455 per GB-hour |
For example, if you archive 100 GB of cold data for a cluster in the Chinese mainland, the hourly fee is 100 GB × USD 0.0000325 per GB-hour = USD 0.00325 per hour.
For more information about how to view the amount of archived cold data, see View cold data archiving information.
How to use
For more information, see Used cold data archiving.