Data Transmission Service (DTS) migrates data between ApsaraDB RDS for MariaDB instances with no application downtime. You can run schema migration, full data migration, and incremental data migration together to keep your applications running throughout the process.
Billing
| Migration type | Fee |
|---|---|
| Full data migration | Free of charge |
| Incremental data migration | Charged. For more information, see Billing overview. |
Migration types
| Type | Description |
|---|---|
| Schema migration | DTS migrates schema definitions, including foreign keys, from the source instance to the destination instance. |
| Full data migration | DTS migrates all existing data from the source instance to the destination instance. |
| Incremental data migration | After full data migration completes, DTS continuously replicates incremental changes to the destination instance, keeping your applications running without downtime during migration. |
To migrate data without stopping your applications, select all three migration types.
SQL operations supported for incremental data migration
| Type | Supported statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | ALTER TABLE, ALTER VIEW, CREATE FUNCTION, CREATE INDEX, CREATE PROCEDURE, CREATE TABLE, CREATE VIEW, DROP INDEX, DROP TABLE, RENAME TABLE, TRUNCATE TABLE |
RENAME TABLE may cause data inconsistency. If you rename a table during migration and selected only that table as the migration object, the renamed table's data is not migrated to the destination. To avoid this, select the entire database as the migration object and make sure both the original and renamed table names belong to databases included in the migration scope.
Prerequisites
Before you begin, make sure you have:
A destination RDS MariaDB instance. For more information, see Create an RDS MariaDB instance
A destination instance with storage space larger than the space used by the source instance
Database accounts on both instances with the required permissions: For more information, see Create an account and Modify or reset account permissions
Database Schema migration Full data migration Incremental data migration Source RDS MariaDB instance SELECT SELECT Read and write Destination RDS MariaDB instance Read and write Read and write Read and write (Incremental migration only) Binary logging enabled on the source instance with the following parameters set: Binary log retention requirements:
ImportantSet the retention period before starting the task. If DTS cannot obtain the binary logs because the retention period is too short, the task fails. Issues caused by insufficient retention are not covered by the DTS Service-Level Agreement (SLA).
Parameter Required value binlog_formatrowbinlog_row_imagefullMigration type combination Minimum retention period Incremental migration only More than 24 hours Full data migration + incremental data migration At least 7 days
Limitations
Source instance limits (all migration types)
The source instance server must have enough outbound bandwidth. Insufficient bandwidth reduces migration speed.
Tables to be migrated must have primary keys or UNIQUE constraints, and all fields must be unique. Otherwise, duplicate data may appear in the destination.
Do not perform DDL operations that change database or table schemas during schema migration or full data migration. The task fails if you do.
During full data migration, DTS queries the source database. This generates metadata locks that may block DDL operations on the source.
During full data migration and incremental data migration, DTS temporarily disables the constraint check and cascade operations on foreign keys at the session level. If you perform cascade update and delete operations on the source database during data migration, data inconsistency may occur.
Source instance limits (full data migration without incremental migration)
Do not write new data to the source database during migration. To maintain real-time data consistency, use schema migration, full data migration, and incremental data migration together.
Source instance limits (table-level migration with name mapping)
A single migration task supports a maximum of 1,000 tables. If the number exceeds this limit, split the tables across multiple tasks, or configure a task to migrate the entire database instead.
Other limits
Run the migration during off-peak hours. Full data migration consumes read and write resources on both the source and destination instances, increasing database load.
After full data migration, the destination instance's storage space is larger than the source. Concurrent INSERT operations during migration cause table fragmentation in the destination.
For FLOAT or DOUBLE columns, DTS reads values using
ROUND(COLUMN, PRECISION). If precision is not explicitly defined, DTS uses 38 for FLOAT and 308 for DOUBLE. Verify that this precision meets your requirements.DTS attempts to resume tasks that failed within the last 7 days. Before switching workloads to the destination, stop or release any failed tasks. Alternatively, run the
REVOKEstatement to revoke write permissions from the DTS accounts on the destination database. Otherwise, a resumed failed task may overwrite destination data with source data.If DDL statements fail to write to the destination, the task continues running. Check the task logs for failed DDL statements. For more information, see Query task logs.
The source and destination instances must not have invisible columns. Invisible columns may cause the DTS instance to fail or data to be lost.
If an instance fails, DTS helpdesk attempts to recover it within 8 hours. During recovery, DTS may restart the instance or modify DTS instance parameters. Database parameters are not modified. For more information about which parameters may change, see Modify instance parameters.
Special cases
If the database name does not comply with RDS MariaDB naming conventions, create the database in the destination instance before configuring the migration task. DTS automatically creates databases only when names are compliant. For more information, see Create a database and an account.
Create a data migration task
Go to the Data Migration page.
DTS console: Log on to the DTS console. In the left-side navigation pane, click Data Migration. In the upper-left corner, select the region where the data migration instance resides.
DMS console: Log on to the DMS console. In the top navigation bar, move the pointer over Data + AI > DTS (DTS) > Data Migration. To the right of Data Migration Tasks, select the region where the data migration instance resides. > Note: The actual steps may vary based on the DMS console mode and layout. For more information, see Simple mode and Customize the layout and style of the DMS console.
Click Create Task.
If the page shows a New Configuration Page button in the upper-right corner, click it to switch to the new configuration page.
Skip this step if Back to Previous Version is displayed instead. Use the new configuration page.
Configure the source and destination databases. Task name DTS automatically generates a task name. Set a descriptive name to identify the task. The name does not need to be unique.
Source database
Parameter Value Select existing connection If the source instance is registered with DTS, select it from the drop-down list. DTS automatically fills in the remaining parameters. Otherwise, fill in the parameters below. For more information about registering instances, see Manage database connections. Database type Select Mariadb. Access method Select Cloud Instance. Instance region Select the region where the source RDS MariaDB instance resides. Replicate data across Alibaba Cloud accounts Select No to migrate data within the same Alibaba Cloud account. Instance ID Select the ID of the source RDS MariaDB instance. Database account Enter the database account for the source instance. See Prerequisites for required permissions. Database password Enter the password for the database account. Encryption The default connection method for RDS MariaDB is Non-encrypted Connection. Destination database
Parameter Value Select existing connection If the destination instance is registered with DTS, select it from the drop-down list. DTS automatically fills in the remaining parameters. Otherwise, fill in the parameters below. For more information, see Manage database connections. Database type Select Mariadb. Access method Select Cloud Instance. Instance region Select the region where the destination RDS MariaDB instance resides. RDS instance ID Select the ID of the destination RDS MariaDB instance. Database account Enter the database account for the destination instance. Database password Enter the password for the database account. Encryption The default connection method for RDS MariaDB is Non-encrypted Connection. Click Test Connectivity and Proceed.
Make sure the DTS server CIDR blocks are added to the security settings of both instances. For more information, see Add DTS server IP addresses to a whitelist.
Configure the objects to migrate on the Configure Objects page. Method to migrate triggers in source database Select how to migrate triggers. This parameter is available only when Schema Migration is selected. For more information, see Configure how to synchronize or migrate triggers. Source objects and selected objects Select the databases, tables, or columns to migrate from Source Objects and add them to Selected Objects. If you select tables or columns, DTS does not migrate views, triggers, or stored procedures. To rename migration objects, right-click an object in Selected Objects and use Map the name of a single object, or click Batch Edit to map multiple object names at once. To filter rows, right-click a table in Selected Objects and specify a WHERE clause. For more information, see Set filter conditions.
- If you skip Schema Migration, create the target databases and tables in the destination before starting the task, and enable object name mapping in Selected Objects. - If you skip Incremental Data Migration, do not write new data to the source during migration to avoid data inconsistency.
Object name mapping may prevent other objects that depend on the renamed object from migrating successfully.
Migration types
Goal Selection Migrate schema and data only (service interruption required) Select Schema Migration and Full Data Migration. Migrate with minimal downtime Select Schema Migration, Full Data Migration, and Incremental Data Migration. Processing mode for conflicting tables
Mode Behavior Precheck and report errors Fails the precheck if any table in the destination has the same name as a source table. Use object name mapping to rename conflicting tables before retrying. Ignore errors and proceed Skips the precheck for conflicting table names. During full data migration, existing records in the destination are retained when a primary key conflict occurs. During incremental data migration, existing records are overwritten. If schemas differ between source and destination, only specific columns may be migrated, or the task may fail. Use with caution. Click Next: Advanced Settings and configure the following parameters.
Parameter Description Dedicated cluster for task scheduling By default, DTS uses a shared cluster. For higher stability, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster. Retry time for failed connections How long DTS retries failed connections after the task starts. Valid values: 10–1,440 minutes. Default: 720 minutes. Set this to greater than 30 minutes. If multiple tasks share the same source or destination database, the most recently set value takes effect. DTS charges you for the instance during retry periods. Retry time for other issues How long DTS retries failed DDL or DML operations. Valid values: 1–1,440 minutes. Default: 10 minutes. Set this to more than 10 minutes. This value must be less than the retry time for failed connections. Enable throttling for full data migration Limits the read/write load on both instances during full data migration. Configure QPS to the source database, RPS of full data migration, and Data migration speed (MB/s). Available only when Full Data Migration is selected. Enable throttling for incremental data migration Limits the load during incremental migration. Configure RPS of incremental data migration and Data migration speed (MB/s). Available only when Incremental Data Migration is selected. Environment tag An optional tag to identify the instance. Configure ETL Enables the extract, transform, and load (ETL) feature. Select Yes to enter data processing statements. For more information, see What is ETL? and Configure ETL in a data migration or data synchronization task. Monitoring and alerting Configures alerts for task failures or migration latency exceeding a threshold. Select Yes to set alert thresholds and notification contacts. For more information, see Configure monitoring and alerting. Click Next: Save Task Settings and Precheck.
- To preview the API parameters for this task configuration, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters. - DTS runs a precheck before starting the task. The task starts only after all precheck items pass. - If a precheck item fails, click View Details to see the cause and fix the issue, then run the precheck again. - If a precheck item triggers an alert that can be ignored, click Confirm Alert Details > Ignore > OK, then click Precheck Again. Ignoring alerts may lead to data inconsistency.
After Success Rate reaches 100%, click Next: Purchase Instance.
On the Purchase Instance page, configure the instance class.
Parameter Description Resource group The resource group for the data migration instance. Default: default resource group. For more information, see What is Resource Management? Instance class The instance class determines migration speed. Select a class based on your requirements. For more information, see Instance classes of data migration instances. Read and accept the Data Transmission Service (Pay-as-you-go) Service Terms, then click Buy and Start > OK.
Verify the migration
After the task starts, go to the Data Migration page to monitor progress.
Tasks without incremental data migration stop automatically when complete. The status shows Completed.
Tasks with incremental data migration run continuously. The status shows Running.