Data Transmission Service (DTS) lets you synchronize data from a self-managed MariaDB database to an ApsaraDB RDS for MariaDB instance in real time. This topic describes how to configure a synchronization task that covers schema synchronization, full data synchronization, and incremental data synchronization.
Prerequisites
Before you begin, make sure that you have:
An ApsaraDB RDS for MariaDB destination instance. For more information, see Create an ApsaraDB RDS for MariaDB instance.
Enough storage space on the destination instance. The available space must exceed the total data size of the source database.
Billing
| Synchronization type | Fee |
|---|---|
| Schema synchronization and full data synchronization | Free of charge |
| Incremental data synchronization | Charged. For more information, see Billing overview. |
Supported synchronization topologies
One-way one-to-one synchronization
One-way one-to-many synchronization
One-way cascade synchronization
One-way many-to-one synchronization
Two-way one-to-one synchronization
For details, see Synchronization topologies.
SQL operations that can be synchronized
| Operation type | SQL statements |
|---|---|
| DML | INSERT, UPDATE, DELETE |
| DDL | ALTER TABLE, ALTER VIEW, CREATE FUNCTION, CREATE INDEX, CREATE PROCEDURE, CREATE TABLE, CREATE VIEW, DROP INDEX, DROP TABLE, RENAME TABLE, TRUNCATE TABLE |
The RENAME TABLE operation can cause data inconsistency. If a synchronized table is renamed during synchronization, its data stops being synchronized to the destination. To prevent this, select the entire database as the synchronization object rather than individual tables. Make sure both the pre-rename and post-rename databases are included in the selected objects.
Permissions required
Grant the following permissions to the database accounts used by DTS.
| Database | Required permissions | Purpose |
|---|---|---|
| Source self-managed MariaDB database | REPLICATION CLIENT | Allows DTS to check binary log status using SHOW MASTER STATUS and SHOW BINARY LOGS. |
| REPLICATION SLAVE | Allows DTS to connect to the source database and read the binary log for incremental synchronization. | |
| SHOW VIEW | Allows DTS to read view definitions during schema synchronization. | |
| SELECT | Allows DTS to read table data during full data synchronization. | |
| Destination ApsaraDB RDS for MariaDB instance | Read and write | Allows DTS to write synchronized data to the destination instance. |
For instructions on creating accounts and granting permissions:
Source database: See CREATE USER and GRANT in the MariaDB documentation.
Destination instance: See Create an account and Modify or reset account permissions.
Limitations
During schema synchronization, DTS synchronizes foreign keys from the source to the destination database. During full and incremental synchronization, DTS temporarily disables foreign key constraint checks and cascade operations at the session level. If you perform cascade update or delete operations on the source database during synchronization, data inconsistency may occur.
Source database limitations
| Limitation | Applies to |
|---|---|
| Tables must have a PRIMARY KEY or UNIQUE constraint, and all fields must be unique. Without this, the destination database may contain duplicate records. | Full data synchronization, incremental data synchronization |
| When synchronizing individual tables (not entire databases): the task supports up to 1,000 tables. Exceeding this limit causes a request error. Split the workload across multiple tasks, or synchronize the entire database instead. | All synchronization types |
Binary logging must be enabled and binlog_row_image must be set to full. If not configured correctly, the precheck fails and the task cannot start. | Incremental data synchronization |
| Binary logs must be retained for at least 7 days. Shorter retention may cause the task to fail or result in data inconsistency or loss. The DTS SLA does not cover reliability or performance if this requirement is not met. | Incremental data synchronization |
Required binary log configuration (add to your MariaDB configuration file):
log_bin = mariadb-bin
binlog_row_image = FULL| Parameter | Required value | Description |
|---|---|---|
log_bin | Any valid path or name | Enables binary logging. Required for incremental synchronization. |
binlog_row_image | FULL | Logs all column values for each changed row. DTS requires this for accurate replication. |
For more information on parameter configuration, see Modify instance parameters.
Other limitations
| Limitation | Applies to |
|---|---|
| The destination database version must be the same as or later than the source version. An older destination version may cause compatibility issues. | All synchronization types |
| Synchronize during off-peak hours to minimize load on the source and destination database servers. During full data synchronization, DTS uses read and write resources on both ends. To control load, configure throttling in Advanced Settings (see step 6). | Full data synchronization |
| After full data synchronization completes, the destination tablespace is larger than the source due to fragmentation from concurrent INSERT operations. | Full data synchronization |
If individual tables (not entire databases) are selected as synchronization objects, do not use tools such as pt-online-schema-change for DDL operations. Use Data Management (DMS) for online DDL instead. | All synchronization types |
| Write data to the destination database only through DTS during synchronization. If you use other tools to write data, do not perform online DDL operations through DMS — this may cause data loss. | All synchronization types |
| If DDL statements fail to execute on the destination, the task continues running. View failed DDL statements in task logs. | All synchronization types |
Set up the synchronization task
Step 1: Go to the Data Synchronization Tasks page
Log on to the Data Management (DMS) console.
In the top navigation bar, click Data + AI.
In the left-side navigation pane, choose DTS (DTS) > Data Synchronization.
Console operations may vary based on your DMS mode and layout. For more information, see Simple mode and Customize the layout and style of the DMS console. Alternatively, go directly to the Data Synchronization Tasks page in the DTS console.
Step 2: Select a region
On the Data Synchronization Tasks page, select the region where the data synchronization instance resides.
In the new DTS console, select the region from the top navigation bar.
Step 3: Configure source and destination databases
Click Create Task. On the Create Task page, configure the source and destination databases.
Read the Limits displayed on the page after configuring source and destination databases. Skipping this step may cause the task to fail or result in data inconsistency.
Source database
| Parameter | Value |
|---|---|
| Task Name | A descriptive name for the task. DTS generates a name automatically; you can change it. Task names do not need to be unique. |
| Database Type | Select MariaDB. |
| Access Method | Select Self-managed Database on ECS. |
| Instance Region | The region where the ECS instance resides. |
| Replicate Data Across Alibaba Cloud Accounts | Select No if the source and destination databases are under the same Alibaba Cloud account. |
| ECS Instance ID | The ID of the ECS instance hosting the source MariaDB database. |
| Port Number | The service port of the source MariaDB database. Default: 3306. |
| Database Account | The account for the source MariaDB database. See Permissions required. |
| Database Password | The password for the database account. |
| Encryption | Default: Non-encrypted. |
Destination database
| Parameter | Value |
|---|---|
| Database Type | Select MariaDB. |
| Access Method | Select Alibaba Cloud Instance. |
| Instance Region | The region where the destination ApsaraDB RDS for MariaDB instance resides. |
| RDS Instance ID | The ID of the destination ApsaraDB RDS for MariaDB instance. |
| Database Account | The account for the destination database. The account must have read and write permissions. |
| Database Password | The password for the database account. |
| Encryption | Default: Non-encrypted. |
Step 4: Test connectivity
Click Test Connectivity and Proceed.
DTS automatically adds its server IP ranges to the whitelist of Alibaba Cloud database instances and to the security group rules of ECS instances. For self-managed databases deployed across multiple ECS instances, manually add the DTS server IP ranges to each instance's security group rules. For databases hosted in data centers or by third-party cloud providers, manually add the DTS server IP ranges to the database whitelist. For more information, see CIDR blocks of DTS servers.
Adding DTS server IP ranges to whitelists or security group rules introduces potential security exposure. Before proceeding, take measures such as strengthening account credentials, limiting exposed ports, authenticating API calls, auditing whitelist rules regularly, and connecting to DTS through Express Connect, VPN Gateway, or Smart Access Gateway.
Step 5: Select synchronization objects and configure settings
| Parameter | Description |
|---|---|
| Synchronization Types | Select Schema Synchronization, Full Data Synchronization, and Incremental Data Synchronization. Incremental synchronization is selected by default. Full synchronization establishes the data baseline for incremental synchronization. |
| Method to Migrate Triggers in Source Database | Select how to handle triggers from the source database. Configure only if triggers need to be synchronized. This parameter appears only when Schema Synchronization is selected. For more information, see Synchronize or migrate triggers from the source database. |
| Processing Mode of Conflicting Tables | Precheck and Report Errors: checks for tables with identical names in the source and destination databases before starting the task. The task starts only if no conflicts are found. If conflicts exist and the tables cannot be deleted or renamed, use object name mapping to rename destination tables. Ignore Errors and Proceed: skips the conflict check. If a record in the destination has the same primary key or unique key as a source record — during full synchronization, the destination record is kept; during incremental synchronization, the destination record is overwritten. If schemas differ, data initialization may fail. |
| Capitalization of Object Names in Destination Instance | Default: DTS default policy. Adjust if you need the destination to match the capitalization of the source or destination database. For more information, see Specify the capitalization of object names. |
| Source Objects | Select objects from the Source Objects section and click the arrow icon to move them to the Selected Objects section. Select columns, tables, or databases. If you select tables or columns, DTS does not synchronize views, triggers, or stored procedures. |
| Selected Objects | Right-click an object to rename it or filter data with a WHERE condition. Click Batch Edit to rename multiple objects at once. For more information, see Map object names and Specify filter conditions. Note: renaming objects may cause dependent objects to fail synchronization. |
Step 6: Configure advanced settings
Click Next: Advanced Settings.
| Parameter | Description |
|---|---|
| Dedicated Cluster for Task Scheduling | By default, DTS schedules the task on the shared cluster. For higher stability, purchase and specify a dedicated cluster. For more information, see What is a DTS dedicated cluster. |
| Monitoring and Alerting | Select Yes to receive notifications when the task fails or synchronization latency exceeds the threshold. Configure the alert threshold and notification settings. For more information, see Configure monitoring and alerting. |
| Retry Time for Failed Connections | The time range during which DTS retries failed connections. Range: 10–1,440 minutes. Default: 720. Set this to at least 30 minutes. If the connection is restored within this period, the task resumes; otherwise, the task fails. If multiple tasks share the same source or destination database, the shortest retry time takes precedence. DTS charges for the instance during retry. |
| Retry Time for Other Issues | The time range during which DTS retries failed DDL or DML operations. Range: 1–1,440 minutes. Default: 10. Set this to at least 10 minutes. This value must be less than Retry Time for Failed Connections. |
| Enable Throttling for Full Data Migration | Limit read/write load on databases during full synchronization by configuring Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s). This parameter appears only when Full Data Synchronization is selected. |
| Enable Throttling for Incremental Data Migration | Limit load during incremental synchronization by configuring RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s). |
| Environment Tag | Tag the task with an environment label. Optional. |
| Configure ETL | Select Yes to enable extract, transform, and load (ETL) processing. Enter data processing statements in the code editor. For more information, see What is ETL? and Configure ETL. |
Step 7: Run the precheck
Click Next: Save Task Settings and Precheck.
To preview the OpenAPI parameters for this task configuration, hover over the button and click Preview OpenAPI parameters before proceeding.
DTS runs a precheck before the task can start.
If a check item fails, click View Details to see the cause, fix the issue, and rerun the precheck.
If a check item generates an alert:
If the alert cannot be ignored, fix the issue and run the precheck again.
If the alert can be ignored, click Confirm Alert Details, then click Ignore in the View Details dialog, confirm with OK, and click Precheck Again.
Ignoring alert items may cause data inconsistency or expose your business to risk.
Step 8: Purchase the synchronization instance
Wait for the Success Rate to reach 100%, then click Next: Purchase Instance.
On the buy page, configure the following parameters.
| Parameter | Description |
|---|---|
| Billing Method | Subscription: pay upfront for a fixed term. More cost-effective for long-term use. Pay-as-you-go: billed hourly. Suitable for short-term use. Release the instance when no longer needed to stop charges. |
| Resource Group Settings | The resource group for the synchronization instance. Default: default resource group. For more information, see What is Resource Management? |
| Instance Class | The instance class determines synchronization speed. For more information, see Instance classes of data synchronization instances. |
| Subscription Duration | Available only for the Subscription billing method. Options: 1–9 months, 1 year, 2 years, 3 years, or 5 years. |
Step 9: Start the task
Read and select Data Transmission Service (Pay-as-you-go) Service Terms.
Click Buy and Start, then click OK in the confirmation dialog.
The task appears in the task list. Monitor its progress from there.