Data migration
Data migration moves data between homogeneous or heterogeneous data sources. Common use cases include migrating to the cloud, moving data between instances within Alibaba Cloud, and splitting or scaling databases.
| Category | Feature | Description | References |
|---|---|---|---|
| Homogeneous migration | Logical migration | Migrate data between databases of the same type. | Migrate data from a self-managed MySQL database to an ApsaraDB RDS for MySQL instance |
| Homogeneous migration | Physical migration | Migrate data from a self-managed database to the cloud using a physical gateway. | Migrate data from a self-managed SQL Server database to an ApsaraDB RDS for SQL Server instance by using a physical protocol gateway / Migrate data from a self-managed SQL Server database to an ApsaraDB RDS for SQL Server instance by using a physical gateway on a bastion host |
| Heterogeneous migration | Logical migration | Migrate data between databases of different types. | Migrate data from a self-managed Oracle database to an AnalyticDB for PostgreSQL instance / Data type mappings between heterogeneous databases |
| Traffic cutover | Database cutover to the cloud | After completing database and application evaluation and transformation, smoothly cut over your business to cloud databases with minimal disruption. | Database cutover |
| Periodic tasks | Scheduled full migration | Migrate schema data and historical data from the source database to the destination database on a regular schedule, using the scheduling policy configurations of the data integration feature. | Configure a data integration task between ApsaraDB RDS for MySQL instances |
| Account migration | Full account migration | Migrate accounts, including passwords and permissions, from the source database to the destination database when configuring a synchronization or migration task. | Migrate database accounts |
Feature set | Feature | Description | References |
Homogeneous validation | Metadata validation | Validate the schema of homogeneous data. | |
Full data validation | Validate the historical data of homogeneous databases. | ||
Incremental data validation | Validate data from incremental synchronization or migration between homogeneous databases. | ||
Heterogeneous validation | Metadata equivalence validation | Validate the schema of heterogeneous data. | |
Full data validation | Validate the historical data of heterogeneous databases. | ||
Incremental data validation | Validate data from incremental synchronization or migration between heterogeneous databases. | ||
Correction | Metadata correction | If a schema inconsistency is found during validation, you can correct it based on the validation results. | |
Full data correction | If a full data inconsistency is found, you can download the correction SQL based on the validation results to make corrections. | ||
Incremental data correction | If an incremental data inconsistency is found, you can correct it based on the validation results. |
Feature set | Feature | Description | References |
Task management | Task creation and configuration | Create data transmission tasks between various data sources. You can configure the task before or after purchase. | |
Task scheduling | By default, DTS schedules tasks on a shared cluster. You can select a dedicated cluster to schedule the task during advanced configuration. | - | |
Task monitoring | View the link status and various performance metrics of a task in the DTS console. This information helps you manage your tasks. | ||
Task diagnostics | Provides diagnostic results and suggestions by checking the performance of the source database, destination database, network, and DTS during incremental migration. | ||
Task modification | Manage environment tags for instances. Add or remove synchronization objects while a data synchronization instance is running. Modify the ETL configuration of a synchronization or migration instance. | ||
Task deletion | For DTS instances that have completed their data transmission tasks and whose configurations you do not need to retain, you can manually release pay-as-you-go instances or unsubscribe from subscription instances to avoid extra charges. | ||
Cross-account access | Configure tasks that span across Alibaba Cloud accounts. This is suitable for scenarios such as resource migration or consolidation between Alibaba Cloud accounts and business architecture adjustments. | ||
Hybrid cloud and cross-cloud access | Connect to databases using public IP addresses to migrate data between accounts with different properties, such as public cloud and Alibaba Finance Cloud accounts. | - | |
Operation audit | Query the operation logs of an instance to get information about operations performed on the instance, the results of those operations, and who performed them. | ||
Data connection management | Enter database information so you can select a pre-recorded database when configuring a task. DTS automatically fills in the database information, so you do not need to enter it manually. | ||
OpenAPI | POP API | Call and test APIs in the OpenAPI Portal. | |
SDK | Call APIs using an SDK. | ||
Terraform | Supports the open source tool Terraform. | - | |
Network management | Cross-region connection | Supports data transmission where the source and destination databases are in different regions. | |
Transmission compression | The DTS link supports concurrent compressed transmission to reduce bandwidth usage. | ||
Private endpoint access | When connecting to a source or destination database through a VPC, you can use a private endpoint to connect to the database. | - | |
Database Gateway (DG) | Connect databases to DTS using Database Gateway (DG). | ||
Internet traffic connection | Connect to databases using public IP addresses. | Add the CIDR blocks of DTS servers to the IP address whitelist | |
Cross-border transmission | By default, only non-cross-border data synchronization tasks are supported. To create a cross-border data synchronization task, you must apply for the required permissions. | ||
Event Center | Event notification | Subscribe to important events and set up custom alert notifications through the Cloud Monitor platform. This keeps you informed about event occurrences and progress, helping you quickly analyze and locate issues during business failures. | |
Proactive O&M platform | Supports O&M event alerts. When the system detects a risk that may prevent a DTS instance from running properly, it triggers a corresponding O&M event and notifies you through the console, email, or internal messages. | ||
Serverless | Serverless instance management | Perform operations on Serverless instances, such as pausing the instance, viewing metrics, and modifying the upper and lower limits of DUs. | - |
Dedicated cluster | DU management | View the creation and usage information of dedicated cluster DUs. Modify the number of DUs for a dedicated cluster task to adjust the task's specifications. | |
Disk space modification | If the disk usage of a dedicated cluster is too high and cannot meet current business needs, you can scale out the disk space to ensure it matches your requirements. | ||
Migration between shared and dedicated clusters | Change the cluster where an instance runs. You can migrate an instance from a dedicated cluster to a public cluster, or from a public cluster to a dedicated cluster. | Migrate an instance between a dedicated cluster and a public cluster | |
Dedicated cluster instance management | Perform operations on dedicated clusters, such as manual renewal, changing node configurations, and setting the overcommit ratio. | ||
Security | Data encryption | Connect to databases using SSL-encrypted connections. | |
Operation isolation | Use RAM identities with least privilege, such as RAM users and RAM roles, to access DTS. This improves data security and reduces security risks from permission abuse. | - | |
Account permission control | Access cloud resources such as RDS and ECS under the current Alibaba Cloud account by granting system policies. Use database accounts with sufficient permissions to complete data transmission tasks. | ||
Reliability | Cluster high availability (HA) | Uses high-specification servers to ensure good transmission performance for every migration or synchronization link. | |
Resumable transmission | Features an automatic resumable transmission mechanism to ensure the reliability of data transmission. | ||
Data source disaster recovery protection | If a source or destination database cannot be connected or has other issues, DTS immediately starts continuous retries. | - |
Feature set | Feature | Description | References |
Change tracking | Change tracking channel | Create a change tracking instance to obtain real-time incremental data changes from a database. | |
Change tracking SDK | Use sample code for the SDK client, flink-dts-connector, or Kafka client to print subscribed data. This allows SDK, Flink, and Kafka clients to consume data from the change tracking channel. | ||
Change tracking traffic management | When the source database of a change tracking instance is MySQL, you can configure the billing type to determine whether to charge for data traffic. |
Feature set | Feature | Description | References |
Homogeneous migration | Logical Migration | Supports data migration between homogeneous databases. | Migrate a self-managed MySQL database to an RDS for MySQL instance |
Physical Migration | Migrate a self-managed database to the cloud using a physical gateway. | ||
Heterogeneous migration | Logical Migration | Supports data migration between heterogeneous databases. | |
Cutover | Database cutover to the cloud | Database cutover helps you smoothly migrate your business to a cloud database after database evaluation and application evaluation and transformation are complete. | |
Periodic task | Scheduled full migration | Use the scheduling policy of the data integration feature to periodically migrate the schema and historical data from the source database to the destination database. | Configure a data integration task between RDS for MySQL instances |
Account migration | Full account migration | Enable the account migration feature when you configure a sync or migration task. This helps you quickly migrate accounts, including passwords and permissions, from the source database to the destination database. |
Data synchronization
Data synchronization keeps data in sync between data sources in real time. Use cases include active geo-redundancy, geo-disaster recovery, zone-disaster recovery, cross-border data synchronization, cloud-based business intelligence (BI) systems, and real-time data warehousing.
| Category | Feature | Description | References |
|---|---|---|---|
| Synchronization instance management | Reverse disaster recovery switching | Create a reverse synchronization instance in a few steps for a running synchronization instance. The reverse instance synchronizes incremental data back from the destination database to the source database. | Create a reverse instance |
| Disaster recovery and multi-active redundancy | Two-way synchronization | Configure real-time two-way data synchronization between two databases, such as an ApsaraDB RDS for MySQL database and a self-managed MySQL database. Suitable for active geo-redundancy based on a cellular architecture and geo-disaster recovery. | Configure two-way data synchronization between MySQL instances / Configure two-way data synchronization between PolarDB for PostgreSQL (Compatible with Oracle) clusters / Configure two-way data synchronization between ApsaraDB RDS for PostgreSQL instances / Configure two-way data synchronization between ApsaraDB for MongoDB replica set instances / Configure two-way synchronization between Tair instances |
| Disaster recovery and multi-active redundancy | Direction switch for two-way synchronization instances | Switch the synchronization direction of an existing two-way synchronization instance without creating a new one. Useful when switching between primary and secondary databases or between cloud platforms. | Switch the direction of a two-way synchronization instance |
| Disaster recovery and multi-active redundancy | Global active database (GAD) cluster | Create a GAD cluster based on ApsaraDB RDS and DTS to enable database disaster recovery and allow users to access the nearest resources. | N/A |
| Synchronization topology management | Topology upgrade | Upgrade the synchronization topology of a task from one-way to two-way synchronization as business requirements evolve. | Upgrade synchronization topology from one-way to two-way |
| Conflict detection and resolution | Conflict detection | Detect conflicts including uniqueness conflicts from INSERT operations, inconsistent records from UPDATE operations, and records targeted for deletion that do not exist. | N/A |
| Conflict detection and resolution | Conflict resolution | Three policies are available: TaskFailed terminates the task and reports an error when a conflict occurs; Ignore keeps the conflicting records in the destination; Overwrite replaces the conflicting records in the destination. | N/A |
| Heterogeneous synchronization | Synchronization to real-time data warehouses | Synchronize data to real-time data warehouses for high-throughput offline processing and high-performance online analysis. | Synchronize data from an ApsaraDB RDS for MySQL instance to an AnalyticDB for MySQL V3.0 cluster |
| Heterogeneous synchronization | Non-database synchronization | Synchronize data to a function in Function Compute, where you can write function code to process the data. | Synchronize data from a MongoDB replica set instance to a Function Compute function / Synchronize data from an ApsaraDB for MongoDB sharded cluster instance to a Function Compute function |
| Heterogeneous synchronization | Heterogeneous database synchronization | Synchronize data between databases of different types. | Data type mappings for schema synchronization / Synchronize data from a self-managed Oracle database to an AnalyticDB for PostgreSQL instance |
| Data shipping | Data shipping channel | Create a data shipping instance to establish a data shipping channel, then use the data shipping SDK to ship data from your sources to DTS. | Create a data shipping instance |
| Data shipping | Data shipping SDK | Use the data shipping SDK to ship data from various sources to DTS, which then synchronizes the data to the destination database. This extends the range of supported source types. | Use the SDK to ship data to DTS |
| Homogeneous synchronization | Real-time synchronization | Synchronize data in real time between databases of the same type. | Synchronize data from a self-managed MySQL database to an ApsaraDB RDS for MySQL instance |
Feature set | Feature | Description | References |
Read/write splitting and shunting | Real-time TP cache | Migrate MySQL databases, such as self-managed MySQL or RDS for MySQL, to Redis instances. This reduces the load on the backend relational database and improves user experience. | |
Metadata filtering and mapping | Database, table, and column filtering | When configuring task objects, you can select them at the database, table, and column granularity. | - |
DDL and DML filtering | When configuring synchronization or migration task objects, you can use SQL operations to filter the data for incremental synchronization or migration. | - | |
Database, table, and column name mapping | When configuring synchronization or migration task objects, you can set the names for the task objects (database, table, column) in the destination database instance. You can use this feature to synchronize or migrate data to a specified object in the destination instance, or to create an object in the destination instance with the same structure but a different name. | ||
Topology mapping | The data synchronization feature supports multiple synchronization topologies. You can plan your synchronization instances based on your business needs. | ||
Data filtering, processing, and mapping | WHERE clause filtering | When configuring synchronization or migration task objects, you can set filter conditions to synchronize or migrate only the data that meets your needs from the source database to the destination database. | |
Data type mapping | When synchronizing or migrating data between heterogeneous databases, data type mapping is performed during the schema synchronization or migration phase. This converts the data types from the source database to types supported by the destination database. |
Feature set | Feature | Description | References |
Synchronization link management | One-click reverse disaster recovery switchover | Create a reverse incremental synchronization instance for a running synchronization instance with one click. | |
Disaster recovery & active-active | Two-way synchronization | Create real-time two-way data synchronization between two databases, such as an RDS for MySQL instance and a self-managed MySQL database. This is suitable for scenarios like active geo-redundancy (unitization) and geo-disaster recovery. | |
One-click reversal of synchronization direction | Reverse the direction of a two-way synchronization instance. When you need to switch between primary and secondary databases or between two clouds, you can easily adjust the direction of an existing two-way synchronization instance without reconfiguring it. | ||
Global Active Database (GAD) cluster | Build a global active-active database cluster using RDS and DTS. This provides database disaster recovery and a nearest access experience for users. | - | |
Synchronization topology management | Upgrade the synchronization topology of a sync task from one-way to two-way to meet evolving business needs. | Upgrade a one-way synchronization task to a two-way synchronization task | |
Conflict Management | Conflict detection | Detect uniqueness conflicts caused by INSERT, incomplete record matches in UPDATE, and non-existent records in DELETE. | - |
Conflict handling | Supports three conflict resolution strategies: TaskFailed (task reports an error and exits), Ignore (uses the conflicting record from the destination instance), and Overwrite (overwrites the conflicting record in the destination instance). | - | |
Heterogeneous synchronization | Real-time data warehouse synchronization | Synchronize data to a real-time data warehouse to support high-throughput offline processing and high-performance online analysis. | Synchronize data from an RDS for MySQL instance to an AnalyticDB for MySQL 3.0 instance |
Non-database synchronization | Synchronize data to a specified function in Function Compute (FC). You can write function code for secondary data processing. | ||
Heterogeneous database synchronization | Supports data synchronization between heterogeneous databases. | ||
Data shipping | Data shipping channel | Create a data shipping instance to establish a data shipping channel. This lets you use an SDK to deliver data from a data source to DTS. | |
Data shipping SDK | Use an SDK to deliver various types of data from a data source to DTS. DTS then synchronizes the data to the destination database, supporting a wider range of data sources. | ||
Homogeneous synchronization | Logical real-time homogeneous synchronization | Supports data synchronization between homogeneous databases. | Synchronize data from a self-managed MySQL database to an RDS for MySQL instance |
Change tracking
Change tracking captures real-time incremental data from databases. Consume or write the incremental data based on your business requirements to support cache updates, asynchronous business decoupling, real-time data synchronization between heterogeneous sources, and complex extract, transform, and load (ETL) pipelines.
| Category | Feature | Description | References |
|---|---|---|---|
| Change tracking | Change tracking channels | Create a change tracking instance to capture real-time incremental data changes from a database. | Overview of change tracking scenarios |
| Change tracking | Change tracking SDK | Consume tracked data using an SDK client, Flink client, or Kafka client. The SDK client demo, flink-dts-connector, and Kafka client demo are all available for displaying tracked data. | Consume tracked data |
| Change tracking | Traffic management | For change tracking instances with a MySQL source database, data transfer fees are charged based on the configuration fee type you select. | Billable items / Track data changes from an ApsaraDB RDS for MySQL instance |
Task management
| Category | Feature | Description | References |
|---|---|---|---|
| Task management | Task creation and configuration | Create DTS tasks between various data sources. Configure a task before or after purchasing a DTS instance. | Supported databases / Purchase a DTS instance |
| Task management | Task scheduling | By default, DTS schedules tasks to a shared cluster. Select a dedicated cluster in the advanced settings to pin a task to dedicated resources. | N/A |
| Task management | Task monitoring | View connection status between DTS and your source and destination databases, along with performance metrics for the DTS instance. | Monitor task performance |
| Task management | Task diagnostics | During incremental migration, check the performance of the source database, destination database, network, and DTS. Diagnostic results and suggestions are provided. | Monitor task performance / View the connection status and performance of incremental data migration / View the connection status and performance of data synchronization |
| Task management | Task modification | Manage environment tags, add or remove synchronized objects while a synchronization instance is running, and modify ETL configurations for migration or synchronization tasks. | Modify the objects to be synchronized / Configure ETL in a data migration or synchronization task / Manage environment tags |
| Task management | Task deletion | Release a pay-as-you-go DTS instance or unsubscribe from a subscription DTS instance when all tasks are complete and the instance is no longer needed, to avoid additional charges. | Release DTS instances |
| Task management | Cross-account access | Configure DTS tasks across Alibaba Cloud accounts for scenarios such as resource migration, account merging, or business architecture adjustments. | Configure a DTS task across Alibaba Cloud accounts |
| Task management | Cross-cloud access in a hybrid cloud | Access databases over public IP addresses to migrate data between accounts with different attributes, such as between a public cloud account and a financial cloud account. | N/A |
| Task management | Operation logs | Query the operation logs of an instance to see what operations were performed, their results, and who performed them. | Query the operations logs of instances |
| Task management | Database connection management | Register databases in advance so that DTS auto-fills connection details when you configure a task, eliminating manual entry. | Manage database connections |
| APIs | OpenAPI (POP) | Call DTS API operations in OpenAPI Explorer for testing and integration. | Integration overview |
| APIs | SDKs | Call DTS API operations programmatically using the provided SDKs. | API usage example |
| APIs | Terraform | Manage DTS resources using the open-source Terraform tool. | N/A |
| Network management | Cross-region connectivity | Transmit data between source and destination databases in different regions. | Cross-border and cross-region instances |
| Network management | Data compression | DTS uses concurrent data compression during transmission to minimize bandwidth utilization. | Benefits |
| Network management | Internal endpoint access | Connect to a source or destination database in a virtual private cloud (VPC) over an internal endpoint. | N/A |
| Network management | Database Gateway | Connect an on-premises or third-party cloud database to DTS using Database Gateway. | Connect an on-premises database or a database hosted on a third-party cloud to Alibaba Cloud by using Database Gateway |
| Network management | Public internet access | Connect a database to DTS over a public IP address. | Add the CIDR blocks of DTS servers |
| Network management | Cross-border data transmission | By default, DTS supports data synchronization only within the same country or region. Apply for cross-border permissions to synchronize data across borders. | Apply for permissions to synchronize data across borders / Cross-border and cross-region instances |
| Event center | Event notifications | Use the event subscription feature of CloudMonitor to configure custom alert notifications for important events, so you are promptly informed of event status and can analyze and address business interruptions. | Subscribe to event notifications |
| Event center | Proactive O&M | When DTS detects risks that may cause instances to fail, O&M events are triggered and notifications are sent via the console, email, or internal messages. | Manage O&M events |
| Serverless instances | Serverless instance management | Suspend a serverless instance, view metric data, and modify the upper and lower limits of DTS units (DUs). | N/A |
| Dedicated clusters | DU management | View DTS units (DUs) allocated to a dedicated cluster and their usage. Modify the number of DUs for a task to adjust its specifications. | Modify the number of DUs for a task in a DTS dedicated cluster / View DU information of a DTS dedicated cluster |
| Dedicated clusters | Disk expansion | Expand the disk capacity of a dedicated cluster when disk usage is too high for your business needs. | Increase the storage space of nodes in a DTS dedicated cluster |
| Dedicated clusters | Instance migration between clusters | Move DTS instances between a dedicated cluster and a public cluster. | Migrate a DTS instance between a dedicated cluster and a public cluster |
| Dedicated clusters | Cluster management | Manually renew a dedicated cluster, modify node configurations, and set the overcommit ratio. | Manage a DTS dedicated cluster |
| Security | Data encryption | Connect to databases using SSL-secured connections. | Synchronize data between ApsaraDB RDS for MySQL instances |
| Security | Operation isolation | Access DTS using Resource Access Management (RAM) users and RAM roles with the minimum required permissions, reducing security risks from permission abuse. | N/A |
| Security | Account permission management | Use system policies to authorize DTS to access Alibaba Cloud resources such as ApsaraDB RDS and Elastic Compute Service (ECS) instances, enabling tasks to run with a database account that has sufficient permissions. | Authorize DTS to access Alibaba Cloud resources / Prepare the database accounts for data migration |
| Reliability | High availability (HA) clusters | DTS uses high-specification servers to ensure consistent performance for each data synchronization or migration instance. | Benefits |
| Reliability | Resumable upload | DTS automatically resumes interrupted transfers from where they left off, ensuring reliable data transmission without starting over. | Benefits |
| Reliability | Disaster recovery protection for data sources | If the source or destination database becomes unreachable, DTS immediately and continuously retries the connection. | N/A |
Data integration
Data integration combines ETL capabilities with DTS data replication. Configure ETL tasks using drag-and-drop operations or Flink SQL statements to extract data from sources, transform it in-stream, and load it to destinations — reducing development overhead and enabling real-time data processing at scale.
| Category | Feature | Description | References |
|---|---|---|---|
| Read/write splitting | Real-time caching | Migrate data from a MySQL database (self-managed or ApsaraDB RDS for MySQL) to a Redis instance to reduce load on backend relational databases. | Synchronize data from an ApsaraDB RDS for MySQL instance to a Tair (Redis OSS-Compatible) instance / Migrate data from an ApsaraDB RDS for MySQL instance to a Tair (Redis OSS-Compatible) instance |
| Metadata filtering and mapping | Database, table, and column filtering | Select specific databases, tables, and columns as task objects when configuring a DTS task. | N/A |
| Metadata filtering and mapping | DDL and DML filtering | Use SQL statements to filter which data is included in incremental synchronization or migration when configuring a task. | N/A |
| Metadata filtering and mapping | Object name mapping | Specify custom names for databases, tables, and columns in the destination instance. Use this to sync or migrate data to differently named objects, or to create objects that match a source schema under a different name. | Map object names |
| Metadata filtering and mapping | Topology mapping | Data synchronization supports multiple synchronization topologies. Plan your synchronization instances based on your business requirements. | Synchronization topologies |
| Data filtering and mapping | WHERE condition-based filtering | Specify SQL conditions when configuring a task so that only matching rows are synchronized or migrated to the destination. | Set filter conditions |
| Data filtering and mapping | Data type mapping | When synchronizing or migrating between heterogeneous databases, DTS maps source data types to compatible types in the destination during schema synchronization or migration. | Data type mappings for schema synchronization / Data type mappings between heterogeneous databases |
Data verification
Data verification monitors differences between source and destination databases without downtime, so you can catch data and schema inconsistencies as early as possible.
| Category | Feature | Description | References |
|---|---|---|---|
| Homogeneous verification | Metadata verification | Verify the schema of homogeneous databases. | What is data verification? |
| Homogeneous verification | Full data verification | Verify historical data in homogeneous databases. | What is data verification? |
| Homogeneous verification | Incremental data verification | Verify incremental data synchronized or migrated between homogeneous databases. | What is data verification? |
| Heterogeneous verification | Metadata equivalence verification | Verify the schema of heterogeneous databases. | What is data verification? |
| Heterogeneous verification | Full data verification | Verify historical data in heterogeneous databases. | What is data verification? |
| Heterogeneous verification | Incremental data verification | Verify incremental data synchronized or migrated between heterogeneous databases. | What is data verification? |
| Data correction | Metadata correction | If schema inconsistencies are detected during verification, correct the schema based on the verification results. | View data verification details |
| Data correction | Full data correction | If full data inconsistencies are detected, download the generated SQL statements to correct the data. | View data verification details |
| Data correction | Incremental data correction | If incremental data inconsistencies are detected, correct the data based on the verification results. | View data verification details |
Scenario-based solution
Feature set | Feature | Description | References |
ZeroETL | Synchronization from PolarDB for MySQL to AnalyticDB for MySQL 3.0 | Use federated analytics together with the AnalyticDB Pipeline Service (APS) feature of AnalyticDB for MySQL to synchronize data from PolarDB for MySQL to AnalyticDB for MySQL Data Lakehouse Edition (V3.0) in real time. | Use federated analytics to synchronize data |