This topic describes the system architecture of Data Transmission Service (DTS) and the basic design concepts of its main features.
System architecture
Architecture description
- High availability
Each module in DTS has primary and secondary nodes to ensure high availability. The disaster recovery module runs a health check on each node in real time. When a node exception is detected, the module only requires a few seconds to switch the channel to another healthy node.
- Connection reliability
To ensure the connection reliability of change tracking and data synchronization channels, the disaster recovery module checks for any changes, such as changes of data source address. When a change of data source address is detected, the module dynamically allocates a new connection method to ensure the stability of the channel.
Design concept of data migration
Data migration supports schema migration, full data migration, and incremental data migration. To migrate data with minimized downtime, you must select schema migration, full data migration, and incremental data migration.
For a migration between heterogeneous databases, DTS reads the source database schema, translates the schema into the syntax of the destination database, and imports the schema to the destination instance.
A full data migration requires a long period of time. In this process, incremental data is continuously written into the source instance. To ensure data consistency, DTS starts the incremental data reading module before full data migration. This module obtains the incremental data from the source instance and parses, encapsulates, and locally stores the data.
When a full data migration is complete, DTS starts the incremental data loading module. This module retrieves the incremental data from the incremental data reading module. After reverse parsing, filtering, and encapsulation, the incremental data is migrated to the destination instance.
Design concept of real-time data synchronization
The real-time data synchronization feature can be used to synchronize incremental data between any two data sources in real time. The real-time data synchronization from OLTP databases to OLAP databases is supported.
- Initial synchronization
DTS synchronizes the schemas and historical inventory data of the objects that you have selected from the source instance to the destination instance.
- Real-time synchronization of incremental data
After initial synchronization is complete, DTS synchronizes incremental data from the source instance to the destination instance in real time.
- Incremental data reading module
The incremental data reading module obtains raw data from the source instance. After parsing, filtering, and syntax conversion, the data is locally stored. The incremental data reading module connects to the source instance by using a database protocol and obtains incremental data from the source instance. If the source database is ApsaraDB RDS for MySQL, the incremental data reading module connects to the source database by using the binary log dump protocol.
- Incremental data loading module
The incremental data loading module requests incremental data from the incremental data reading module and filters data based on the specified objects to be synchronized. Then, the incremental data loading module synchronizes data to the destination instance without compromising transactional sequence and consistency.
DTS ensures high availability of the incremental data reading module and incremental data loading module. When a channel exception is detected, the disaster recovery module switches the channel to a healthy node. This ensures high availability of the synchronization channel.
Design concept of change tracking
The change tracking feature allows you to obtain incremental data from an RDS instance in real time. You can subscribe to the incremental data on the change tracking server by using DTS SDKs. You can also customize data consumption rules based on your business requirements.
The incremental data reading module on the server side of DTS obtains raw data from the source instance. After parsing, filtering, and syntax conversion, the incremental data is locally stored.
The incremental data reading module connects to the source instance by using a database protocol and obtains the incremental data from the source instance in real time. If the source database is ApsaraDB RDS for MySQL, the incremental data reading module connects to the source database by using the binary log dump protocol.
- When an exception is detected in the incremental data reading module, the disaster recovery module restarts the incremental data reading module on a healthy service node. This ensures high availability of the incremental data reading module.
- DTS ensures high availability of consumption SDK processes on the server. If you start multiple consumption SDK processes for the same channel, the server pushes incremental data to only one process at a time. If an exception occurs on a process, the server pushes data to another healthy consumption process.