Dave
Assistant Engineer
Assistant Engineer
  • UID627
  • Fans2
  • Follows0
  • Posts55
Reads:585Replies:0

It's time to migrate data without stopping services

Created#
More Posted time:Mar 23, 2017 10:59 AM
Introduction
Database migration occurs throughout system development. For example, there are scenarios where you need to split the database to ease the pressure when it reaches the bottleneck, migrate business systems to the cloud or test data structures. Many database products provide their native migration tools, such as mysqldump for MySQL, SSMA for SQL Server and imp/exp for Oracle. However, these tools only support full data replication and require services to be stopped before data migration to ensure data consistency. Obviously, this traditional migration solution imposes significant impact to services. To lower the data migration threshold, Alibaba Cloud DTS provides data migration functionality without stopping services. With this function, the service downtime for data migration can be reduced to minutes.

Traditional database migration process
This section describes the complete data migration process of the traditional full data migration solution.
 
The traditional database migration process consists of the following steps, as shown in the figure above:
1. Stop ongoing services of the application.
2. Begin data migration with the migration tool.
3. Start the application and restore those services.
As you can see, services of the application need to be stopped throughout the data migration process. When there is a large amount of data to be migrated, the service downtime may rise to hours or even days, imposing significant impact on application availability.

Database migration process without stopping services
This section describes the complete data migration process of the incremental data migration solution provided by DTS.
 
The DTS migration process without stopping services consists of the following steps, as shown in the figure above:
1. Configure a data migration task using DTS.
2. When the migration task enters the incremental data sync phase, stop services of the source database application when the incremental data has been fully synchronized. When the data is synchronized quickly, switch services to the target database and start those services.
With this solution, you can only stop the services before the switchover during the data migration process, reducing the service downtime to minutes and minimizing the impact of data migration on those services.

Technical principle of the database migration solution without stopping services
This section describes the implementation principle of the DTS data migration solution without stopping services.
 
DTS will start an incremental log pulling and parsing procedure in the background before full data migration, as shown in the figure above. This procedure will retrieve any incremental logs generated during the full data migration process in real time, parse those logs, encapsulate the incremental data into the DTS format and store the encapsulated data in the local storage system. After the full data migration is complete, the incremental data restoration module will read the stored incremental log data and formulate the SQL statement for restoring the data by going through several steps including parsing, filtering and encapsulation. Then, the incremental data will be stored to the target database to implement real-time incremental data synchronization between the source and target databases.

Summary
The data migration without stopping services feature is a low-threshold data migration solution provided by Alibaba Cloud DTS. Currently, this feature is available for MySQL, SQL Server and PostgreSQL. In the future, it will also be available for Oracle, MongoDB and Redis.
The data migration without stopping services feature is one of the extensive data transfer solutions provided by DTS. In addition to that, DTS also provides other data transfer solutions such as real-time data synchronization and real-time data subscription. With the real-time data synchronization, you can implement remote data disaster recovery, accelerate the access to cross-border services and streamline the data channels from OLTP to OLAP. With the data subscription, you can implement various service scenarios such as lightweight cache updating and asynchronous decoupling of core services.
Guest