This topic describes how to migrate data from a self-managed SQL Server database to an Alibaba Cloud ApsaraDB RDS for SQL Server instance by using the Data Transmission Service (DTS) console. You can flexibly configure schema migration, full data migration, and incremental data migration. When these three migration types are configured together, you can migrate data without service interruption.
Prerequisites
You have created a destination ApsaraDB RDS for SQL Server instance with storage space larger than that of the source database. If the space is insufficient, you need to increase the instance space in advance.
Usage notes
Please pay attention to the following key notes before migration, ignoring them may cause task failure or errors:
Database quantity limit: A single migration task cannot migrate more than 10 databases. Otherwise, stability and performance risks may occur.
Table quantity limit: When incremental migration is included, the number of tables to be synchronized from the source database cannot exceed 1000. Otherwise, task delay or instability may occur.
Source database operation restrictions: During schema migration and full migration phases, do not execute DDL operations (such as modifying database or table structures). Otherwise, the task will fail.
Table structure requirements: Tables to be migrated must have primary keys or unique constraints, and fields must have uniqueness. Otherwise, duplicate data may appear in the destination database.
Foreign keys and triggers: If the migration task includes incremental data migration, you need to disable triggers and foreign keys that have been enabled in the destination database. Otherwise, the task may fail or data may be lost.
Database name standards: If the name of the database to be migrated does not comply with the definition standards of RDS SQL Server, you need to manually create a database in RDS SQL Server in advance. Otherwise, the task may not run properly.
Data log retention time: Incremental migration tasks require the source database's data logs to be retained for more than 24 hours. Full + incremental migration tasks require data logs to be retained for at least 7 days. Otherwise, the task may fail or data inconsistency may occur.
Billing
Migration type | Instance configuration fee | Internet traffic fee |
Schema migration and full data migration | Free of charge. | When the Access Method parameter of the destination database is set to Public IP Address, you are charged for Internet traffic. For more information, see Billing overview. |
Incremental data migration | Charged. For more information, see Billing overview. |
Permissions required for database accounts
To successfully complete the data migration task, ensure that the database accounts of the source and destination databases have the following permissions:
For permission granting in self-managed SQL Server databases, see CREATE USER.
You can create an account through the RDS console and modify account permissions as needed.
Database | Schema migration | Full migration | Incremental migration |
Self-managed SQL Server database | SELECT permission | SELECT permission | sysadmin |
ApsaraDB RDS for SQL Server instance | Read and write permissions | ||
Preparations
If you need to perform incremental migration, before formally configuring the data migration task, you need to set the recovery mode of the specified database in the self-managed SQL Server database to full mode (FULL) to ensure that transaction logs are completely recorded. You also need to save full data and incremental data through logical backups and log backups respectively, providing a foundation for subsequent data migration.
If you need to migrate multiple databases, you need to repeat steps 1 to 3 in the preparation work. Otherwise, data inconsistency may occur.
Execute the following command in the self-managed SQL Server database to change the recovery mode of the database to be migrated to full mode.
use master; GO ALTER DATABASE <database name to be migrated> SET RECOVERY FULL WITH ROLLBACK IMMEDIATE; GOExample:
use master; GO ALTER DATABASE mytestdata SET RECOVERY FULL WITH ROLLBACK IMMEDIATE; GOExecute the following command to perform a logical backup of the database to be migrated. If you have already performed a logical backup, you can skip this step.
BACKUP DATABASE <database name to be migrated> TO DISK='<specify the storage path and file name of the backup file>'; GOExample:
BACKUP DATABASE mytestdata TO DISK='D:\backup\dbdata.bak'; GOExecute the following command to back up the logs of the database to be migrated.
BACKUP LOG <database name to be migrated> to DISK='<specify the storage path and file name of the backup file>' WITH init; GOExample:
BACKUP LOG mytestdata TO DISK='D:\backup\dblog.bak' WITH init; GO
Procedure
Visit the Data Transmission Service (DTS) console.
In the left-side navigation pane, click Data Migration, and select a region at the top.
Click Create Task, and configure the source and destination database information.
Category
Parameter
Description
N/A
Task Name
Configure a name with business meaning (no uniqueness requirement) for easy identification later; or keep the system-generated task name.
Source Database
Select Existing Connection
If you have already entered source database information on the DTS Data Connection Management page, you can directly select the database that has been entered here, and you can avoid manually entering source database information later.
Database Type
Select SQL Server.
Access Method
Select Public IP Address.
NoteWhen selecting a self-managed database, you also need to perform corresponding preparations.
Instance Region
Select the region to which the self-managed SQL Server database belongs.
Hostname Or IP Address
Enter the access address of the self-managed SQL Server database. In this example, enter the public IP address.
Port Number
Enter the service port of the self-managed SQL Server database. The default is 1433.
Database Account
Enter the database account of the self-managed SQL Server. For permission requirements, see Permissions required for database accounts.
Database Password
The password that is used to access the database instance.
Encryption
If the source database has not enabled SSL encryption, select Non-encrypted.
If the source database has enabled SSL encryption, select SSL-encrypted, and DTS will trust the server certificate by default.
Destination Database
Select Existing Connection
If you have already entered destination database information on the DTS Data Connection Management page, you can directly select the database that has been entered here, and you can avoid manually entering destination database information later.
Database Type
Select SQL Server.
Access Method
Select Alibaba Cloud Instance.
Instance Region
Select the region to which the destination ApsaraDB RDS for SQL Server instance belongs.
Instance ID
Select the destination ApsaraDB RDS for SQL Server instance ID.
Database Account
Enter the database account of the destination ApsaraDB RDS for SQL Server instance. For permission requirements, see Permissions required for database accounts.
Database Password
The password that is used to access the database instance.
Encryption
If the destination database has not enabled SSL encryption, select Non-encrypted.
If the destination database has enabled SSL encryption, select SSL-encrypted, and DTS will trust the server certificate by default.
After configuration is complete, click Test Connectivity and Proceed at the bottom of the page. In the CIDR Blocks of DTS Servers dialog box that appears, click Test Connectivity.
ImportantPlease ensure that you have added the IP address ranges of the DTS service to the security settings of the source database to allow access from DTS servers.
Configure the objects to be migrated.
On the Configure Objects page, configure the objects that you want to migrate.
Parameter
Description
Migration Types
For full data migration: We recommend that you select Schema Migration and Full Data Migration.
For migration without downtime: We recommend that you select Schema Migration, Full Data Migration, and Incremental Data Migration.
NoteFor more information, see Appendix 1: SQL operations that support incremental data migration and Appendix 2: Objects that support schema migration.
If you do not select Schema Migration, make sure that the database and tables that will receive data exist in the destination database. Based on your business requirements, you can use the column mapping feature in the Selected Objects section.
If you do not select Incremental Data Migration, to ensure data consistency, do not write new data to the source database during data migration.
Method to Migrate Triggers in Source Database
Select the method for migrating triggers as needed. If the objects you want to migrate do not involve triggers, you do not need to configure this parameter. For more information, see Configure the method for synchronizing or migrating triggers.
NoteYou can configure this parameter only when you select both Migration Types and Schema Migration and Incremental Data Migration.
SQL Server Incremental Synchronization Mode
NoteThis parameter is available only when Migration Types includes Incremental Data Migration.
The maximum number of tables for which CDC is enabled that DTS supports.
Please set a reasonable number of tables that are allowed to enable CDC for the current migration instance. The default value is 1000.
NoteWhen SQL Server Incremental Synchronization Mode is set to Incremental Synchronization Based on Logs of Source Database (Heap tables are not supported), this configuration item will not appear.
Processing Mode of Conflicting Tables
Precheck and Report Errors: DTS checks whether tables with the same names exist in the destination database. If no tables with the same names exist in the destination database, the precheck is passed and the data migration task starts. Otherwise, an error is returned during the precheck and the data migration task does not start.
Solution: If you cannot delete or rename the tables with the same names in the destination database, you can change the names of the tables in the destination database by configuring object name mapping.
Ignore Errors and Proceed: DTS skips the precheck for tables with the same names in the destination database.
WarningIf you select Ignore Errors and Proceed, data inconsistency may occur and your business may be exposed to risks. For example:
If the source and destination tables have the same schema and a record in the destination table has the same primary key value as a record in the source table:
During full data migration, DTS retains the existing records in the destination table and does not migrate the records with the same primary key values from the source table to the destination table.
During incremental data migration, the data in the destination table may be overwritten by the new data from the source table, resulting in data loss in the destination table.
If the source and destination tables have different schemas, only some columns can be migrated or the data migration task may fail. Proceed with caution.
Source Objects
Select one or more objects from the Source Objects section. Click the
icon to add the objects to the Selected Objects section. NoteYou can select columns, tables, or schemas as the objects to be migrated. If you select tables or columns as the objects to be migrated, DTS does not migrate other objects, such as views, triggers, or stored procedures, to the destination database.
Selected Objects
To rename an object that you want to migrate to the destination instance, right-click the object in the Selected Objects section. For more information, see Map the name of a single object.
To rename multiple objects at a time, click Batch Edit in the upper-right corner of the Selected Objects section. For more information, see Map multiple object names at a time.
NoteIf you use the object name mapping feature, objects that depend on the renamed objects may fail to be migrated.
To set WHERE conditions to filter data, right-click the table to be migrated in Selected Objects and set filter conditions in the dialog box that appears.
To select SQL operations to be migrated at the database or table level, right-click the object to be migrated in Selected Objects and select the SQL operations that you want to migrate in the dialog box that appears.
Click Next: Advanced Settings to configure advanced settings.
Parameter
Description
Dedicated Cluster for Task Scheduling
By default, DTS schedules the data migration task to the shared cluster if you do not specify a dedicated cluster. If you want to improve the stability of data migration tasks, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster.
Retry Time for Failed Connections
The retry time range for failed connections. If the source or destination database fails to be connected after the data migration task is started, DTS immediately retries a connection within the retry time range. Valid values: 10 to 1,440. Unit: minutes. Default value: 720. We recommend that you set the parameter to a value greater than 30. If DTS is reconnected to the source and destination databases within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails.
NoteIf you specify different retry time ranges for multiple data migration tasks that share the same source or destination database, the value that is specified later takes precedence.
When DTS retries a connection, you are charged for the DTS instance. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at the earliest opportunity after the source database and destination instance are released.
Retry Time for Other Issues
The retry time range for other issues. For example, if DDL or DML operations fail to be performed after the data migration task is started, DTS immediately retries the operations within the retry time range. Valid values: 1 to 1440. Unit: minutes. Default value: 10. We recommend that you set the parameter to a value greater than 10. If the failed operations are successfully performed within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails.
ImportantThe value of the Retry Time for Other Issues parameter must be smaller than the value of the Retry Time for Failed Connections parameter.
Enable Throttling for Full Data Migration
Specifies whether to enable throttling for full data migration. During full data migration, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers. You can enable throttling for full data migration based on your business requirements. To configure throttling, you must configure the Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) parameters. This reduces the loads of the destination database server.
NoteYou can configure this parameter only if you select Full Data Migration for the Migration Types parameter.
Enable Throttling for Incremental Data Migration
Specifies whether to enable throttling for incremental data migration. To configure throttling, you must configure the RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) parameters. This reduces the loads of the destination database server.
NoteYou can configure this parameter only if you select Incremental Data Migration for the Migration Types parameter.
Environment Tag
The environment tag that is used to identify the DTS instance. You can select an environment tag based on your business requirements. In this example, you do not need to configure this parameter.
Configure ETL
Specifies whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:
Yes: configures the ETL feature. You can enter data processing statements in the code editor. For more information, see Configure ETL in a data migration or data synchronization task.
No: does not configure the ETL feature.
Monitoring and Alerting
Specifies whether to configure alerting for the data migration task. If the task fails or the migration latency exceeds the specified threshold, the alert contacts receive notifications. Valid values:
No: does not configure alerting.
Yes: configures alerting. In this case, you must also configure the alert threshold and alert notification settings. For more information, see the Configure monitoring and alerting when you create a DTS task section of the Configure monitoring and alerting topic.
Click Next Step: Data Verification to configure the data verification task.
For more information about how to use the data verification feature, see Configure a data verification task.
Save the task settings and run a precheck.
To view the parameters to be specified when you call the relevant API operation to configure the DTS task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.
If you do not need to view or have viewed the parameters, click Next: Save Task Settings and Precheck in the lower part of the page.
NoteBefore you can start the data migration task, DTS performs a precheck. You can start the data migration task only after the task passes the precheck.
If the task fails to pass the precheck, click View Details next to each failed item. After you analyze the causes based on the check results, troubleshoot the issues. Then, run a precheck again.
If an alert is triggered for an item during the precheck:
If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.
If the alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur, and your business may be exposed to potential risks.
Purchase an instance.
Wait until Success Rate becomes 100%. Then, click Next: Purchase Instance.
On the Purchase Instance page, configure the Instance Class parameter for the data migration instance. The following table describes the parameters.
Section
Parameter
Description
New Instance Class
Resource Group
The resource group to which the data migration instance belongs. Default value: default resource group. For more information, see What is Resource Management?
Instance Class
DTS provides instance classes that vary in the migration speed. You can select an instance class based on your business scenario. For more information, see Instance classes of data migration instances.
Read and agree to Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.
Click Buy and Start. In the message that appears, click OK.
You can view the progress of the task on the Data Migration page.
NoteIf a data migration task cannot be used to migrate incremental data, the task automatically stops. The Completed is displayed in the Status section.
If a data migration task can be used to migrate incremental data, the task does not automatically stop. The incremental data migration task never stops or completes. The Running is displayed in the Status section.
Appendix 1: SQL operations that support incremental migration
DML operations
INSERT, UPDATE, DELETE
If an UPDATE operation updates only the large fields, DTS does not migrate the operation.
DDL operations
CREATE TABLE
NoteIf a CREATE TABLE operation creates a partitioned table or a table that contains functions, DTS does not migrate the operation.
ALTER TABLE
ALTER TABLE operations include only ADD COLUMN and DROP COLUMN.
DROP TABLE
CREATE INDEX, DROP INDEX
Transactional DDL statements cannot be migrated. For example, DTS does not migrate an SQL operation that contains DDL operations on multiple columns or an SQL operation that contains both DDL operations and DML operations. Data loss may occur after such SQL operations are migrated.
DTS does not migrate DDL operations that contain user-defined types.
DTS does not migrate online DDL operations.
DTS does not migrate DDL operations performed on objects whose names contain reserved keywords.
DTS does not migrate DDL operations performed in system stored procedures.
DTS does not migrate the TRUNCATE TABLE operation.
Appendix 2: Objects that support structure migration
DTS supports schema migration for the following types of objects: table, view, trigger, synonym, SQL stored procedure, SQL function, plan guide, user-defined type, rule, default, and sequence.
DTS does not migrate the schemas of assemblies, service brokers, full-text indexes, full-text catalogs, distributed schemas, distributed functions, Common Language Runtime (CLR) stored procedures, CLR scalar-valued functions, CLR table-valued functions, internal tables, systems, or aggregate functions.