Data Transmission Service:Data migration between ApsaraDB RDS instances

Prerequisites

The source and destination ApsaraDB RDS instances must use one of the following database engines:

Usage notes

• Data migration reads from the source and copies data to the destination without deleting source data. How data migration works.

• A full data migration consumes read and write resources on both the source and destination databases, increasing their server load. This added load can degrade performance or cause service unavailability, especially if your databases have poor performance, low specifications, or are already busy (for example, with many slow SQL queries, tables without primary keys, or deadlocks in the destination database). Before you start the migration, evaluate the performance of both databases and perform the migration during off-peak hours, for example, when CPU utilization is below 30%.

• If a source table to be migrated lacks a primary key or unique constraint, duplicate data may occur in the destination database.

• To ensure data consistency, do not write new data to the source ApsaraDB RDS instance during full data migration.

• DTS automatically retries failed tasks. Stop or release the task before switching services to prevent resumed tasks from overwriting destination data.

• DTS automatically creates databases in the destination instance. If a source database name does not comply with ApsaraDB RDS naming conventions, manually create the database in the destination instance before configuring the task.

  Note

  ApsaraDB RDS naming conventions and database creation:(Deprecated, redirected to "Step 1") Create a database and an account.

• When migrating between ApsaraDB RDS for PostgreSQL instances, sequence values in the destination do not continue from the source maximum after switchover. Before switching, query the maximum value of each source sequence and set it as the starting value in the destination. Run the following command to query source sequence values:

  do language plpgsql $$
  declare
    nsp name;
    rel name;
    val int8;
  begin
    for nsp,rel in select nspname,relname from pg_class t2 , pg_namespace t3 where t2.relnamespace=t3.oid and t2.relkind='S'
    loop
      execute format($_$select last_value from %I.%I$_$, nsp, rel) into val;
      raise notice '%',
      format($_$select setval('%I.%I'::regclass, %s);$_$, nsp, rel, val+1);
    end loop;
  end;
  $$;

Billing

Migration types

• Schema migration

  DTS migrates schema definitions of the selected objects to the destination instance.

• Full data migration

  DTS migrates all existing data of the selected objects from the source instance to the destination instance.

• Incremental data migration

  After full data migration completes, DTS continuously synchronizes incremental changes from source to destination, enabling migration with no downtime.

Supported SQL operations for incremental migration

Database account permissions

Procedure

1. Log on to the Data Transmission Service (DTS) console.

   Note

   If you are automatically redirected to the Data Management (DMS) console, you can click the icon in the lower-right corner and then click to return to the classic DTS console.

2. In the navigation pane on the left, click Data Migration.

3. At the top of the Migration Tasks page, select the region where the destination instance is located.

4. In the upper-right corner of the page, click Create Data Migration Task.

5. Configure the source and destination databases.

   Section	Parameter	Description
   N/A	Task Name	DTS auto-generates a task name. Use a descriptive name for easy identification. Names do not need to be unique.
   Source Database	Instance Type	Select RDS Instance.
   	Instance Region	Select the region of the source ApsaraDB RDS instance.
   	RDS Instance ID	Select the source ApsaraDB RDS instance ID. Note The source and destination can be the same or different RDS instances. DTS supports migration between two instances or within a single instance.
   	Database Name	Enter the name of the source database in the ApsaraDB RDS for PostgreSQL instance. Note This parameter is displayed and required only when the engine of the ApsaraDB RDS instance is PostgreSQL.
   	Database Account	Enter the database account of the source ApsaraDB RDS instance. Required permissions: Permissions required for database accounts.
   	Database Password	Enter the password for the database account. Note After you enter the source database information, you can click Test Connectivity next to Database Password to verify that the information is correct. If the information is correct, the message Passed is displayed. If the message Failed is displayed, click Diagnose next to the Failed message and adjust the source database information based on the prompts.
   	Connection Method	Select Non-encrypted or SSL-encrypted based on your requirements. If you select SSL-encrypted, you must first enable SSL encryption for the ApsaraDB RDS instance. Configure SSL encryption. Note This parameter is displayed and required only when the engine of the ApsaraDB RDS instance is MySQL. The Encryption parameter is supported only in the China (Hong Kong) region and regions in the Chinese mainland.
   Destination Database	Instance Type	Select RDS Instance.
   	Instance Region	Select the region of the destination ApsaraDB RDS instance.
   	RDS Instance ID	Select the destination ApsaraDB RDS instance ID. Note The source and destination can be the same or different RDS instances. DTS supports migration between two instances or within a single instance.
   	Database Name	Enter the destination database name in the ApsaraDB RDS for PostgreSQL instance. The name can differ from the source database name. Note This parameter is displayed and required only when the engine of the ApsaraDB RDS instance is PostgreSQL.
   	Database Account	Enter the database account of the destination ApsaraDB RDS instance. Required permissions: Permissions required for database accounts.
   	Database Password	Enter the password for the database account. Note After you enter the destination database information, you can click Test Connectivity after Database Password to verify that the entered information is correct. If the information is correct, a Passed message is displayed. If a Failed message is displayed, click Diagnose after Failed and adjust the destination database information based on the prompts.
   	Connection Method	Select Non-encrypted or SSL-encrypted based on your requirements. If you select SSL-encrypted, you must first enable SSL encryption for the ApsaraDB RDS instance. Configure SSL encryption. Note This parameter is displayed and required only when the engine of the ApsaraDB RDS instance is MySQL. The Encryption parameter is supported only in the China (Hong Kong) region and regions in the Chinese mainland.

6. Click Set Whitelist and Next in the lower-right corner.

   DTS automatically adds its server CIDR blocks for the selected region to the whitelist of your ApsaraDB RDS instance. For the CIDR blocks of DTS servers, see CIDR blocks of DTS servers.

   Warning

   Adding public IP addresses of DTS servers, whether automatically or manually, may introduce security risks. By using this product, you acknowledge and accept these potential risks. You are responsible for implementing basic security measures, such as using strong passwords, restricting open ports, using authentication for internal APIs, regularly reviewing and limiting unnecessary network segments, or connecting by using private connections such as Express Connect, VPN Gateway, or Smart Access Gateway.

7. Select the migration types and migration objects.

   Parameter	Description
   Migration Types	Select the migration types based on your requirements. For a full migration only, select Schema Migration and Full Data Migration. For zero-downtime migration, select all three types: Schema Migration, Full Data Migration, and Incremental Data Migration. Note If you do not select Incremental Data Migration, do not write new data to the source instance during migration to ensure data consistency.
   Migration Objects	In the Available box, select the objects to migrate and click to move them to the Selected Objects box. Note You can select objects at the database, table, or column level. If you select only tables or columns, other objects such as views, triggers, and stored procedures are not migrated. By default, object names in the destination database are the same as in the source database. To rename an object in the destination database, use the object name mapping feature. For more information, see Object name mapping. Using the object name mapping feature may cause the migration of dependent objects to fail.
   Specify whether to rename objects	To rename migration objects in the destination instance, use the object name mapping feature. Map databases, tables, and columns.
   Specify the retry time range for failed connections to the source or destination database	By default, DTS retries for 720 minutes (12 hours) if it loses connection to the source or destination database. You can customize this duration. The task resumes automatically if the connection is restored within the retry window. Note DTS charges continue during retries. Set the retry duration based on your needs, or release the DTS instance promptly after the source and destination instances are released.

8. After you complete these settings, click Pre-check and Start in the lower-right corner of the page.

   Note

   • DTS performs a precheck before the task starts. The task can start only after it passes the precheck.

   • If the precheck fails, click the icon next to the failed item to view details.

     • Fix the issues based on the details, and then run the precheck again.

     • For warnings that do not require a fix, you can select Ignore or Ignore and Rerun Precheck to run the precheck again.

9. After the task passes the precheck, click Next.

10. On the Confirm Order page, select a Instance Class and select the Data Transmission Service (Pay-As-You-Go) Terms of Service checkbox.

11. Click Buy and Start. The migration task starts.

    • Full data migration

      Do not manually stop the migration task. This can cause data loss. Wait for the task to finish. The task stops automatically.

    • Incremental data migration

      The migration task does not stop automatically. You must stop it manually.

      Note

      Stop the task manually at a suitable time, such as during off-peak hours or when you are ready to switch your business to the destination instance.

      1. Wait until the task status changes to Incremental Data Migration and shows Undelayed. Then, stop writing data to the source database for a few minutes. The status of Incremental Data Migration may show a latency.

      2. Wait until the status of Incremental Data Migration changes back to Undelayed. Then, stop the migration task manually. The progress of schema migration and full data migration both show 100%. To pause the migration task, select the task and click Pause in the batch operations bar at the bottom.