All Products
Search

This Product

    Data Transmission Service:Synchronize data from an ApsaraDB RDS for MySQL instance to a self-managed Doris database

    Document Center

    Data Transmission Service:Synchronize data from an ApsaraDB RDS for MySQL instance to a self-managed Doris database

    Last Updated:Sep 16, 2026

    Data Transmission Service (DTS) allows you to synchronize data from a MySQL database (for example, a self-managed MySQL database or an ApsaraDB RDS for MySQL) to a Doris database, which meets massive data analysis requirements. This topic describes the procedure by using an ApsaraDB RDS for MySQL instance as the source database and a Doris database deployed on an ECS instance as the destination database.

    Prerequisites

    A destination Doris database whose storage capacity is larger than the used storage capacity of the source ApsaraDB RDS for MySQL instance has been created.

    Note

    For information about the versions of the source and destination databases that are supported, see Data synchronization scenarios.

    Precautions

    Category

    Description

    Limits on the source database

    • Requirements for the objects to be synchronized:

      • If all tables to be synchronized have a primary key or unique constraint:

        Ensure that the columns that form the key or constraint contain unique values.

      • If the synchronization objects include tables that have neither a primary key nor a unique constraint:

        When you configure the data synchronization instance, we recommend that you select Schema Synchronization for Synchronization Types. Then, in the Configurations for Databases, Tables, and Columns step, set the Engine for the table to duplicate. Otherwise, the instance may fail or data loss may occur.

        Note

        During schema synchronization, DTS adds columns to the destination table. For more information, see Additional columns.

    • If you synchronize at the table level and need to edit mappings (such as column name mapping), each synchronization task supports up to 1,000 tables. If you exceed this limit, the task fails with an error. To fix this, split the tables across multiple tasks or configure a full-database synchronization task.

    • Binary logs:

      • ApsaraDB RDS for MySQL enables binary logging by default. Ensure that the binlog_row_image parameter is set to full. Otherwise, the precheck fails and the synchronization task cannot start. For instructions, see Configure instance parameters.

        Important

        • If your source instance is a self-managed MySQL database, enable binary logging and set binlog_format to row and binlog_row_image to full.

        • If your self-managed MySQL database is a dual-primary cluster (where both nodes act as primary and secondary), enable the log_slave_updates parameter so DTS can capture all binary log events. For instructions, see Create an account and configure binary logging for a self-managed MySQL database.

      • The local binary logs for an ApsaraDB RDS for MySQL instance must be retained for at least three days (seven days is recommended). For a self-managed MySQL database, retain local binary logs for at least seven days. Otherwise, DTS may fail to retrieve binary logs, causing the task to fail. In extreme cases, this may cause data inconsistency or data loss. Issues caused by binary log retention periods shorter than DTS requires are not covered under the DTS SLA.

        Note

        To configure the retention period for local binary logs on an ApsaraDB RDS for MySQL instance, see Automatically delete local logs.

    • Do not run DDL operations that change database or table schemas during schema synchronization or full synchronization. Otherwise, the synchronization task fails.

      Note

      During full synchronization, DTS queries the source database. This creates metadata locks that may block DDL operations on the source database.

    • Data generated by changes that do not write to binary logs—such as data restored from physical backups or created by cascade operations—is not synchronized to the destination database.

      Note

      If this occurs, remove the affected database or table from the synchronization objects. Then add it back. You can do this only if your business allows it. For more information, see Modify synchronization objects.

    • If your source database is MySQL 8.0.23 or later and contains invisible hidden columns, DTS may not read those columns. This may cause data loss.

      Note

      Run the ALTER TABLE <table_name> ALTER COLUMN <column_name> SET VISIBLE; command to make the hidden column visible. For more information, see Invisible Columns.

    Other limits

    • Only synchronization to tables that use the Unique or Duplicate engine in the Doris database is supported. If the destination table uses the Duplicate engine, duplicate data may occur in the destination database in the following cases. You can deduplicate the data based on the additional columns (_is_deleted, _version, and _record_id):

      • The synchronization instance has been retried.

      • The synchronization instance has been restarted.

      • After the synchronization instance starts running, DML operations are performed on the same row of data two or more times.

        Note

        If the destination table uses the Duplicate engine, DTS converts UPDATE or DELETE statements to INSERT statements.

    • When you configure parameters in the Selected Objects section, you can only set the bucket_count (number of buckets) parameter.

      Note

      The value of the bucket_count parameter must be a positive integer. The default value is auto.

    • During data synchronization, do not create new clusters in the destination Doris database. Otherwise, the task fails. You can restart the synchronization instance to recover the failed task.

    • The names of databases and tables in a Doris database must start with a letter. If a database name or table name to be synchronized does not start with a letter, you must use the object name mapping feature to modify the name.

    • If the name of an object (database, table, or column) to be synchronized contains Chinese characters, you must use the object name mapping feature to modify the name (for example, change it to English). Otherwise, the task may fail.

    • DTS does not support DDL operations that modify multiple columns at once or consecutive DDL operations on the same table.

    • During data synchronization, do not add BE (Backend) nodes to the Doris database. Otherwise, the task fails. You can restart the synchronization instance to recover the failed task.

    • In a many-to-one table synchronization scenario, where data from multiple source tables is synchronized to a single destination table, ensure that the schemas of all source tables are identical. Otherwise, data inconsistency or task failure may occur.

    • In MySQL, the M in the VARCHAR(M) type indicates the character length. In a Doris database, the N in the VARCHAR(N) type indicates the byte length. If you do not use the schema synchronization feature provided by DTS, we recommend that you set the length of VARCHAR fields in the Doris database to four times the length of the VARCHAR fields in the MySQL database.

    • When you use DMS or the gh-ost tool to perform online DDL operations on the source, DTS synchronizes only the original DDL statements to the destination. In this scenario, DTS does not need to synchronize a large amount of temporary table data, but this may cause tables to be locked at the destination.

      Note

      DTS does not support synchronizing online DDL changes made with tools like pt-online-schema-change. If such changes exist at the source, data may be lost at the destination, or the synchronization instance may fail.

    • During the full data initialization phase, DTS uses certain read and write resources of the source and destination databases, which may increase the database load. We recommend that you evaluate the performance of the source and destination databases before data synchronization, and perform data synchronization during off-peak hours (for example, when the CPU load of the source and destination databases is below 30%).

    • Full data initialization performs concurrent INSERT operations, which causes fragmentation in the tables of the destination database. As a result, after full data initialization is complete, the storage space of the tables in the destination database is larger than that in the source instance.

    • During data synchronization, do not use tools such as pt-online-schema-change to perform online DDL changes on the objects to be synchronized in the source database. Otherwise, the task fails.

    • During DTS synchronization, if data other than that from DTS is written to the destination database, the data in the source and destination databases may be inconsistent.

    • If your ApsaraDB RDS for MySQL instance has Always-Encrypted enabled, full data synchronization is not supported.

      Note

      ApsaraDB RDS for MySQL instances with Transparent Data Encryption (TDE) enabled support schema synchronization, full data synchronization, and incremental data synchronization.

    • During incremental synchronization, DTS employs a batching strategy to reduce the load on the destination. By default, DTS performs a write operation for a single synchronization object at most once every 5 seconds. Consequently, DTS synchronization tasks may exhibit a baseline synchronization latency, typically within 10 seconds. To reduce this baseline latency, you can modify the `selectdb.reservoir.timeout.milliseconds` parameter for the DTS instance in the console to adjust the batching interval; the allowed range is [1000, 10000] milliseconds.

      Note

      Note that a shorter batching interval increases the DTS write frequency, which may raise the load and write response time (RT) on the destination, potentially leading to increased synchronization latency; therefore, please make adjustments based on the load conditions of the destination.

    • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

      Note

      Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

    Special cases

    • For a self-managed MySQL source database:

      • If a primary/secondary switchover occurs in the source database during synchronization, the task fails.

      • DTS calculates latency by comparing the timestamp of the last synchronized record with the current time. If no DML operations run for a long time in the source database, latency reporting may become inaccurate. If latency appears too high, run a DML operation in the source database to update the latency.

        Note

        If you select a full database for synchronization, create a heartbeat table. Update or write to this table every second.

      • DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command in the source database to advance the binary log offset.

      • If your source database is Amazon Aurora MySQL or another clustered MySQL instance, ensure the domain name or IP address used in the task configuration—and its DNS resolution—always points to a read/write (RW) node. Otherwise, synchronization may fail.

    • For an ApsaraDB RDS for MySQL source database:

      • Read-only instances—such as ApsaraDB RDS for MySQL 5.6 read-only instances—that do not record transaction logs cannot serve as source databases.

      • DTS periodically runs the CREATE DATABASE IF NOT EXISTS `test` command in the source database to advance the binary log offset.

    Billing

    Synchronization type

    Pricing

    Schema synchronization and full data synchronization

    Free of charge.

    Incremental data synchronization

    Charged. For more information, see Billing overview.

    SQL operations supported for incremental synchronization

    Operation type

    SQL statement

    DML

    INSERT, UPDATE, DELETE

    DDL

    • ADD COLUMN

    • MODIFY COLUMN

    • CHANGE COLUMN

    • DROP COLUMN, DROP TABLE

    • TRUNCATE TABLE

    • RENAME TABLE

      Important

      The RENAME TABLE operation may cause data inconsistency. For example, if the synchronization object is only a specific table and you rename the table in the source instance during synchronization, the data of the table is not synchronized to the destination database. To prevent this issue, select the entire database to which the table belongs as the synchronization object when you configure the data synchronization task. Make sure that the databases to which the table belongs before and after the RENAME TABLE operation are both included in the synchronization object.

    Permissions required for database accounts

    Database

    Required permissions

    References for creating and granting permissions to accounts

    Source ApsaraDB RDS for MySQL instance

    Read and write permissions on the objects to be synchronized

    Create an account and Modify the permissions of a standard account on an ApsaraDB RDS for MySQL instance

    Destination Doris database

    The database access permission (Usage_priv) and the read and write permissions (Select_priv, Load_priv, Alter_priv, Create_priv, and Drop_priv)

    Authentication and Authorization

    Note

    If the source database account is not created and authorized in the ApsaraDB RDS for MySQL console, make sure that the account has the REPLICATION CLIENT, REPLICATION SLAVE, SHOW VIEW, and SELECT permissions.

    Procedure

    1. Go to the data synchronization task list page in the destination region. You can do this in one of two ways.

      DTS console

      1. Log on to the DTS console.

      2. In the navigation pane on the left, click Data Synchronization.

      3. In the upper-left corner of the page, select the region where the synchronization instance is located.

      DMS console

      Note

      The actual steps may vary depending on the mode and layout of the DMS console. For more information, see Simple mode console and Customize DMS console layout and style.

      1. Log on to the DMS console.

      2. In the top menu bar, choose Data + AI > DTS (DTS) > Data Synchronization.

      3. To the right of Data Synchronization Tasks, select the region of the synchronization instance.

    2. Click Create Task to navigate to the task configuration page.

    3. Configure the source and destination databases.

      Category

      Parameter

      Description

      N/A

      Task Name

      DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

      Source Database

      Select Existing Connection

      • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

        Note

        In the DMS console, this configuration item is Select a DMS database instance.

      • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select MySQL.

      Access Method

      Select Alibaba Cloud Instance.

      Instance Region

      Select the source ApsaraDB RDS for MySQL instance region.

      Replicate Data Across Alibaba Cloud Accounts

      For this example, select No, as the database instance belongs to the current Alibaba Cloud account.

      RDS Instance ID

      Select the source ApsaraDB RDS for MySQL instance ID.

      Database Account

      Enter the database account of the source ApsaraDB RDS for MySQL instance. For information about the required permissions, see Permissions required for database accounts.

      Database Password

      Enter the password for the specified database account.

      Encryption

      Select Non-encrypted or SSL-encrypted as needed. If you set this to SSL-encrypted, you must enable SSL encryption for the RDS for MySQL instance beforehand. For more information, see Use a cloud certificate to quickly enable SSL link encryption.

      Destination Database

      Select Existing Connection

      • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

        Note

        In the DMS console, this configuration item is Select a DMS database instance.

      • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select Doris.

      Access Method

      Select a value based on where the destination database is deployed. In this example, Self-managed Database on ECS.

      Note

      If you use another method to connect a self-managed database to DTS, you must complete the corresponding preparations. For more information, see Preparations Overview.

      Instance Region

      Select the region where the destination Doris database resides.

      ECS Instance ID

      Select the ID of the ECS instance where the destination Doris database resides.

      Note

      If the destination Doris database is deployed on multiple ECS instances (for example, BE or FE nodes are deployed on other ECS instances), you must manually add the CIDR blocks of DTS to the security group of each of the other ECS instances where the nodes are deployed.

      Port Number

      Enter the service port of the destination Doris database. The default port is 9030.

      Database Account

      Enter the database account of the destination Doris database. For information about the required permissions, see Permissions required for database accounts.

      Database Password

      Enter the password for the specified database account.

    4. After the configuration is complete, click Test Connectivity and Proceed at the bottom of the page. In the CIDR Blocks of DTS Servers dialog box, click Test Connectivity.

      Note

      Ensure that the IP address blocks of the DTS service are added to the security settings of the source and destination databases, either automatically or manually, to allow access from DTS servers. For more information, see Add the IP address whitelist of DTS servers.

    5. Configure the task objects.

      1. On the Configure Objects page, specify the objects to synchronize.

        Parameter

        Description

        Synchronization Types

        DTS always selects Incremental Data Synchronization. By default, you must also select Schema Synchronization and Full Data Synchronization. After the precheck, DTS initializes the destination cluster with the full data of the selected source objects, which serves as the baseline for subsequent incremental synchronization.

        Important

        Data types are converted when MySQL data is synchronized to Doris. If you do not select Schema Synchronization, you must create Unique or Duplicate model tables with the corresponding schema in the destination Doris database in advance. For more information, see Data type mappings, Additional columns and Unique key model.

        Processing Mode of Conflicting Tables

        • Precheck and Report Errors: DTS checks for tables with the same name in the destination database. If a conflict exists, DTS reports an error during the precheck and does not start the task.

          Note

          If it is not practical to delete or rename the conflicting table, you can map the table to a different name. For more information, see Map object names.

        • Ignore Errors and Proceed: Skips the check for tables with the same name in the destination database.

          Warning

          If you select Ignore Errors and Proceed, data inconsistency may occur. For example:

          • If the table schemas are consistent, destination records with the same primary or unique key as source records are overwritten.

          • If the table schemas are inconsistent, the task may fail or only partially synchronize data. Proceed with caution.

        Capitalization of Object Names in Destination Instance

        Configure the case-sensitivity policy for database, table, and column names in the destination instance. By default, the DTS default policy is selected. You can also choose to use the default policy of the source or destination database. For more information, see Case policy for destination object names.

        Source Objects

        In the Source Objects box, click the objects, and then click 向右 to move them to the Selected Objects box.

        Note

        The granularity for selecting objects to be synchronized is database or table.

        Selected Objects

        • To change the name of a synchronization object in the destination instance, right-click the object in the Selected Objects box. For more information, see Map object names.

        • If you select Schema Synchronization for Synchronization Types, select tables as the synchronization objects, and need to set the number of buckets (the bucket_count parameter), right-click the table in the Selected Objects box. In the Parameter Settings section, set Enable Parameter Settings to Yes, specify a Value based on your requirements, and then click OK.

        Note

        • To select the incremental SQL operations to be synchronized at the database or table level, right-click the object in the Selected Objects box and select the desired SQL operations in the dialog box that appears.

        • To filter data by using a WHERE clause, right-click the table in the Selected Objects box and set the filter conditions in the dialog box that appears. For more information, see Configure filter conditions.

        • If you use the object name mapping feature, other objects that depend on the renamed object may fail to be synchronized.

      2. Click Next: Advanced Settings.

        Parameter

        Description

        Dedicated Cluster for Task Scheduling

        By default, DTS uses a shared cluster for tasks, so you do not need to make a selection. For greater task stability, you can purchase a dedicated cluster to run the DTS synchronization task. For more information, see What is a DTS dedicated cluster?.

        Retry Time for Failed Connections

        If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails.

        Note

        • If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances.

        • DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.

        Retry Time for Other Issues

        If a non-connection issue (e.g., a DDL or DML execution error) occurs, DTS reports an error and immediately retries the operation. The default retry duration is 10 minutes. You can also customize the retry time to a value from 1 to 1,440 minutes. We recommend a duration of 10 minutes or more. If the related operations succeed within the set retry time, the synchronization task automatically resumes. Otherwise, the task fails.

        Important

        The value of Retry Time for Other Issues must be less than that of Retry Time for Failed Connections.

        Enable Throttling for Full Data Synchronization

        During full data synchronization, DTS consumes read and write resources from the source and destination databases, which can increase their load. To mitigate pressure on the destination database, you can limit the migration rate by setting Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s).

        Note

        Enable Throttling for Incremental Data Synchronization

        You can also limit the incremental synchronization rate to reduce pressure on the destination database by setting RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s).

        Whether to delete SQL operations on heartbeat tables of forward and reverse tasks

        Choose whether DTS writes heartbeat SQL information to the source database while the instance is running.

        • Yes: Does not write heartbeat SQL information to the source database. The DTS instance may display latency.

        • No: Writes heartbeat SQL information to the source database. This may interfere with source database operations like physical backups and cloning.

        Environment Tag

        You can select an environment tag to identify the instance based on your business requirements. In this example, no environment tag is selected.

        Configure ETL

        Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

        Monitoring and Alerting

        Choose whether to set up alerts. If the synchronization fails or the latency exceeds the specified threshold, DTS sends a notification to the alert contacts.

      3. Optional: After completing the preceding configurations, click Next: Configure Database and Table Fields to set the Primary Key Column, Distribution Key, and Engine for the destination tables.

        Note

        • This step is available only if you select Schema Synchronization for Synchronization Types when you configure the synchronization objects. You can set Definition Status to All to modify the settings.

        • You can select multiple columns to form a composite Primary Key Column. You must also select one or more columns from the Primary Key Column as the Distribution Key.

        • For a table without a primary key or a unique constraint, you must select duplicate for the Engine. Otherwise, the task may fail or data loss may occur.

    6. Save the task and perform a precheck.

      • To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.

      • If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

      Note

      • Before a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.

      • If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.

      • If the precheck generates warnings:

        • For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.

        • For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.

    7. Purchase the instance.

      1. When the Success Rate reaches 100%, click Next: Purchase Instance.

      2. On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.

        Category

        Parameter

        Description

        New Instance Class

        Billing Method

        • Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks.

        • Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.

        Resource Group Settings

        The resource group to which the instance belongs. The default is default resource group. For more information, see What is resource management?.

        Instance Class

        DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.

        Subscription Duration

        In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years.

        Note

        This option appears only when the billing method is Subscription.

      3. Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.

      4. Click Buy and Start, and then click OK in the OK dialog box.

        You can monitor the task progress on the data synchronization page.

    Data type mappings

    Category

    MySQL data type

    Doris data type

    Numeric

    TINYINT

    TINYINT

    TINYINT UNSIGNED

    SMALLINT

    SMALLINT

    SMALLINT

    SMALLINT UNSIGNED

    INT

    MEDIUMINT

    INT

    MEDIUMINT UNSIGNED

    BIGINT

    INT

    INT

    INT UNSIGNED

    BIGINT

    BIGINT

    BIGINT

    BIGINT UNSIGNED

    LARGEINT

    BIT(M)

    INT

    Decimal

    Decimal

    Note

    zerofill is not supported.

    Numeric

    Decimal

    Float

    Float

    Double

    DOUBLE

    • BOOL

    • BOOLEAN

    BOOLEAN

    DATE AND TIME

    DATE

    DATEV2

    DATETIME[(fsp)]

    DATETIMEV2

    Timestamp[(fsp)]

    DATETIMEV2

    Time[(fsp)]

    VARCHAR

    YEAR[(4)]

    INT

    STRING

    • CHAR

    • VARCHAR

    VARCHAR

    Important

    To avoid data loss, CHAR and VARCHAR(n) data is converted to VARCHAR(4*n) after being synchronized to Doris.

    • If the data length is not specified, the Doris default value VARCHAR(65533) is used.

    • If the data length exceeds 65533, the data is converted to STRING after being synchronized to Doris.

    • BINARY

    • VARBINARY

    STRING

    • TINYTEXT

    • TEXT

    • MEDIUMTEXT

    • LONGTEXT

    STRING

    • TINYBLOB

    • BLOB

    • MEDIUMBOLB

    • LONGBLOB

    STRING

    ENUM

    STRING

    SET

    STRING

    JSON

    STRING

    Additional columns

    Note

    This table lists the additional columns that DTS automatically adds or that you must manually add to the destination duplicate model table.

    Parameter

    Type

    Default

    Description

    _is_deleted

    Int

    0

    Indicates whether the row is deleted.

    • Insert: The value is 0.

    • Update: The value is 0.

    • Delete: The value is 1.

    _version

    Bigint

    0

    • For data from a full synchronization, the value is 0.

    • For data from an incremental synchronization, the value is the commit timestamp (in seconds) of the change from the source database binlog.

    _record_id

    Bigint

    0

    • For data from a full synchronization, the value is 0.

    • For data from an incremental synchronization, the value is the record ID from the incremental log. This ID uniquely identifies the log entry.

      Note

      The ID is unique and increases monotonically.