All Products
Search

This Product

    Data Transmission Service:Synchronize Db2 for LUW to self-managed Kafka

    Document Center

    Data Transmission Service:Synchronize Db2 for LUW to self-managed Kafka

    Last Updated:Aug 26, 2026

    Use Data Transmission Service (DTS) to synchronize data from Db2 for LUW to a self-managed Kafka cluster.

    Prerequisites

    • You have created a destination self-managed Kafka cluster.

      Note

      For the supported versions of the source and destination databases, see synchronization solution overview.

    • If full data synchronization is required, the destination database must have more storage space than the source database.

    Limitations

    Note

    DTS does not synchronize foreign keys from the source database to the destination database. Therefore, cascade and delete operations on the source database are not synchronized to the destination database.

    Limit type

    Description

    Limits on the source database

    • The server that hosts the source database must have sufficient outbound bandwidth. Otherwise, the data synchronization speed decreases.

    • The tables to be synchronized must have PRIMARY KEY or UNIQUE constraints, and all fields must be unique. Otherwise, the destination database may contain duplicate data records.

    • If you select tables as the objects to be synchronized and you want to modify the tables in the destination database, such as renaming tables or columns, you can synchronize up to 5,000 tables in a single task. If you synchronize more than 5,000 tables, a request error occurs. In this case, we recommend that you split the tables across multiple tasks or synchronize the entire database.

    • The data logging feature must be enabled. Otherwise, error messages are returned during precheck, and the data synchronization task cannot be started.

      Note

      If you perform only incremental data synchronization, the data logs of the source database must be retained for more than 24 hours. If you perform both full and incremental data synchronization, the data logs must be retained for at least seven days. Otherwise, DTS may fail to obtain the data logs and the task may fail. In exceptional circumstances, data inconsistency or loss may occur. After full data synchronization is complete, you can set the retention period to more than 24 hours. Make sure that you set the retention period based on these requirements. Otherwise, the service reliability or performance stated in the SLA of DTS cannot be guaranteed.

    • Do not run DDL operations that change database or table schemas during schema synchronization or full synchronization. Otherwise, the synchronization task fails.

      Note

      During full synchronization, DTS queries the source database. This creates metadata locks that may block DDL operations on the source database.

    Other limits

    • The synchronization of indexes, partitions, views, stored procedures, functions, triggers, and foreign keys is not supported.

    • DTS synchronizes incremental data from a Db2 for LUW database to the destination database based on the CDC replication technology of Db2 for LUW. The CDC replication technology has its own limits. For more information, see General data restrictions for SQL Replication.

    • Evaluate the impact of data synchronization on the performance of the source and destination databases before you start. We recommend that you synchronize data during off-peak hours. During initial full data synchronization, DTS uses read and write resources of the source and destination databases, which may increase the load on the database servers.

    • During initial full data synchronization, concurrent INSERT operations cause fragmentation in the destination tables. After full data synchronization is complete, the used tablespace of the destination database is larger than that of the source database.

    • During data synchronization, we recommend that you use only DTS to write data to the destination database to prevent data inconsistency.

    • If a task fails, DTS support staff will attempt to restore it within eight hours. During restoration, they may restart the task or adjust its parameters.

      Note

      Only DTS task parameters are modified—not database parameters. Parameters that may be adjusted include those listed in Modify instance parameters.

    Special cases

    • If you perform a primary/secondary switchover on the source database while the data synchronization task is running, the task fails.

    • DTS calculates synchronization latency based on the timestamp of the latest synchronized data record in the destination database and the current timestamp in the source database. If no DML operation is performed on the source database for a long time, the synchronization latency may be inaccurate. If the synchronization latency is too high, you can perform a DML operation on the source database to update the latency.

      Note

      If you select an entire database as the object to be synchronized, you can create a heartbeat table. The heartbeat table is updated or receives data every second.

    • During data synchronization, if the destination ApsaraMQ for Kafka instance is scaled, you must restart the instance.

    Billing

    Synchronization type

    Pricing

    Schema synchronization and full data synchronization

    Free of charge.

    Incremental data synchronization

    Charged. For more information, see Billing overview.

    Synchronization topologies

    • One-way one-to-one synchronization

    • One-way one-to-many synchronization

    • One-way cascade synchronization

    • One-way many-to-one synchronization

    For details about synchronization topologies, see Data Synchronization Topologies.

    Supported SQL operations

    Type

    SQL statement

    DML

    INSERT, UPDATE, and DELETE

    Prerequisites

    Procedure

    1. Go to the data synchronization task list page in the destination region. You can do this in one of two ways.

      DTS console

      1. Log on to the DTS console.

      2. In the navigation pane on the left, click Data Synchronization.

      3. In the upper-left corner of the page, select the region where the synchronization instance is located.

      DMS console

      Note

      The actual steps may vary depending on the mode and layout of the DMS console. For more information, see Simple mode console and Customize DMS console layout and style.

      1. Log on to the DMS console.

      2. In the top menu bar, choose Data + AI > DTS (DTS) > Data Synchronization.

      3. To the right of Data Synchronization Tasks, select the region of the synchronization instance.

    2. Click Create Task to open the task configuration page.

    3. Configure the source and destination databases.

      Category

      Parameter

      Description

      N/A

      Task Name

      DTS automatically generates a task name. We recommend that you specify a descriptive name for easy identification. The name does not need to be unique.

      Source Database

      Select Existing Connection

      • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

        Note

        In the DMS console, this configuration item is Select a DMS database instance.

      • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select DB2 for LUW.

      Connection Type

      Select a connection type based on where the source database is deployed. In this example, Self-managed Database on ECS is used.

      Note

      If the source instance is a self-managed database, you must complete the required preparations. For more information, see Preparation overview.

      Instance Region

      Select the region where the Db2 for LUW database is located.

      Replicate Data Across Alibaba Cloud Accounts

      For this example, select No, as the database instance belongs to the current Alibaba Cloud account.

      ECS Instance ID

      Enter the ID of the ECS instance that hosts the DB2 for LUW database.

      Port

      Enter the service port of the Db2 for LUW database. The default port is 50000.

      Database Name

      Enter the name of the Db2 for LUW database that contains the objects to synchronize.

      Database Account

      Enter the Db2 for LUW database account. For information about the required permissions, see Create an account and grant permissions.

      Database Password

      Enter the password for the specified database account.

      Destination Database

      Select Existing Connection

      • Select the registered database instance with DTS from the drop-down list. The database information below is automatically configured.

        Note

        In the DMS console, this configuration item is Select a DMS database instance.

      • If you have not registered the database instance or do not need to use a registered instance, manually configure the database information below.

      Database Type

      Select Kafka.

      Connection Type

      Select a connection type based on where the destination database is deployed. In this example, Self-managed Database on ECS is used.

      Note

      If the destination instance is a self-managed database, you must complete the required preparations. For more information, see Preparation overview.

      Instance Region

      Select the region where the destination Kafka cluster is deployed.

      ECS Instance ID

      Select the ID of the ECS instance where the Kafka cluster is deployed.

      Note

      If the Kafka instance is deployed as a cluster, select the ECS instance ID of any node. DTS automatically retrieves the topic information of all nodes in the cluster.

      Port

      Enter the service port of the self-managed Kafka cluster. The default port is 9092.

      Database Account

      Enter the username of the self-managed Kafka cluster. If authentication is disabled, leave this parameter empty.

      Database Password

      Enter the password for the database account. If authentication is disabled, leave this parameter empty.

      Kafka Version

      Select the version of your self-managed Kafka cluster.

      Encryption

      Select Non-encrypted or SCRAM-SHA-256 based on your business and security requirements.

      Topic

      Select the topic to which data is written.

      Use Kafka Schema Registry

      Kafka Schema Registry provides a RESTful interface for storing and retrieving Avro schemas.

      • No: Do not use Kafka Schema Registry.

      • Yes: Use Kafka Schema Registry. You must enter the URL or IP address that is registered in Kafka Schema Registry for your Avro schema.

    4. After the configuration is complete, click Test Connectivity and Proceed at the bottom of the page. In the CIDR Blocks of DTS Servers dialog box, click Test Connectivity.

      Note

      Ensure that the IP address blocks of the DTS service are added to the security settings of the source and destination databases, either automatically or manually, to allow access from DTS servers. For more information, see Add the IP address whitelist of DTS servers.

    5. Configure the task objects.

      1. On the Configure Objects page, specify the objects to synchronize.

        Parameter

        Description

        Synchronization Types

        The synchronization types. By default, Incremental Data Synchronization is selected. You must also select Schema Synchronization and Full Data Synchronization. After the precheck is complete, DTS synchronizes the historical data of the selected objects from the source database to the destination cluster. The historical data is the basis for subsequent incremental synchronization.

        Processing Mode of Conflicting Tables

        • Precheck and Report Errors: DTS checks for tables in the destination database with the same name. If a matching table name is found, the precheck fails and the task does not start. Otherwise, the check passes.

          Note

          If you cannot easily delete or rename the conflicting table in the destination database, you can map the table to a different name. For more information, see Map table and column names.

        • Ignore Errors and Proceed: Skips the check for tables with the same name in the destination database.

          Warning

          If you select Ignore Errors and Proceed, data inconsistency may occur, which poses risks to your business. For example:

          • If the table schemas are consistent and the destination database has a record with the same primary key value as a record in the source database:

            • During full data synchronization, DTS does not overwrite the existing record in the destination database. The corresponding record from the source database is skipped.

            • During incremental data synchronization, the record from the source database overwrites the existing record in the destination database.

          • If table schemas are inconsistent, the task might fail, or only partial data may be synchronized.

        Data Format in Kafka

        Select the data format for Kafka delivery. For more information, see Data storage formats in message queues.

        Kafka Data Compression Format

        Select the compression format for Kafka message compression based on your business requirements.

        • LZ4 (default): The compression ratio is low, but the compression speed is high.

        • GZIP: The compression ratio is high, but the compression speed is low.

          Note

          CPU consumption is high.

        • Snappy: The compression ratio and compression speed are medium.

        Policy for Shipping Data to Kafka Partitions

        Select a strategy based on your business requirements.

        Message acknowledgement mechanism

        Select a message acknowledgment mechanism based on your business requirements.

        Topic That Stores DDL Information

        Select the topic that is used to store DDL information from the drop-down list.

        Note

        If not specified, the DDL information is stored in the topic that receives the data by default.

        Capitalization of Object Names in Destination Instance

        Configure the case-sensitivity policy for database, table, and column names in the destination instance. By default, the DTS default policy is selected. You can also choose to use the default policy of the source or destination database. For more information, see Case policy for destination object names.

        Source Objects

        In the Source Objects box, click the objects, and then click 向右 to move them to the Selected Objects box.

        Note

        You can select objects at the database, table, or column level. If you select only tables or columns, DTS does not synchronize other object types (such as views, triggers, and stored procedures).

        Selected Objects

        No additional configuration is required in this example. You can use the mapping feature to configure the topic name, partition count, and partition key for the source table in the destination Kafka instance. For more information, see Mapping information.

        Note

        • If you map an object name, dependent objects might fail to synchronize.

        • To select specific SQL operations for incremental data synchronization, right-click an object in the Selected Objects box and choose the operations from the shortcut menu.

      2. Click Next: Advanced Settings.

        Parameter

        Description

        Dedicated Cluster for Task Scheduling

        By default, DTS uses a shared cluster for tasks, so you do not need to make a selection. For greater task stability, you can purchase a dedicated cluster to run the DTS synchronization task. For more information, see What is a DTS dedicated cluster?.

        Retry Time for Failed Connections

        If the connection to the source or destination database fails after the synchronization task starts, DTS reports an error and immediately begins to retry the connection. The default retry duration is 720 minutes. You can customize the retry time to a value from 10 to 1,440 minutes. We recommend a duration of 30 minutes or more. If the connection is restored within this period, the task resumes automatically. Otherwise, the task fails.

        Note

        • If multiple DTS instances (e.g., Instance A and B) share a source or destination, DTS uses the shortest configured retry duration (e.g., 30 minutes for A, 60 for B, so 30 minutes is used) for all instances.

        • DTS charges for task runtime during connection retries. Set a custom duration based on your business needs, or release the DTS instance promptly after you release the source/destination instances.

        Retry Time for Other Issues

        If a non-connection issue (e.g., a DDL or DML execution error) occurs, DTS reports an error and immediately retries the operation. The default retry duration is 10 minutes. You can also customize the retry time to a value from 1 to 1,440 minutes. We recommend a duration of 10 minutes or more. If the related operations succeed within the set retry time, the synchronization task automatically resumes. Otherwise, the task fails.

        Important

        The value of Retry Time for Other Issues must be less than that of Retry Time for Failed Connections.

        Enable Throttling for Full Data Synchronization

        During full data synchronization, DTS consumes read and write resources from the source and destination databases, which can increase their load. To mitigate pressure on the destination database, you can limit the migration rate by setting Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s).

        Note

        Enable Throttling for Incremental Data Synchronization

        You can also limit the incremental synchronization rate to reduce pressure on the destination database by setting RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s).

        Environment Tag

        Select an environment tag to identify the instance. No selection is required for this example.

        Configure ETL

        Choose whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values:

        Monitoring and Alerting

        Choose whether to set up alerts. If the synchronization fails or the latency exceeds the specified threshold, DTS sends a notification to the alert contacts.

    6. Save the task and perform a precheck.

      • To view the parameters for configuring this instance via an API operation, hover over the Next: Save Task Settings and Precheck button and click Preview OpenAPI parameters in the tooltip.

      • If you have finished viewing the API parameters, click Next: Save Task Settings and Precheck at the bottom of the page.

      Note

      • Before a synchronization task starts, DTS performs a precheck. You can start the task only if the precheck passes.

      • If the precheck fails, click View Details next to the failed item, fix the issue as prompted, and then rerun the precheck.

      • If the precheck generates warnings:

        • For non-ignorable warning, click View Details next to the item, fix the issue as prompted, and run the precheck again.

        • For ignorable warnings, you can bypass them by clicking Confirm Alert Details, then Ignore, and then OK. Finally, click Precheck Again to skip the warning and run the precheck again. Ignoring precheck warnings may lead to data inconsistencies and other business risks. Proceed with caution.

    7. Purchase the instance.

      1. When the Success Rate reaches 100%, click Next: Purchase Instance.

      2. On the Purchase page, select the billing method and link specifications for the data synchronization instance. For more information, see the following table.

        Category

        Parameter

        Description

        New Instance Class

        Billing Method

        • Subscription: You pay upfront for a specific duration. This is cost-effective for long-term, continuous tasks.

        • Pay-as-you-go: You are billed hourly for actual usage. This is ideal for short-term or test tasks, as you can release the instance at any time to save costs.

        Resource Group Settings

        The resource group to which the instance belongs. The default is default resource group. For more information, see What is resource management?.

        Instance Class

        DTS offers synchronization specifications at different performance levels that affect the synchronization rate. Select a specification based on your business requirements. For more information, see Data synchronization link specifications.

        Subscription Duration

        In subscription mode, select the duration and quantity of the instance. Monthly options range from 1 to 9 months. Yearly options include 1, 2, 3, or 5 years.

        Note

        This option appears only when the billing method is Subscription.

      3. Read and select the checkbox for Data Transmission Service (Pay-as-you-go) Service Terms.

      4. Click Buy and Start, and then click OK in the OK dialog box.

        You can monitor the task progress on the data synchronization page.

    Mappings

    1. In the Selected Objects area, hover over the destination topic name.

    2. Click the Edit button that appears next to the destination topic name.

    3. In the Edit Table dialog box, configure the mapping settings.

      Note

      • The Edit Schema dialog box is for database-level settings and has fewer configurable parameters than the Edit Table dialog box, which is for table-level settings.

      • If you do not synchronize an entire database, you cannot modify the Name of target Topic and Number of Partitions parameters in the Edit Schema dialog box.

      Parameter

      Description

      Name of target Topic

      The destination topic to which DTS writes data from the source table. By default, this is the Topic that you selected in the Destination Database section during the Configurations for Source and Destination Databases step.

      Important

      • If the destination database is an Alibaba Cloud Message Queue for Apache Kafka instance, the specified topic must exist in the destination Kafka instance. Otherwise, the data synchronization task fails. If the destination database is a self-managed Kafka database and the synchronization instance includes a schema synchronization task, DTS attempts to create the specified topic in the destination database.

      • If you change the Name of target Topic, DTS writes the data to the new topic.

      Filter Conditions

      For more information, see Set Filter Conditions.

      Number of Partitions

      The number of partitions in the destination topic.

      Partition Key

      This parameter is available when you set Policy for Shipping Data to Kafka Partitions to Ship Data to Separate Partitions Based on Hash Values of Primary Keys. You can select one or more columns as the partition key. DTS calculates a hash value based on this key and distributes rows across the partitions of the destination topic.

      Note

      You can select Partition Key only in the Edit Table dialog box.

    4. Click OK.

    FAQ