All Products
Search

This Product

    Data Transmission Service:Configure ETL

    Document Center

    Data Transmission Service:Configure ETL

    Last Updated:Jun 02, 2026

    Configure an ETL task in DAG mode to cleanse and transform streaming data through a visual drag-and-drop interface.

    Background information

    Note

    This feature is being deprecated and is available for free only to users who have previously used it. New users can no longer access this feature. We recommend that you configure ETL tasks within a data synchronization or data migration instance. For more information, see Configure an ETL task in a DTS data migration or synchronization task.

    • Before you configure an ETL task:

      • Input/Dimension Table refers to the source database of the ETL task.

      • Output refers to the destination database that receives the processed data from the ETL task.

    • DTS supports streaming ETL for data synchronization. Add transformation components between the source and destination databases to process data in real time. For example, join two stream tables into a larger table, or add a field and populate its values with a function.

    Prerequisites

    • ETL tasks are available only in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Guangzhou), and China (Hong Kong).

    • The source database is one of the following types: MySQL, PolarDB for MySQL, Oracle, PostgreSQL, DB2 for iSeries (AS/400), DB2 for LUW, PolarDB-X (Formerly DRDS), PolarDB for PostgreSQL, MariaDB, PolarDB for Oracle, SQL Server, or PolarDB-X 2.0.

    • The destination database is one of the following types: MySQL, PolarDB for MySQL, Oracle, AnalyticDB for MySQL V3.0, PolarDB for PostgreSQL, PostgreSQL, DB2 for LUW, DB2 for iSeries (AS/400), AnalyticDB for PostgreSQL, SQL Server, MariaDB, PolarDB-X (Formerly DRDS), PolarDB for Oracle, or Tablestore.

    • ETL does not support schema migration. Create the required table schemas in the destination database based on your transformation logic. For example, if you join Table A (Columns 1, 2, 3) and Table B (Columns 2, 3, 4) to output Columns 2 and 3, first create a destination table (Table C) with Columns 2 and 3.

    • ETL does not support full data synchronization. Only incremental data can be transformed in real time.

    Usage notes

    • The source and destination databases must be in the same region.

    • All Stream Table must originate from the same instance.

    • Database and table names must be unique.

    • Transformation components have the following limitations:

      • Table Join: You can join only two tables at a time.

      • Field Calculator and Table Record Filter: These components support only a single table structure.

    Workflow

    ETL task workflow

    To create an ETL task, select source databases, transformation components, and a destination database. The following table describes each step.

    Step

    Description

    Reference

    Configure the source database

    Select one or more data source types for the Input/Dimension Table node and configure the following information for each source:

    Note

    You can select the same data source type multiple times.

    1. Configure node information.

    2. Configure field information.

    3. Optional: If you select the Stream Table type, you must also configure time attributes.

    Configure a source database

    Configure transformation components

    Select one or more components from the Transform node and configure the settings for each component:

    Note

    You can use the same transformation component multiple times.

    • Table Join

      1. Configure node information.

      2. Configure field information.

    • Field Calculator

      1. Configure the transformation name.

      2. Configure the calculator.

    • Table Record Filter

      1. Configure the transformation name.

      2. Configure the WHERE condition.

    Configure transformation components

    Configure the destination database

    Select a destination database type for the Output node and configure the following settings:

    1. Configure node information.

    2. Configure field mapping information.

    Configure a destination database

    Procedure

    1. Go to the Streaming ETL page.

      1. Log in to the DTS console.

      2. In the left-side navigation pane, click ETL.

    2. In the upper-left corner, click the Add data flow icon. In the Create Data Flow dialog box, enter a name in the Data Flow Name field, and set Development Method to DAG.

    3. Click Confirm.

    4. Configure the ETL task on the canvas.

      1. Build the ETL workflow.

        Note

        • This example uses an Input/Dimension Table (MySQL) node, a Field Calculator node, and an Output (MySQL) node.

        • You can select one or more source databases, including the same type multiple times.

        • You can use one or more transformation components, including the same component multiple times.

        • You can select only one destination database.

        1. From the component list, drag an Input/Dimension Table (MySQL) node to the canvas.

        2. Click the Input/Dimension Table MySQL-1 node on the canvas. Configure the node information, field information, and time attributes. Configure a source database.

        3. From the component list, drag a Field Calculator node to the canvas.

        4. Hover over the Input/Dimension Table MySQL-1 source node, and then click the dot and drag a line to connect the Input/Dimension Table MySQL-1 node to the Field Calculator-1 node.Field calculator

        5. Click the Field Calculator-1 node on the canvas and configure its settings. Configure transformation components.

        6. From the component list, drag an Output MySQL node to the canvas.

        7. Hover over Field Transformer-1, click the dot, and drag a connection line to connect Field Transformer-1 to the destination database Output MySQL-1.Connect to destination database

        8. Click the Output MySQL-1 node and configure the node information and field mappings. Configure a destination database.

        9. After configuring the nodes, click Generate Flink SQL Validation or Publish.

          • Generate Flink SQL Validation: On success, a Flink SQL validation succeeds. message appears. On failure, click View ETL Validation Details next to Generate Flink SQL Validation, fix the issues, and rerun validation.

          • Publish: The system first runs Flink SQL validation. The precheck starts only after validation succeeds. Click View ETL Validation Details to view details.

      2. After Flink SQL validation, run a Precheck. The ETL task starts only after it passes the precheck. If the precheck fails, click View Details next to the failed item, fix the issue, and rerun the precheck.

      3. After the precheck passes, click Next: Purchase Instance at the bottom of the page.

      4. On the Purchase page, select an Instance Class and Compute Units (CUs). Read and accept the Data Transmission Service (Pay-As-You-Go) Service Terms and Public Preview Service Terms.

      5. Click Buy and Start to start the ETL task.

        Return to the Streaming ETL page or the page to view the Task Status.