DataWorks:Configure a real-time sync task in Data Studio

Configure a real-time sync task for a single table

1. Configure the source data source.

   The following source data source types and their configurations are supported for single-table data synchronization:

   • Configure MySQL as a source

   • Configure DataHub as a source

   • Configure LogHub (SLS) as a source

   • Configure Kafka as a source

   • Configure PolarDB as a source

   1. In the Input section on the left side of the real-time sync task configuration page, drag the target source data source component to the panel on the right.

      

   2. Click the source component and configure its parameters in the Node Configuration panel on the right.

2. Optional: Configure a data transform method.

   If you want to transform the source data into the required output format during real-time synchronization, you can configure a data transform method.

   The following transform methods are supported for single-table data synchronization:

   • Configure a data filtering transform: You can filter data based on rules, such as the size of a field. Only data that meets the rules is retained.

   • Configure string replacement: You can replace string fields.

   • Configure data masking: You can mask single-table data that is synchronized in real time and then store it in a specified database location.

   1. In the Conversion section on the left side of the configuration page, drag the required data transform component to the panel on the right. Hover over the source component to display its connection points. Connect the lower point of the source component to the upper point of the transform component. After they are connected, you can configure the transform component in the Node Configuration panel.

      

   2. Click the transform component and configure its parameters in the Node Configuration panel on the right.

3. Configure the destination data source.

   The following destination data source types and their configurations are supported for single-table data synchronization:

   • Configure MaxCompute as a destination

   • Configure Hologres as a destination

   • Configure DataHub as a destination

   • Configure Kafka as a destination

   • Configure Elasticsearch as a destination

   1. In the Output section on the left side of the configuration page, drag the target destination data source component to the panel on the right and connect it to the upstream component. Configure the destination data source, table, and field mapping. If the destination table does not exist, you can click Create Table to quickly create it.

      

   2. Click the destination component and configure its parameters in the Node Configuration panel on the right.

4. In the toolbar above the canvas, click Save to complete the task configuration.

Configure a real-time sync task for an entire database

1. Set the source and synchronization rules.

   1. In the Data Source section, select the Type and Data Source name of the data source to synchronize.

   2. Select the tables to synchronize.

      In the Select Source Table for Synchronization section, all tables in the selected data source are displayed. In the Source Databases And Tables section, select all or some of the tables from the database that you want to synchronize. Then, click the icon to move them to the Selected Tables list.

      Important

      If a selected table does not have a primary key, it cannot be synchronized in real time.

   3. Set the mapping rule for table names.

      In this step, you can select the databases and tables from the source data source to synchronize. By default, the synchronization solution writes the source database and tables to a schema or table with the same name in the destination. If the schema or table does not exist in the destination, it is automatically created. You can also define the names of the destination schema or tables using Set Mapping Rules for Table/Database Names. This lets you write data from multiple tables into a single table, or to update a fixed prefix for source database or table names to another prefix when writing to the destination.

      • Source table name and destination Table Name conversion rules: You can use a regular expression to convert the source table name to the final destination table name.

        • Example 1: Write data from source tables with the `doc_` prefix to destination tables with the `pre_` prefix.

        • Example 2: Write data from multiple tables to a single destination table.

          To synchronize source tables named "table_01", "table_02", and "table_03" to a single destination table named "my_table", configure the regular expression conversion rule for table names as follows: Set Source to `table.*` and Destination to `my_table`.

      • Rule for Destination Table Table Name: You can use a combination of built-in variables to generate the destination table name. You can also add a prefix and a suffix to the converted destination table name. The available built-in variables are:

        • ${db_table_name_src_transed}: The table name after the conversion in the "Source and destination table name conversion rule".

        • ${db_name_src_transed}: The destination schema name after the conversion in the "Source database and destination schema name conversion rule".

        • ${ds_name_src}: The source data source name.

        Example: Further process the table name that was converted by the source and destination table name conversion rule in the previous step. Use `${db_table_name_src_transed}` to represent the result of the previous step, which is "my_table". Then, add a prefix and a suffix to this built-in variable, such as `pre_${db_table_name_src_transed}_post`. The final destination table name is "pre_my_table_post".

      • Source Database And Destination Schema Name Conversion Rule: You can use a regular expression to convert the source schema name to the final destination schema name.

        Example: Replace the `doc_` prefix of the source database name with the `pre_` prefix.

2. Select the destination data source and configure the destination table or topic.

   1. On the Set Destination Table Or Topic page, configure basic information for the Destination Data Source. This includes parameters such as the write mode and partition settings. The specific configuration varies based on the real-time synchronization interface of each data source.

   2. Click Refresh Source table and {dataSourceLabel} table mapping to create a mapping between the source and destination tables to synchronize.

      You can customize the destination schema and table names and add constants and variables to the destination table by clicking Edit additional fields. The specific configuration varies based on the real-time synchronization interface of each data source.

      Note

      If you synchronize many tables, the process may be slow. Please wait.

3. Optional: Set table-level synchronization rules.

   Some synchronization solutions support custom table-level DML processing policies. This means you can define corresponding processing policies for insert, update, or delete operations that occur in the source table.

   Note

   The supported DML operations may vary between data sources. Whether a synchronization solution supports DML processing policies depends on the product interface. For information about the DML operations supported by data sources, see Supported DML and DDL operations.

4. Set DDL message processing rules.

   The source data source may involve many DDL operations. During real-time synchronization, you can set processing policies for different DDL messages sent to the destination based on your business needs. The supported DDL operations may vary between data sources. For more information, see Supported DML and DDL operations. On the Data Integration > Configuration Options > Processing Policy for DDL Messages in Real-time Sync page, you can set DDL processing policies for each destination database type. The following table describes the DDL message processing policies.

   DDL Message Type	Processing Policy
   CreateTable	When DataWorks receives a DDL message of the corresponding type, the processing policies are as follows: Normal Processing: Forwards the message to the destination data source for processing. Because different destination data sources may have different policies for handling DDL messages, DataWorks only forwards the message. Ignore: Discards the message and does not send it to the destination data source. Alert: Discards the message and records an alert in the real-time synchronization log, indicating that the message was discarded due to an execution error. Error: The real-time sync task immediately enters an error state and stops running.
   DropTable
   AddColumn
   DropColumn
   RenameTable
   RenameColumn
   ChangeColumn
   TruncateTable

5. Set runtime resources.

   • The task concurrency control feature limits the maximum number of concurrent connections for reading from and writing to the database.

   • You can control whether the sync task tolerates dirty data.

     • If dirty data is not allowed, the task fails and exits if dirty data is generated during execution.

     • If dirty data is allowed, the task ignores the dirty data (it is not written to the destination) and continues to run normally.

6. Click Complete.

You can migrate a real-time integration task for a single table that is configured in DataStudio to the Data Integration page by clicking Migrate to Data Integration.

1. Double-click the real-time integration task for a single table that you want to migrate. On the task configuration page, click Migrate to Data Integration to migrate the task.

   

2. In the upper-left corner, click the icon and choose All Products > Data Integration. On the Synchronization Task page, you can view the successfully migrated real-time integration task for a single table in the task list.