DataWorks:Real-time synchronization node

Configure the input data source.

1. In the Input section on the left side of the node configuration page, drag the data source component to the canvas.

   

2. Click the input component to configure its parameters in the Node Configuration dialog box on the right.

   The following input data source types are supported for single-table synchronization. For more information about the configuration of each data source type, see the linked topics.

   • Configure MySQL input

   • Configure DataHub input

   • Configure LogHub (SLS) input

   • Configure Kafka input

   • Configure PolarDB input

(Optional) Configure data transformation.

1. In the Conversion section on the left, drag a transform component onto the canvas and connect it to the upstream component.

   

2. Click the transform component, and then configure its parameters in the Node Configuration dialog box on the right.

   The following transformation methods are supported for single-table synchronization. For more information about the configuration of each method, see the linked topics.

   • Data filtering: Filters data based on rules, such as field size. Only data that meets the specified rules is retained.

   • String replacement: Replaces the values of string fields.

   • Data masking: Masks data from a single table during real-time synchronization and stores the masked data in a specified database location.

   Note

   You can use multiple transform components to filter and process data.

Configure the output data source.

1. In the Output section on the left, drag the target output data source component onto the canvas. Connect the component to its upstream component. Configure the destination data source, table, and field mappings. If the destination table does not exist, click Create Table to create it.

   

2. Click the output component to configure its parameters in the Node Configuration dialog box on the right.

   The following output data source types are supported for single-table synchronization. For more information about the configuration of each data source type, see the linked topics.

   • Configure MaxCompute output

   • Configure Hologres output

   • Configure AnalyticDB for MySQL 3.0 output

   • Configure DataHub output

   • Configure Kafka output

   • Configure Elasticsearch output

In the toolbar above the canvas, click Save to save the task configuration.

Configure the synchronization source and rules.

1. In the Data Source section, select the Type and Data Source.

2. In the Select Source Table for Synchronization section, select one or more tables from the Source Tables area and click the icon to move them to the Selected Tables area.

3. Set the table name mapping rules.

   Note

   In this step, you can select the databases and tables from the source to sync. By default, data from a source database or table is written to a destination schema or table with the same name. If a destination schema or table does not exist, the system creates it automatically. You can also use Set Mapping Rules for Table/Database Names to customize the names of the destination schemas or tables.

   • Source table name and destination Table Name conversion rules: You can use a regular expression to convert source table names to destination table names.

     • Example 1: This example synchronizes data from source tables with the prefix doc_ to destination tables with the prefix pre_.

     • Example 2: Write data from multiple tables to a single destination table.

       To synchronize data from the source tables table_01, table_02, and table_03 to the single table my_table, configure the regular expression conversion rule by setting Source to table_* and Destination to my_table.

   • Rule for Destination Table Table Name: You can use built-in variables to generate destination table names. You can also add a prefix and a suffix to the generated names. The available built-in variables are:

     • ${db_table_name_src_transed}: The name of the table after it is converted based on the source and destination table name conversion rules.

     • ${db_name_src_transed}: The destination schema name generated by the conversion rules for the source database and destination schema name.

     • ${ds_name_src}: The name of the source data source.

     For example, if you used a table name conversion rule in the previous step to rename a source table to my_table, you can then perform further string concatenation on the new table name. To add a prefix and a suffix, use the ${db_table_name_src_transed} variable, which represents the result from the previous step. For example, the expression pre_${db_table_name_src_transed}_post maps the source table to a destination table named pre_my_table_post.

4. Click Next to continue to the Set Destination Table page.

Set the destination table.

1. On the Set Destination Table or Topic page, you can configure basic information for the Destination Data Source, such as the write mode and partition settings. The available settings depend on the real-time synchronization interface of each data source.

2. Click Refresh source and destination table mapping to create mappings between the source and destination tables.

   • You can use Edit Additional Fields or Batch Edit Additional Fields In Destination Table to add constants, variables, and other operations to the destination table. The configuration depends on the real-time synchronization interface for each data source.

   • If the destination table does not have a primary key, migration is not supported. As an alternative, you can set a Synchronized Primary Key.

   Note

   If you are synchronizing many tables, the process may be slow. Please wait for the process to complete.

3. Click Next to proceed to the Configure Table-level Synchronization Rule page.

   Note

   If the system detects that some of your tables need to be automatically created, you must wait for the tables to be created before you can proceed to the next step.

(Optional) Set table-level synchronization rules.

1. Some synchronization solutions support custom table-level DML processing policies. This lets you define how to handle insert, update, and delete operations that occur on the source table.

   Note

   The supported DML operations may vary based on the data source. Whether a specific synchronization solution supports DML processing policies depends on the product interface. For more information about the DML operations that are supported by different data sources, see Supported DML and DDL operations.

2. Click Next to proceed to the Set Processing Policy For DDL Messages page.

Set DDL message processing rules.

On the Set Processing Policy for DDL Messages page, you can configure how to process DDL operations that occur in the source data source. During real-time synchronization, you can set processing policies for different DDL messages that are synchronized to the destination as needed. The supported DDL operations vary based on the data source. For more information, see Supported DML and DDL operations. You can set a DDL processing policy for each destination database type. The following table describes the DDL message processing policies.

Click Next to navigate to the Configure Resource page.

Set runtime resources.

• Advanced configuration > Maximum Number of Parallel Threads Allowed for Destination: Specifies the maximum number of concurrent threads for a task to write to the destination database.

• Advanced Settings > Allow Dirty Data Records: Specifies whether the sync task tolerates dirty data.

  • If you do not allow dirty data, the task fails and stops if dirty data is generated during execution.

  • If you allow dirty data, the task ignores the dirty data and continues to run. The dirty data is not written to the destination.

Complete the configuration.

Click Complete to finish the configuration of the real-time sync task for the entire database.