Dataphin:Create a real-time integration task

Parameter

Description

Data Source Configuration

Data Source Type

Select MySQL.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a MySQL data source.

Time Zone

Displays the time zone configured for the selected data source.

Sync Rule Configuration

Sync Policy

Select Real-time Incremental or Real-time Incremental + Full. The default value is Real-time Incremental.

• Real-time Incremental: Collects incremental changes from the source database and writes them to the downstream destination database in the order they occur.

• Real-time Incremental + Full: Imports the full data from the source database at once, and then collects and writes incremental changes to the downstream destination database in the order they occur.

Selection Method

You can select Entire Database, Select Tables, or Exclude Tables.

• Entire Database: Synchronizes all tables in all databases under the selected data source.

• Select Tables/Exclude Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select/Exclude Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches. Deletion is not supported for regular expression matches.

  • Batch Select/Batch Exclude: When you use Batch Select, the selected tables in the current database are synchronized in real time. When you use Batch Exclude, the selected tables are not synchronized.

    You can select all tables in all databases under the selected data source. The tables are displayed in the DBname.Tablename format.

  • Regular Expression Match: Enter a regular expression for table names in the Regular Expression box. Java regular expressions are supported, such as schemaA.*|schemaB.*.

    You can match all tables in all databases under the selected data source in batches. You can use the database name (DBname) and table name (Tablename) for regular expression matching.

Parameter

Description

Data Source Configuration

Data Source Type

Select Microsoft SQL Server.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a Microsoft SQL Server data source.

Time Zone

Displays the time zone configured for the selected data source.

Sync Rule Configuration

Sync Policy

Only Real-time Incremental is supported. Incremental changes from the source database are collected and written to the downstream destination database in real time in the order they occur.

Selection Method

You can select Entire Database, Select Tables, or Exclude Tables.

• Entire Database: Synchronizes the entire current database.

• Select Tables/Exclude Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select/Exclude Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches.

  Batch Select/Batch Exclude: When you use Batch Select, the selected tables in the current database are synchronized in real time. When you use Batch Exclude, the selected tables are not synchronized.

Parameter

Description

Data Source Configuration

Data Source Type

Select PostgreSQL.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a PostgreSQL data source.

Time Zone

Displays the time zone configured for the selected data source.

Sync Rule Configuration

Sync Policy

Only Real-time Incremental is supported. Incremental changes from the source database are collected and written to the downstream destination database in real time in the order they occur.

Selection Method

You can select Entire Database or Select Tables.

• Entire Database: Synchronizes the entire current database.

• Select Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches.

  Batch Select: The selected tables in the current database are synchronized in real time.

Parameter

Description

Data Source Configuration

Data Source Type

Select Oracle.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create an Oracle data source.

Time Zone

Displays the time zone configured for the selected data source.

Sync Rule Configuration

Sync Policy

Only Real-time Incremental is supported. Incremental changes from the source database are collected and written to the downstream destination database in real time in the order they occur.

Selection Method

You can select Entire Database, Select Tables, or Exclude Tables.

• Entire Database: Synchronizes all tables in all databases under the selected data source.

• Select Tables/Exclude Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select/Exclude Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches. Deletion is not supported for regular expression matches.

  • Batch Select/Batch Exclude: When you use Batch Select, the selected tables in the current database are synchronized in real time. When you use Batch Exclude, the selected tables are not synchronized.

  • Regular Expression Match: Enter a regular expression for table names in the Regular Expression box. Java regular expressions are supported, such as schemaA.*|schemaB.*.

Parameter

Description

Data Source Configuration

Data Source Type

Select IBM DB2.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create an IBM DB2 data source.

Sync Rule Configuration

Sync Policy

Only Real-time Incremental is supported. Incremental changes from the source database are collected and written to the downstream destination database in real time in the order they occur.

Selection Method

You can select Entire Database, Select Tables, or Exclude Tables.

• Entire Database: Synchronizes all tables in all databases under the selected data source.

• Select Tables/Exclude Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select/Exclude Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches.

  Batch Select/Batch Exclude: When you use Batch Select, the selected tables in the current database are synchronized in real time. When you use Batch Exclude, the selected tables are not synchronized.

Parameter

Description

Data Source Configuration

Data Source Type

Select Kafka.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a Kafka data source.

Source Topic

Select the topic of the source data. You can enter a keyword of the topic name to perform a fuzzy search.

Data Format

Only Canal JSON is supported. Canal JSON is a format compatible with Canal, and its data storage format is Canal JSON.

Key Type

The key type of Kafka, which determines the key.deserializer configuration when initializing KafkaConsumer. Only STRING is supported.

Value Type

The value type of Kafka, which determines the value.deserializer configuration when initializing KafkaConsumer. Only STRING is supported.

Consumer Group ID (optional)

Enter the ID of the consumer group. The consumer group ID is used to report the status offset.

Sync Rule Configuration

Table List

Enter the names of the tables to be synchronized. Separate multiple table names with line breaks. The value can be up to 1,024 characters in length.

Table names can be in one of the following three formats: tablename, db.tablename, or schema.tablename.

Parameter

Description

Data Source Configuration

Data Source Type

Select Hive.

Datasource

You can only select a Hive data source in Hudi table format. You can also click New to create a data source on the Datasource page. For more information, see Create a Hive data source.

Sync Rule Configuration

Sync Policy

Only Real-time Incremental is supported. Incremental changes from the source database are collected and written to the downstream destination database in real time in the order they occur.

Select Table

Select a single table for real-time synchronization.

Parameter

Description

Data Source Configuration

Data Source Type

Select PolarDB.

Datasource

You can only select a PolarDB data source of the MySQL database type. You can also click New to create a data source on the Datasource page. For more information, see Create a PolarDB data source.

Time Zone

Displays the time zone configured for the selected data source.

Sync Rule Configuration

Sync Policy

Select Real-time Incremental or Real-time Incremental + Full. The default value is Real-time Incremental.

• Real-time Incremental: Collects incremental changes from the source database and writes them to the downstream destination database in the order they occur.

• Real-time Incremental + Full: Imports the full data from the source database at once, and then collects and writes incremental changes to the downstream destination database in the order they occur.

Selection Method

You can select Entire Database, Select Tables, or Exclude Tables.

• Entire Database: Synchronizes all tables in all databases under the selected data source.

• Select Tables/Exclude Tables: Selects some tables in the current database for real-time synchronization. After you select the tables, you can click Preview to view all matched tables in the Select/Exclude Table Preview dialog box. In the dialog box, you can search for tables by keyword and delete tables individually or in batches. Deletion is not supported for regular expression matches.

  • Batch Select/Batch Exclude: When you use Batch Select, the selected tables in the current database are synchronized in real time. When you use Batch Exclude, the selected tables are not synchronized.

  • Regular Expression Match: Enter a regular expression for table names in the Regular Expression box. Java regular expressions are supported, such as schemaA.*|schemaB.*.

Parameter

Description

Data Source Configuration

Data Source Type

Select MaxCompute.

Datasource

Select a destination data source. You can select a MaxCompute data source and project. You can also click New to create a data source on the data source page. For more information, see Create a MaxCompute data source.

New Sink Table Configuration

New Table Type

You can select Standard Table or Delta Table. The default value is Standard Table.

If you select Delta Table and set the sink table creation method to Auto Create Table, a MaxCompute Delta table is created. Additional fields are not used when creating a Delta table.

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Partition Format

If you set New Table Type to Standard Table, only Multiple Partitions is supported. If you set New Table Type to Delta Table, you can select No Partition or Multiple Partitions.

Partition Interval

If you set Partition Format to No Partition, you cannot configure the partition interval. If you set Partition Format to Multiple Partitions, you can set Partition Interval to Hour or Day.

Parameter

Description

Data Source Configuration

Data Source Type

Select MySQL.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a MySQL data source.

Time Zone

Displays the time zone configured for the selected data source.

New Sink Table Configuration

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Parameter

Description

Data Source Configuration

Data Source Type

Select Microsoft SQL Server.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a Microsoft SQL Server data source.

Time Zone

Displays the time zone configured for the selected data source.

New Sink Table Configuration

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Parameter

Description

Data Source Configuration

Data Source Type

Select Oracle.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create an Oracle data source.

Time Zone

Displays the time zone configured for the selected data source.

New Sink Table Configuration

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Parameter

Description

Data Source Configuration

Data Source Type

Select Kafka.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a Kafka data source.

Destination Topic

The topic for the destination data. You can select Single Topic or Multiple Topics. If you select Single Topic, you must select a destination topic. You can enter a keyword of the topic name to search. If you select Multiple Topics, you can configure topic name transformation and topic parameters.

• Single Topic: All table messages are written to the same topic.

• Multiple Topics: A topic with the same name is created for each table.

Data Format

Set the storage format for the written data. Supported formats include DTS Avro and Canal Json.

• DTS Avro: A data serialization format that converts data structures or objects into a format that is easy to store or transmit.

• Canal Json: A format compatible with Canal. The data storage format is Canal Json.

Destination Topic Configuration

Topic Name Transform

Click Configure Topic Name Transform. In the Configure Topic Name Transformation Rules dialog box, you can configure Topic Name Transformation Rules and a prefix and suffix for the topic name.

• Topic Name Transformation Rules: Click New Rule to add a rule. You must enter the Source Table String To Replace and the Destination Topic Replacement String. Neither can be empty. The Destination Topic Replacement String can contain only letters, digits, and underscores (_) and be up to 32 characters in length.

• Prefix and suffix for the topic name: You can enter letters, digits, and underscores (_). The length cannot exceed 32 characters.

Topic Parameters

Additional parameters for creating a topic. The format is key=value. Separate multiple parameters with line breaks.

Parameter

Description

Destination Data

Data Source Type

Select DataHub.

Datasource

Select a destination data source.

The system provides a shortcut to create a new data source. You can click New to create a DataHub data source on the data source page. For more information, see Create a DataHub data source.

Destination Topic Creation Method

You can select New Topic or Use Existing Topic.

• New Topic: Manually enter the destination topic to create it.

• Use Existing Topic: Use an existing topic in the destination database. Ensure that the schema of the topic is consistent with the format of the synchronization message. Otherwise, the sync task will fail.

Destination Topic

• Set Target Topic Creation Method to Create Topic.

  You must manually enter the Destination Topic. The Destination Topic name must start with a lowercase letter and can contain 3 to 64 digits, letters, and underscores (_).

  After you enter the name, you can click Validate to check if the topic already exists in the destination database.

  • If the topic does not exist in the destination database, it is automatically created. Its schema is the schema of the synchronization message, and its default lifecycle is 7 days.

  • If the topic already exists in the destination database, ensure that its schema is consistent with the schema of the synchronization message. Otherwise, the task will fail.

• Target Topic Creation Method is Use Existing Topic.

  Click the drop-down list to select an existing topic in the destination database. If there are many topics, you can enter a topic name to search for the one you need.

Parameter

Description

Data Source Configuration

Data Source Type

Select Databricks.

Datasource

Select a destination data source. You can select a Databricks data source and project. You can also click New to create a data source on the data source page. For more information, see Create a Databricks data source.

Time Zone

Time-formatted data is processed based on the current time zone. The default value is the time zone configured in the selected data source and cannot be changed.

New Sink Table Configuration

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Partition Format

You can select No Partition or Multiple Partitions.

Partition Interval

If you set Partition Format to No Partition, you cannot configure the partition interval. If you set Partition Format to Multiple Partitions, you can set Partition Interval to Hour or Day.

Parameter

Description

Data Source Configuration

Data Source Type

Select SelectDB.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a SelectDB data source.

New Sink Table Configuration

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Parameter

Description

Data Source Configuration

Data Source Type

Set Data Source Type to Hive.

Datasource

Select a data source. You can also click New to create a data source on the Datasource page. For more information, see Create a Hive data source.

New Sink Table Configuration

Data Lake Table Format

You can select None, Hudi, Iceberg, or Paimon.

• None: Data is written and tables are created as standard Hive tables.

• Hudi: Data is written and tables are created in Hudi format. You can select Hudi only when the Hive data source version is CDP7.x Hive 3.1.3.

• Iceberg: Data is written and tables are created in Iceberg format. You can select Iceberg only when the Hive data source version is EMR5.x Hive 3.1.x.

• Paimon: Data is written and tables are created in Paimon format. You can select Paimon only when the Hive data source version is EMR5.x Hive 3.1.x.

Hudi Table Type/Paimon Table Type

For Hudi Table Type, you can select MOR (merge on read) or COW (copy on write).

For Paimon Table Type, you can select MOR (merge on read), COW (copy on write), or MOW (merge on write).

Table Creation Execution Engine

You can select Hive or Spark. After you select a data lake table format, Spark is selected by default.

• Hive: Uses the Hive engine to create tables. The table creation syntax is Hive syntax.

• Spark: Uses the Spark engine to create tables. The table creation syntax is Spark syntax. You can select Spark only when Spark is enabled for the Hive data source.

  Note

  When Data Lake Table Format is set to Paimon, only the Spark table creation execution engine is supported.

Table Name Transform

Destination table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

Click Configure Table Name Transform to open the Configure Table Name Transformation Rules dialog box.

• Replace String: Click New Rule to add a rule. Configure the Source Table String To Replace and Destination Table Replacement String. You can add up to 5 rules.

• Table Name Prefix/Suffix: This cannot be empty. It can contain only letters, digits, and underscores (_). The length cannot exceed 32 characters.

Partition Format

You can select Single Partition, Multiple Partitions, or Fixed Partition.

Partition Interval

The default value is Hour. You can also select Day. Click the icon next to the partition interval to view the partition setting details.

• Single Partition:

  • Hour: Displays a hash partition named ds (yyyyMMddhh).

  • Day: Displays a hash partition named ds (yyMMdd).

• Multiple Partitions:

  • Hour: Displays four levels of partitions: yyyy, mm, dd, and hh.

  • Day: Displays three levels of partitions: yyyy, mm, and dd.

Partition Value

Enter a fixed partition value, for example, 20250101.

Area

Description

① View Additional Fields

During real-time incremental synchronization, additional fields are automatically added by default when a table is created to facilitate data use. Click View Additional Fields to view the fields. In the Additional Fields dialog box, you can view information about the currently added fields.

Click View DDL For Adding Fields to view the DDL statement for adding the additional fields.

② Search and Filter Area

You can search by Source Table and Sink Table Name. To quickly filter sink tables, click the icon at the top. You can filter by Mapping Status and Creation Method.

③ Add Global Fields, Refresh Mappings

• Add Global Fields

  Click Add Global Fields to add global fields in the Add Global Fields dialog box.

  • Name: The name of the global field.

  • Type: Supported data types are String, Long, Double, Date, and Boolean.

  • Value: The value of the global field.

  • Description: A description of the field.

  Note

  • If a field is added both globally and to a single table, only the field added to the single table takes effect.

  • Currently, only constants can be added.

  • Global fields take effect only for sink tables created using the Auto Create Table method.

  • Adding global fields is not supported when the source data source type is Kafka.

• Refresh Mappings

  To refresh the sink table configuration list, click Refresh Mappings.

  Important

  • If the sink table configuration already has content, reselecting the data source type and data source will reset the sink table list and mapping status. Proceed with caution.

  • You can click to refresh again at any time during the refresh process. Each time you click Refresh Mappings, only the configured global fields are saved. Other information, including the sink table creation method, sink table name, and deletion records, is not saved.

  • When the source data source type is Kafka, clicking Refresh Mappings will map the tables according to the table list in the Sync Rule Configuration. An error is reported if a table does not exist.

④ Destination Database List

The destination database list includes Serial Number, Source Table, Mapping Status, Sink Table Creation Method, and Sink Table Name. You can also Add Field, View Fields, Refresh, or Delete a sink table.

• Mapping Status:

  • Completed: The mapping is completed normally.

  • Incomplete: The mapping has not been refreshed after a status change.

  • Mapping: Waiting for mapping or in the process of mapping.

  • Abnormal: A data source or internal system error exists.

  • Failed: The destination partitioned table is inconsistent with the partition set for the real-time task.

  • Alerting: The source and sink tables may have incompatible data types.

• Sink Table Creation Method has three options:

  • If a table with the same name as the source table exists in the destination database, the creation method is Use Existing Table. This table is used as the sink table by default. To change to Auto Create Table, you must add a table name transformation rule or a prefix/suffix and then remap.

  • If no table with the same name is found in the destination database, the creation method defaults to Auto Create Table. You can also change the method to Use Existing Table and select an existing table for synchronization.

  • You can add fields or use custom DDL to create tables only for tables that are automatically created. Global fields also take effect only for automatically created tables.

  Note

  • When the destination data source type is Hive:

    • If you use Auto Create Table and the data lake table format is None, a standard Hive table is created. Otherwise, a table of the selected format is created. Hudi and Iceberg are currently supported.

    • If you use Custom Table Creation and the data lake table format is None, the DDL for a standard Hive table is used. Otherwise, you must use the DDL for the selected table format. Hudi and Iceberg are currently supported.

  • When the source data source type is Kafka, the only supported sink table creation method is Use Existing Table.

  • When the destination data source type is SelectDB, if the source table has no primary key during automatic table creation, a Duplicate table is created. If the source table has a primary key, a Unique table is created.

  • When the partition format is Single Partition or Fixed Partition and the sink table creation method is Use Existing Table, the system automatically checks if the sink table partition meets the partition settings. An error is reported if it does not.

• Sink Table Name: Sink table names can contain only letters, digits, and underscores (_). If a source table name contains other characters, you must configure table name transformation rules.

  When the destination data source type is MaxCompute, the sink table creation method is Auto Create Table, and the new table type is Delta Table, the icon is displayed after the sink table name, indicating that a new Delta table will be created. When the sink table creation method is Use Existing Table, if the user selects a Delta table in the sink table list, the icon is also displayed after the sink table name, indicating that the table is a Delta table.

• Actions:

  • Custom Table Creation: You can create a table using Add Field or DDL. After you enable custom table creation, global fields no longer take effect.

    Note

    • After a field is added, it is displayed only in the actions column for Auto Create Table.

    • You cannot modify an existing sink table, which is a sink table created using the Use Existing Table method.

  • View Fields: View the fields and types of the source and sink tables.

  • Refresh: Remap the source and sink tables.

  • Delete: Deleting a source table cannot be undone.

⑤ Batch Operations

You can perform batch Delete operations on sink tables.

Area

Description

① Search and Filter Area

You can search by Source Table and Destination Topic Name. To quickly filter sink tables, click the icon at the top. You can filter by Mapping Status and Destination Topic Creation Method.

② Refresh Mappings

To refresh the sink table configuration list, click Refresh Mappings.

③ List

The list includes Serial Number, Source Table, Mapping Status, Destination Topic Creation Method, and Destination Topic Name. You can also delete a sink table.

• Destination Topic Creation Method: If the destination topic already exists, the creation method is Use Existing Topic. If the destination topic does not exist, the creation method is Auto Create Topic.

  When you use Auto Create Topic, the system creates the topic based on the generated destination topic name and topic parameters.

• Mapping Status: Only checks if the destination topic exists.

• Delete: Deletes the corresponding row. This operation cannot be undone.

④ Batch Operations

You can perform batch Delete operations on sink tables.