Dataphin:Create an Express-Sync pipeline task

Prerequisites

Before you create an Express-Sync pipeline task, configure the source and destination data sources to be synchronized. This allows you to select them when configuring the input and output components. Express-Sync pipeline tasks support the following data sources:

• Create a MySQL data source

• Create an Oracle data source

• Create a PostgreSQL data source

• Create a Microsoft SQL Server data source

• Create a MaxCompute data source

• Create a Hive data source

• Create an HDFS data source

• Create a Doris data source

• Create a SelectDB data source

Procedure

Step 1: Create an Express-Sync pipeline

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Integration.

2. In the top navigation bar of the Data Integration page, select a Project. If you are using Dev-Prod mode, you must also select an environment.

3. In the offline integration list, click the New icon and select Express-Sync. The Create Express-Sync pipeline dialog box opens.

4. In the Create Express-Sync pipeline dialog box, configure the parameters.

   Parameter	Description
   Pipeline name	Enter a name for the pipeline. The name can be up to 64 characters long and cannot contain the following special characters: vertical bars (|), forward slashes (/), backslashes (\), colons (:), question marks (?), less-than (<) and greater-than (>) signs, asterisks (*), or single quotation marks (').
   Scheduling type	Select a scheduling type for the pipeline. The available options are: Periodic node: A task that runs at regular intervals. Manual node: A task that has no dependencies and runs only when manually triggered.
   Description	Enter a brief description for the Express-Sync pipeline. The description can contain up to 1,000 characters.
   Select directory	Select a directory to store the task. If the desired directory does not exist, you can create it. Click New Folder to open the New Folder dialog box. In the New Folder dialog box, enter a Name for the folder and, as needed, select a Directory location. Click OK.

5. Click OK to create the Express-Sync pipeline.

Step 2: Develop the pipeline script

The component library for Express-Sync pipeline tasks supports only input and output components. The configuration process is the same as for offline integration tasks. You can also switch to script mode.

The input and output components for Express-Sync pipeline tasks are categorized by their level of Arrow support:

• Apache Arrow acceleration: The data source provides a native columnar read/write path, allowing data to be transmitted in Arrow format without row-to-column conversion.

• Apache Arrow adaptation: The data source does not provide a native columnar path. Data must be converted between the row-based format and the Arrow columnar format during read and write operations.

The following input components are supported:

The following output components are supported:

Step 3: Configure pipeline scheduling

On the menu bar of the development canvas, click Scheduling Configuration to configure the basic information, scheduling configuration, scheduling dependency, scheduling parameters, runtime configuration, and resource configuration for the Express-Sync pipeline task. For more information, see the following topics:

• Configure basic information for an offline integration pipeline

• Configure scheduling configuration for an offline pipeline

• Configure scheduling dependency for an offline pipeline

• Runtime configuration for an offline pipeline task

• Configure resources for an offline integration pipeline task

Step 4: Save and commit the task

1. Click the Save icon above the canvas to save the Express-Sync pipeline task.

2. Click the Submit icon above the canvas. In the Submit Notes dialog box, enter notes and click OK and Submit.

Next steps

• If you are using Dev-Prod mode, you must publish the task so it can be scheduled in the production environment.

• If you are using Basic mode, the task becomes available for scheduling in the production environment after you commit it. You can view the committed task in the O&M Center.