Dataphin:Create an express synchronization pipeline

Prerequisites

Configure the required source and destination data sources before you begin. This allows you to select them when configuring the input and output components. Express synchronization pipeline tasks support the following data sources:

• Create a MySQL data source

• Create an Oracle data source

• Create a PostgreSQL data source

• Create a Microsoft SQL Server data source

• Create a MaxCompute data source

• Create a Hive data source

• Create an HDFS data source

• Create a Doris data source

• Create a SelectDB data source

• Create a Hologres data source

• Create a ClickHouse data source

Procedure

Step 1: Create an express synchronization pipeline

1. In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.

2. In the top navigation bar of the integration page, select a project. In Dev-Prod mode, you must also select an environment.

3. In the offline integration list, click the New icon, select Express Synchronization, and open the Create Express Synchronization Pipeline dialog box.

4. In the Create Express Synchronization Pipeline dialog box, configure the pipeline parameters.

   Parameter	Description
   Pipeline name	Enter a name for the pipeline. The name can be up to 64 characters long and cannot contain the following special characters: vertical bar (|), forward slash (/), backslash (\), colon (:), question mark (?), angle brackets (<>), asterisk (*), or single quotation mark (').
   Scheduling type	Select a scheduling type for the pipeline: Periodic node: A task that runs at regular intervals. Manual node: A task that has no dependencies and must be triggered manually.
   Description	Enter a description for the express synchronization pipeline, up to 1,000 characters.
   Select directory	Select the directory to store the task. If you have not created a directory, you can create one: Above the compute task list on the left, click the new folder icon to open the New Folder dialog box. In the New Folder dialog box, enter a folder Name and select its location under Select Directory. Click OK.

5. Click OK to create the express synchronization pipeline.

Step 2: Develop the pipeline script

The component library for express synchronization pipeline tasks supports only input and output components. The configuration for these components is the same as for those in offline integration tasks. You can also switch to script mode.

The input and output components for express synchronization pipeline tasks are divided into two categories based on whether the data source natively supports Arrow:

• Apache Arrow acceleration: The data source provides a native columnar read/write path, allowing data to be transmitted in Arrow format without row-to-column conversion.

• Apache Arrow adaptation: The data source does not provide a native columnar path, requiring conversion between the row format and the Arrow columnar format during read and write operations.

The following input components are supported:

The following output components are supported:

Step 3: Configure pipeline scheduling

Click Scheduling Configuration on the toolbar to configure the basic information, scheduling configuration, scheduling dependencies, scheduling parameters, runtime configuration, and resource configuration for the express synchronization pipeline task. For more information, see:

• Configure basic information for an offline integration pipeline

• Configure scheduling for an offline pipeline

• Configure scheduling dependencies for an offline pipeline

• Runtime configuration for offline pipeline tasks

• Configure resources for an offline integration pipeline task

Step 4: Save and submit the task

1. Click the save icon above the canvas to save the express synchronization pipeline task.

2. Click the submit icon above the canvas. In the Submission Comments dialog box, enter your comments and click OK and Submit.

Next steps

• If your development mode is Dev-Prod mode, you must publish a task before it can be scheduled in the production environment.

• If your development mode is Basic mode, a successfully submitted task becomes eligible for scheduling in the production environment, and you can then go to the O&M Center to view the task.