Dataphin:Create an integration task using a single pipeline

Prerequisites

Configure the source and target data sources before you start. For more information about supported data sources, see Supported data sources.

Procedure

Step 1: Create a single offline pipeline

1. On the Dataphin home page, select Development > Data Integration from the top menu bar.

2. On the Data Integration page, select a Project. In Dev-Prod mode, also select an Environment.

3. In the Batch Integration list, click the icon and select Batch Pipeline. The Create Offline Pipeline dialog box appears.

4. In the Create Offline Pipeline dialog box, configure the pipeline parameters.

   Parameter	Description
   Pipeline Name	Enter a pipeline name, up to 64 characters. Unsupported characters: | / \ : ? <> * '.
   Schedule Type	Select a Schedule Type: Recurring Task Node: Runs on a regular schedule. Manual Node: No dependencies; triggered manually.
   Description	Enter a description, up to 1,000 characters.
   Select Directory	Select the folder where the task is stored. If no folder exists, create one: Above the compute task list on the left, click the icon. The New Folder dialog box opens. In the New Folder dialog box, enter a Name and select a Directory. Click OK.

5. Click OK.

Step 2: Develop the offline pipeline script

Develop the pipeline using visual components. Select a component type from the Component Library in the sidebar and drag it to the canvas.

• Component types: The component library provides five types: Input, Transform, Flow, Output, and Custom.Development of the integration component library.

• Components: Each component provides a specific function for pipeline development.Component configuration details.

Step 3: Configure pipeline scheduling

1. In the canvas menu bar, click Scheduling Configuration.

2. Configure the following settings on the scheduling configuration page:

   • Basic information: Set the developer, owner, and description.Configure basic information for an offline integration pipeline.

   • Scheduling configuration: For a recurring task node, define the scheduling type, cycle, logic, and execution mode.Scheduling configuration for an offline integration pipeline.

   • Scheduling dependencies: For a recurring task node, define upstream node dependencies. Dataphin runs nodes in order based on these dependencies.Scheduling dependency configuration for an offline integration pipeline.

   • Runtime configuration: Set a timeout period and retry policy for failed tasks.Runtime configuration for an offline integration pipeline.

   • Resource configuration: Assign a resource group for task scheduling.Configure resources for an offline integration pipeline task.

Step 4: Save and submit the offline integration task

1. At the top of the canvas, click the icon to save the pipeline task.

2. At the top of the canvas, click the icon. In the Commit Remarks dialog box, enter your remarks and click OK And Submit.

   Dataphin parses the data lineage and runs a validation check on submission.Instructions on how to submit an integration task.

What to do next

• In Dev-Prod mode, publish the task.Manage publish tasks.

• In Basic mode, the task is scheduled in production after submission. For more information, see Operation Center.