Dataphin:Offline integration

Block

Description

① Offline integration directory

Created offline pipeline tasks are managed in the task directory. The task directory supports the following management operations:

• Directory toolbar: Supports Filter, Upload (upload pipeline development scripts), Refresh, New Folder, and Locate.

• Create integration task: Click the icon to create a single pipeline task or a script pipeline task.

  • Single pipeline task: Use the drag-and-drop visualization to sync single or multiple tables from one or more source data sources to one or more target data sources. For more information, see Create an integration task using a single pipeline.

  • Script pipeline task: Use code in the script to sync single or multiple tables from one or more source data sources to one or more target data sources. For more information, see Create an integration task using the script mode.

• Offline pipeline task: Supports Copy, Acquire Lock, Edit, Clone, Rename, Move, Delete, Unpublish, Unpublish and Delete, and Change Owning Project.

  Important

  In Basic mode, when you submit a delete task, the change is immediately applied to the production environment. Proceed with caution. In Dev-Prod mode, after you submit a delete task, go to the list of items to be published to publish the object to the production environment.

• Offline integration task list: Click the offline integration task list to manage created offline integration tasks in a list view. For more information, see View and manage the offline integration task list.

• Move task: Offline script tasks, offline pipeline tasks, and regular folders can be moved within the Offline integration directory. Offline script tasks and offline pipeline tasks can also be moved to an offline full database migration folder. Regular folders cannot be moved into or out of an offline full database migration folder.

  A regular folder cannot be deleted if it contains child nodes. When an offline full database migration folder is deleted, all tasks it contains, including offline script tasks and offline pipeline tasks, are also deleted.

② Toolbar

The toolbar supports Save, Run, Preview, Scheduling Configuration, Channel Configuration, Quality Monitoring, Submit, Go to Publish, Go to O&M, Refresh, Acquire Lock, and Locate.

• Save: Click the icon on the toolbar to save the offline pipeline you are developing.

• Run: Click the icon on the toolbar to run the current offline pipeline. If your account does not have row-level permissions, click the Request Permission icon to go to the Management Center > Permission Management > My Permissions > Row-level Permissions page to request permissions, and then run the pipeline again.

• Preview: Click the icon on the toolbar to preview the integration results after the current offline pipeline runs.

• Scheduling Configuration: Click the icon on the toolbar to open the Scheduling Configuration tab in the properties panel. Use the scheduling configuration to define scheduling rules for periodically scheduled offline pipeline tasks. For more information, see Offline pipeline scheduling configuration.

• Channel Configuration: Click the icon on the toolbar to open the Channel Configuration tab in the properties panel. Use the offline integration pipeline channel configuration to define fault tolerance, concurrency, Java Virtual Machine (JVM) resources, database configurations, and integration pipeline traffic monitoring for the runtime. For more information, see Offline integration pipeline channel configuration.

• Quality Monitoring: Click the icon on the toolbar to open the quality monitoring panel. You can configure quality monitoring rules for data tables in the integration pipeline. For more information, see Configure integration pipeline quality monitoring.

• Submit: Click the icon on the toolbar to submit the offline integration pipeline to the release center or management center. For more information, see Release Center.

• Go to Publish: Click the icon on the toolbar to publish the latest content that is ready for release. For an offline integration pipeline that has been submitted to the release center, you can go to the release center to publish it.

• Go to O&M: Click the icon on the toolbar to perform O&M on published instances. For an offline integration pipeline that has been submitted to the Operation Center, you can go to the Operation Center to perform O&M operations. For more information, see Operation Center.

• Switch to script mode: Click the icon and select Switch to script mode. This switches the current offline pipeline to an offline script. This is useful in scenarios where you need to automatically get source table columns and generate complete JSON configurations, which can improve the efficiency of script mode configuration. After switching to script mode, you cannot switch back to the offline pipeline mode.

• Refresh: Click the icon on the toolbar to refresh the current offline pipeline task.

• Acquire Lock: Click the icon on the toolbar to acquire the edit lock for the offline pipeline task.

• Locate: Click the icon on the toolbar to locate the current offline integration pipeline task in the task directory.

• Unpublish: Click the icon on the toolbar and select Unpublish to unpublish a published integration task.

• Delete: Click the icon on the toolbar and select Delete to delete the current integration task.

  Important

  In Basic mode, when you submit a delete task, the change is immediately applied to the production environment. Proceed with caution. In Dev-Prod mode, after you submit a delete task, go to the list of items to be published to publish the object to the production environment.

③ Developer canvas

• Visual development: On the developer canvas, drag components from the component library drawer to the canvas for development. For a description of the development components supported by Dataphin, see Integration component library development guide.

• Code script development: For script tasks, use the code editor to develop offline pipeline tasks with code.

④ Configuration and management

The configuration and management panel supports property configuration, quality monitoring, related objects, and history for integration tasks.

• Property configuration: Configure the properties of the offline pipeline task, including basic information, scheduling configuration, scheduling dependencies, and runtime configuration. For more information, see Configure offline pipeline task properties.

• Quality monitoring: By configuring quality monitoring for the offline integration pipeline, Dataphin automatically monitors the quality of your data tables. If a data table has a quality threat, the system automatically sends an alert to the specified recipients. This helps you stay informed about the quality of your data tables. For more information, see Configure integration pipeline quality monitoring.

• Related: The related objects of an offline pipeline refer to the tables that the current offline pipeline task reads from, tables that read from the current task, and tasks that use the current task as an upstream dependency.

• History: View and manage task history versions.

⑤ Console

When you run or preview a task, the structured results are displayed in the console. This includes preview results, runtime information, channel information, step measures, log information, abnormal information, performance diagnostics, and runtime code.