Dataphin:Offline integration

Section

Description

①Offline integration directory

Created offline pipeline tasks are managed in the task directory. The task directory supports the following management operations:

• Directory toolbar: Supports Filter, Upload (upload pipeline development scripts), Refresh, New Folder, and Locate.

• Create integration task: Click the button to create a single pipeline task or a script pipeline task.

  • Single pipeline task: Use the drag-and-drop method to sync one or more non-partitioned or partitioned tables from one or more source data sources to one or more target data sources. For more information, see Create an integration task using a single pipeline.

  • Script pipeline task: Use the script code method to sync one or more non-partitioned or partitioned tables from one or more source data sources to one or more target data sources. For more information, see Create an integration task using the script mode.

• Offline pipeline task: Supports Copy, Get Lock, Edit, Clone, Rename, Move, Delete, Unpublish, and Unpublish and Delete.

  Important

  In Basic mode, after you submit a delete request for a task, the task is updated to the production environment. Perform this operation with caution. In Dev-Prod mode, after you submit a delete request for a task, go to the pending release list to publish the object to the production environment.

• Offline Integration Task List: Click an offline integration task list to manage your created offline integration tasks in a list. For more information, see View and manage the offline integration task list.

• Move tasks: You can move offline script tasks, offline pipeline tasks, and regular folders within the offline integration directory. Offline script tasks and offline pipeline tasks can also be moved to the offline full database migration folder. Regular folders cannot be moved into or out of the offline full database migration folder.

  A regular folder cannot be deleted if it contains child nodes. When the offline full database migration folder is deleted, all tasks contained in the folder are also deleted, including offline script tasks and offline pipeline tasks.

②Toolbar

The toolbar supports Save, Run, Preview, Scheduling Configuration, Channel Configuration, Quality Monitoring, Submit, Go to Publish, Go to O&M, Refresh, Get Lock, and Locate.

• Save: Click the icon in the toolbar to save the offline pipeline that is in development.

• Run: Click the icon in the toolbar to run the current offline pipeline. If the current account does not have row-level permissions, click the Request Permission icon to go to the Management Center > Permission Management > My Permissions > Row-level Permissions page. Request the permissions and then run the node again.

• Preview: Click the icon in the toolbar to preview the integration results after the current offline pipeline runs.

• Scheduling Configuration: Click the icon in the toolbar to open the Scheduling Configuration tab on the Properties panel. Use the scheduling configuration to define scheduling rules for periodic offline pipeline tasks. For more information, see Offline pipeline scheduling configuration.

• Channel Configuration: Click the icon in the toolbar to open the Channel Configuration tab on the Properties panel. Use the offline integration pipeline channel configuration to define fault tolerance, concurrency, JVM resources, database configurations, and traffic monitoring for the integration pipeline. For more information, see Offline integration pipeline channel configuration.

• Quality monitoring: Click the icon in the toolbar to open the quality monitoring panel. You can configure quality monitoring rules for data tables in the integration pipeline. For more information, see Configure integration pipeline quality monitoring.

• Submit: Click the icon in the toolbar to submit the offline integration pipeline to the Publishing Center or Management Center. For more information, see Publishing Center.

• Go to Publish: Click the icon in the toolbar to publish the latest content that is pending release. For an offline integration pipeline that has been submitted to the Publishing Center, you can go to the Publishing Center to publish it.

• Go to O&M: Click the icon in the toolbar to perform O&M on published instances. For an offline integration pipeline that has been submitted to the Operation Center, you can go to the Operation Center to perform O&M operations. For more information, see Operation Center.

• Switch to script mode: Click the icon and select Switch to script mode from the drop-down list. This converts the current offline pipeline into an offline script. This is useful in scenarios where you need to automatically obtain field information and generate complete JSON configurations, which can improve the efficiency of script mode configuration. After you switch to script mode, you cannot switch back to offline pipeline mode.

• Refresh: Click the icon in the toolbar to refresh the current offline pipeline task.

• Get Lock: Click the icon in the toolbar to get the edit lock for the offline pipeline task.

• Locate: Click the icon in the toolbar to locate the current offline integration pipeline task in the task directory.

• Unpublish: Click the icon in the toolbar and select Unpublish to unpublish a published integration task.

• Delete: Click the icon in the toolbar and select Delete to delete the current integration task.

  Important

  In Basic mode, after you submit a delete request for a task, the task is updated to the production environment. Perform this operation with caution. In Dev-Prod mode, after you submit a delete request for a task, go to the pending release list to publish the object to the production environment.

③Development canvas

• Visual development: On the development canvas, you can drag components from the component library drawer to the canvas for development. For more information about the development components and instructions supported by Dataphin, see Development instructions for the integration component library.

• Code script development: For script tasks, you can use the code editor to develop offline pipeline tasks.

④Configuration and management

The configuration and management section supports property configuration, quality monitoring, related objects, and history for integration tasks.

• Property Configuration: Configure the properties of offline pipeline tasks, including basic information, scheduling configuration, scheduling dependencies, and runtime configurations. For more information, see Configure offline pipeline task properties.

• Quality Monitoring: By configuring quality monitoring for offline integration pipelines, Dataphin automatically monitors the quality of your data tables. If a quality threat is detected in a data table, the system automatically sends an alert to the specified recipients. This helps you stay informed about the quality of your data tables. For more information, see Configure integration pipeline quality monitoring.

• Related: Related objects of an offline pipeline are the tables that the current offline pipeline task reads, tables that read from the current task, and tasks that use the current task as an upstream dependency.

• History: View and manage task version history.

⑤Console

When you run or preview a task, the console displays structured information about the results. This includes preview results, runtime information, channel information, step measures, log information, exception information, performance diagnostics, and the runtime code.