Dataphin:Offline integration

Section

Description

①Offline Integration Catalog

Use the task catalog to manage all your offline pipeline tasks. The following management operations are supported:

• Catalog toolbar: Provides options to Filter, Upload (pipeline development scripts), Refresh, create a New Folder, and Locate tasks.

• Create Integration Task: Click the button to create a Single Pipeline Task, a Script Pipeline Task, or a High-speed Sync Pipeline Task.

  • Single Pipeline Task: Lets you use a visual drag-and-drop interface to synchronize single or multiple tables from one or more source data sources to one or more destination data sources. For more information, see Create an integration task by using a single pipeline.

  • Script Pipeline Task: Lets you use a script-based approach to synchronize single or multiple tables from one or more source data sources to one or more destination data sources. For more information, see Create an integration task in script mode.

  • High-speed Sync Pipeline Task: Uses the Apache Arrow columnar memory format for end-to-end data transfer, which is ideal for high-throughput synchronization of large data volumes. For more information, see Create a high-speed sync pipeline task.

• Offline Pipeline Task: Supports operations such as Copy, Acquire Lock, Edit, Clone, Rename, Move, Delete, Take Offline, Take Offline and Delete, and Change Owning Project .

  Important

  In Basic mode, deleting a task takes effect in the production environment immediately after you submit the request. Proceed with caution. In Dev-Prod mode, after submitting a delete request, you must publish the change from the pending release list to the production environment.

• Offline Integration Task List: Click the Offline Integration Task List to manage your integration tasks in a list view. For more information, see View and manage the offline integration task list.

• Move Task: Offline script tasks, offline pipeline tasks, and standard folders can be moved within the Offline Integration Catalog. Offline script tasks and offline pipeline tasks can also be moved into a folder for offline full-database migration. Standard folders cannot be moved into or out of a folder for offline full-database migration.

  You cannot delete a standard folder if it contains sub-nodes. When you delete a folder for offline full-database migration, all tasks within it, including offline script tasks and offline pipeline tasks, are also deleted.

②Toolbar

The toolbar provides options for Save, Run, Preview, scheduling configuration, channel configuration, quality monitoring, Submit, Go to Release, Go to O&M, Refresh, Acquire Lock, and Locate.

• Save: Click the icon on the toolbar to save the offline pipeline that you are developing.

• Run: Click the icon on the toolbar to run the offline pipeline that you are developing. If you do not have the required row-level permission, you can click the Apply for Permissions icon to go to Management Center > Permission Management > My Permissions > row-level permission to apply for the permission and then run the task again.

• Preview: Click the icon on the toolbar to preview the integration results of the current offline pipeline.

• Scheduling configuration: Click the icon on the toolbar to open the Scheduling Configuration tab in the Properties pane. Scheduling configuration defines the scheduling rules for periodically scheduled offline pipeline tasks. For more information, see Offline pipeline scheduling configuration.

• Channel configuration: Click the icon on the toolbar to open the Channel Configuration tab in the Properties pane. You can define fault tolerance, the number of concurrent processes, Java Virtual Machine (JVM) resources, database configurations, and pipeline traffic monitoring for the offline integration task. For more information, see Offline integration pipeline channel configuration.

• Quality monitoring: Click the icon on the toolbar to open the Quality Monitoring panel. You can configure quality monitoring rules for data tables in the integration pipeline. For more information, see Configure quality monitoring for an integration pipeline.

• Submit: Click the icon on the toolbar to submit the offline integration pipeline to the Release Center or Management Center. For more information, see Release Center.

• Go to Release: Click the icon on the toolbar to publish the latest content pending release.

• Go to O&M: Click the icon on the toolbar to perform O&M on published instances. For more information, see O&M Center.

• Switch to Script Mode: Click the icon and select Switch to Script Mode from the drop-down list. This action converts the current offline pipeline to an offline script. This is useful when you need to automatically fetch column metadata and generate a complete JSON configuration, which can improve the efficiency of script mode configuration. After switching to script mode, you cannot switch back to the offline pipeline mode.

• Refresh: Click the icon on the toolbar to refresh the current offline pipeline task.

• Acquire Lock: Click the icon on the toolbar to acquire the edit lock for the offline pipeline task.

• Locate: Click the icon on the toolbar to locate the current offline integration pipeline task in the task catalog.

• Take Offline: Click the icon and select Take Offline to take a deployed integration task offline.

• Delete: Click the icon and select Delete to delete the current integration task.

  Important

  In Basic mode, deleting a task takes effect in the production environment immediately after you submit the request. Proceed with caution. In Dev-Prod mode, after submitting a delete request, you must publish the change from the pending release list to the production environment.

③Development canvas

• Visual development: On the development canvas, you can drag components from the component library drawer to build your task. For more information about the development components supported by Dataphin, see Integration component development guide.

• Code-based script development: For script tasks, you can use the code editor to develop offline pipeline tasks.

④Configuration and management

This section lets you configure and manage task properties, quality monitoring, dependencies, and history.

• Properties: Configure the properties of the offline pipeline task, including basic information, scheduling configuration, scheduling dependencies, and runtime configurations. For more information, see Configure offline pipeline task properties.

• Quality monitoring: When you configure quality monitoring for an offline integration pipeline, Dataphin automatically monitors the quality of its data tables. If a data quality risk is detected, the system sends an alert to the specified recipients, helping you stay informed about the data quality status. For more information, see Configure quality monitoring for an integration pipeline.

• Dependencies: Dependencies for an offline pipeline include the source tables it reads, the downstream tasks that consume its output, and other tasks that have a scheduling dependency on it.

• History: View and manage historical versions of the task.

⑤Console

When you run or preview a task, the console displays structured information about the results. This includes the Preview Result, Runtime Information, Channel Information, Step Metrics, Log Information, Exception Information, performance diagnostics, and Runtime Code.