Dataphin:Integrate the SDK offline

Section

Description

①Offline integration directory

The task directory centrally manages created offline pipeline tasks. It supports various management operations:

• Directory toolbar: Supports filtering, uploading (upload pipeline development scripts), refreshing, creating a new folder, and locating.

• Create integration task: Click the button to initiate a single pipeline task or a script pipeline task.

  • Single pipeline task: Supports visual drag-and-drop to sync single or multiple tables from one or more source data sources to one or more target data sources. For specific operations, see Create integration tasks through a single pipeline.

  • Script pipeline task: Supports script code method to sync single or multiple tables from one or more source data sources to one or more target data sources. For specific operations, see Create integration tasks through script mode.

• Offline pipeline task: Allows for copying, locking, editing, cloning, renaming, moving, deleting, unpublishing, and deleting and unpublishing.

  Important

  Please exercise caution when submitting a task for deletion in Basic mode, as it will be updated to the production environment. In Dev-Prod mode, after submitting a task for deletion, proceed to the pending release list to publish the object to the production environment.

• Offline Integration Task List: To manage your offline integration tasks, click the offline integration task list where you can view them in a list format. For more information, see View and manage the offline integration task list.

• Shift tasks: Both offline script and pipeline tasks, along with standard folders, can be shifted within the offline integration folder. However, only offline script tasks and offline pipeline tasks are eligible for shifting into the offline full database migration folder. Standard folders cannot be shifted into or out of the offline full database migration folder.

  When there are child nodes in a regular folder, it cannot be deleted. When the offline full database migration folder is deleted, it will also delete all tasks contained in the folder, including offline script tasks and offline pipeline tasks.

②Toolbar

The toolbar provides options for saving, running, previewing, scheduling configuration, channel configuration, quality monitoring, submitting, publishing, maintenance, refreshing, locking, and locating.

• Save: To save the pipeline you are developing offline, click the icon in the toolbar.

• Run: Click the icon to execute the current offline pipeline.

• Preview: Click the icon to view integration results after running the current offline pipeline.

• Scheduling Configuration: To access the scheduling configuration, click the icon located on the toolbar. This opens the scheduling configuration tab within the property panel. The scheduling configuration is essential for establishing scheduling rules for offline pipeline tasks that require periodic scheduling during the scheduling process. For more information, see Offline Pipeline Scheduling Configuration.

• Channel Configuration: Click the icon in the toolbar to access the channel configuration options in the property panel. Here, you can set up fault tolerance, concurrency, JVM resources, database parameters, and traffic monitoring for integration pipelines during offline task execution. For more details, see Offline Integration Pipeline Channel Configuration.

• Quality monitoring: Click the icon in the toolbar to open the quality monitoring panel. You can configure quality monitoring rules for data tables in the integration pipeline. For more information, see Configure integration pipeline quality monitoring.

• Submit: To submit the offline integration pipeline, click the icon in the toolbar. This action sends it to either the Publishing Center or the Management Center. For more information, see Deployment center.

• Publish: Click the icon to release the latest pending content. Visit the publishing center to perform publishing operations for offline integration pipelines.

• Maintain: Click the icon in the toolbar to maintain published instances. For offline integration pipelines submitted to the operation center, you can go to the operation center to perform maintenance operations. For more information, see Operation Center.

• Switch to script mode: Click the icon and select Switch to script mode to convert the current offline pipeline to a script. This is useful for scenarios requiring automatic field information retrieval and JSON configuration generation, enhancing script mode efficiency. Note that reverting to offline pipeline mode is not supported after switching to script mode.

• Refresh: Click the icon to update the current offline pipeline task.

• Lock: Click the icon to secure the edit lock for the offline pipeline task.

• Locate: Click the icon to find the current offline integration pipeline task in the task directory.

• Unpublish: To unpublish a published integration task, click the icon in the toolbar and select Unpublish.

• Delete: Click the icon in the toolbar and select Delete to delete the current integration task.

  Important

  Please exercise caution after successfully submitting a deletion task in Basic mode, as it immediately updates the production environment. In Dev-Prod mode, upon successful submission, navigate to the pending release list to publish the object to the production environment.

③Development canvas

• Visual development: Utilize the development canvas by dragging components from the component library drawer onto the canvas to facilitate development. For detailed guidance on the supported components and development procedures within Dataphin, refer to Development instructions for the integration component library.

• Code script development: For script tasks, the code editor is available to develop offline pipeline tasks.

④Configuration and management

Configuration and management sections enable property configuration, quality monitoring, and the tracking of related objects and history for integration tasks.

• Property Configuration: This involves setting up the properties for offline pipeline tasks, which encompasses basic information, scheduling rules, dependency management, and runtime settings, among others. For more information, see Configure offline pipeline task properties.

• Quality Monitoring: Dataphin enhances the integrity of your offline integration pipelines by automatically overseeing the quality of data tables. Should a quality threat arise, the system promptly issues alerts and notifies designated recipients, ensuring you are quickly informed about the data table's quality status. For detailed instructions, see Configure integration pipeline quality monitoring.

• Related: Related objects of the offline pipeline refer to the current offline pipeline task that references the tables read, references the tables that read the current task, and tasks that use the current task as an upstream dependency.

• History: This feature allows you to view and manage the version history of tasks.

⑤Console

The console displays structured information during task execution or preview, including results, runtime data, channel metrics, step details, logs, error messages, performance diagnostics, and executed code.