When you develop and submit an integration task, Dataphin automatically parses the data lineage for tables and fields in the task. Dataphin also verifies the change type and content of the task object and runs a pre-check on the task. This process ensures that the task meets submission requirements and reduces the risk of submission failures.
Lineage parsing overview
When you submit a task, the system parses the data lineage of tables and fields in the development environment. When you publish the task, the system parses the data lineage in the production environment. A single task submission or publication can parse a maximum of 100,000 lineage relationships. If this limit is exceeded, the additional relationships are not recorded and will not appear in the asset folder.
Offline integration tasks support automatic parsing of field lineage for tables from data sources supported by the Metadata Center. For more information, see Overview of metadata acquisition.
If an input component uses a schema to select a table and that schema is later updated, you must resubmit the integration task to update the lineage automatically.
If you use the MySQL input component and select multiple tables based on a rule, the system automatically parses the lineage for only the first 50 tables. If you use the
select *expression to select tables, the system parses the lineage for only the first table in the query results.If a task configuration includes a table from a custom data source that does not have a specified database or schema, the system automatically adds the
default_schemaprefix to the table name when parsing the lineage.If you use a
whereclause in a filter component or a built-in function in a field calculation component to define field logic, the field lineage graph displays a direct lineage relationship.When you submit a real-time integration task, the system collects the lineage of tables in the development environment. When you publish the task, the system collects the lineage of tables in the production environment. You can view the lineage in the table details on the Asset Checklist page. If a table from the real-time integration task is not in the Asset Checklist, you must first use metadata acquisition to add the table's metadata to Dataphin.
When a real-time integration task is unpublished, its lineage is deleted. This only affects tasks that are unpublished from the real-time integration task page. Instances that are taken offline on the O&M page are not affected.
Once a real-time integration task starts running, the lineage information for added or deleted tables cannot be collected or updated.
Offline integration tasks
Submission details
In the Submit dialog box, review the submission content and pre-check results, and enter submission remarks.
Submission Content
Displays the name, type, and change type of the submitted task object.
Pre-check
When you submit an offline integration task, the system performs the following pre-checks. If any check item is configured incorrectly, the task cannot be submitted.
Check item
Description
Schedule dependency
Dataphin uses the schedule dependency configuration of each node to run the nodes in a business flow in order. This ensures that business data is generated effectively and on time. For more information, see Configure offline pipeline schedule dependencies.
Run parameters
Assigns values to the variables used in the integration task to support node scheduling. Parameter variables are automatically replaced with their corresponding values. For more information, see Configure offline pipeline run parameters.
Cross-node parameters
Variable parameters that are passed through to the direct descendant nodes of the current object node. For more information, see Cross-node variables.
Submission Remarks
Enter remarks for the task submission. The remarks can be a maximum of 128 characters.
Check item details
After submitting the integration task, click Confirm And Submit in the Submission Content dialog box. You can then view the check items and results in the Submit dialog box.
Check item | Description |
Configuration check | Checks whether the configurations for the integration pipeline are correct. This includes the ID, name, FileId, object type, schedule, and general configurations. |
Parameter configuration | Checks whether the current values of the integration pipeline parameters are correct. |
Permission check | Checks whether you have permissions on the objects in the integration pipeline. If the permission check for an object fails, click the The system parses all objects in the integration pipeline and displays their operational permissions in a list. The list includes the object name, object type, permission status, row-level permission status, and the permission request operation.
|
Table schema check | Checks if the production table exists. If it does not, the status is Alert, and the error details show: <Table name> does not exist in the production environment. Then, it checks if the table schemas in the development and production environments are consistent. If they are not, the status is Alert, and the error details show: <Table name> table schema is inconsistent between the development and production environments. Note A failed check only triggers an alert and does not block the submission. |
Table duplication check | Checks if any tables in the integration pipeline are duplicates. |
icon to request permissions. For more information, see Real-time integration tasks
Submission details
When you submit a real-time integration task, you can view the submission content and remarks in the Submit dialog box.
Submission Content
Displays the name, type, and change type of the submitted task object.
Submission Remarks
Enter remarks for the task submission. The remarks can be a maximum of 128 characters.
Check item details
Check item | Description |
Permission verification | Checks if you have permissions for the objects in the real-time integration task. |
Operation execution | At this step, the system executes the submission task. You cannot revoke the submission during execution. |